diff --git a/Copy_of_LS_DS_432_Convolution_Neural_Networks_Assignment.ipynb b/Copy_of_LS_DS_432_Convolution_Neural_Networks_Assignment.ipynb new file mode 100644 index 00000000..05294eb2 --- /dev/null +++ b/Copy_of_LS_DS_432_Convolution_Neural_Networks_Assignment.ipynb @@ -0,0 +1,863 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "kernelspec": { + "display_name": "U4-S2-NNF-DS10", + "language": "python", + "name": "u4-s2-nnf-ds10" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.6" + }, + "nteract": { + "version": "0.23.1" + }, + "colab": { + "name": "Copy of LS_DS_432_Convolution_Neural_Networks_Assignment.ipynb", + "provenance": [], + "include_colab_link": true + } + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "view-in-github", + "colab_type": "text" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fc4yMj7mtCAZ", + "colab_type": "text" + }, + "source": [ + "\n", + "

\n", + "

\n", + "\n", + "## *Data Science Unit 4 Sprint 3 Assignment 2*\n", + "# Convolutional Neural Networks (CNNs)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "0lfZdD_cp1t5" + }, + "source": [ + "# Assignment\n", + "\n", + "- Part 1: Pre-Trained Model\n", + "- Part 2: Custom CNN Model\n", + "- Part 3: CNN with Data Augmentation\n", + "\n", + "\n", + "You will apply three different CNN models to a binary image classification model using Keras. Classify images of Mountains (`./data/train/mountain/*`) and images of forests (`./data/train/forest/*`). Treat mountains as the positive class (1) and the forest images as the negative (zero). \n", + "\n", + "|Mountain (+)|Forest (-)|\n", + "|---|---|\n", + "|![](https://github.com/LambdaSchool/DS-Unit-4-Sprint-3-Deep-Learning/blob/main/module2-convolutional-neural-networks/data/train/mountain/art1131.jpg?raw=1)|![](https://github.com/LambdaSchool/DS-Unit-4-Sprint-3-Deep-Learning/blob/main/module2-convolutional-neural-networks/data/validation/forest/cdmc317.jpg?raw=1)|\n", + "\n", + "The problem is relatively difficult given that the sample is tiny: there are about 350 observations per class. This sample size might be something that you can expect with prototyping an image classification problem/solution at work. Get accustomed to evaluating several different possible models." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "1eawBP-otCAb" + }, + "source": [ + "# Pre - Trained Model\n", + "\n", + "\n", + "Load a pretrained network from Keras, [ResNet50](https://tfhub.dev/google/imagenet/resnet_v1_50/classification/1) - a 50 layer deep network trained to recognize [1000 objects](https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt). Starting usage:\n", + "\n", + "```python\n", + "import numpy as np\n", + "\n", + "from tensorflow.keras.applications.resnet50 import ResNet50\n", + "from tensorflow.keras.preprocessing import image\n", + "from tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions\n", + "\n", + "from tensorflow.keras.layers import Dense, GlobalAveragePooling2D\n", + "from tensorflow.keras.models import Model # This is the functional API\n", + "\n", + "resnet = ResNet50(weights='imagenet', include_top=False)\n", + "\n", + "```\n", + "\n", + "The `include_top` parameter in `ResNet50` will remove the full connected layers from the ResNet model. The next step is to turn off the training of the ResNet layers. We want to use the learned parameters without updating them in future training passes. \n", + "\n", + "```python\n", + "for layer in resnet.layers:\n", + " layer.trainable = False\n", + "```\n", + "\n", + "Using the Keras functional API, we will need to additional additional full connected layers to our model. We we removed the top layers, we removed all preivous fully connected layers. In other words, we kept only the feature processing portions of our network. You can expert with additional layers beyond what's listed here. The `GlobalAveragePooling2D` layer functions as a really fancy flatten function by taking the average of each of the last convolutional layer outputs (which is two dimensional still). \n", + "\n", + "```python\n", + "x = resnet.output\n", + "x = GlobalAveragePooling2D()(x) # This layer is a really fancy flatten\n", + "x = Dense(1024, activation='relu')(x)\n", + "predictions = Dense(1, activation='sigmoid')(x)\n", + "model = Model(resnet.input, predictions)\n", + "```\n", + "\n", + "Your assignment is to apply the transfer learning above to classify images of Mountains (`./data/train/mountain/*`) and images of forests (`./data/train/forest/*`). Treat mountains as the positive class (1) and the forest images as the negative (zero). \n", + "\n", + "Steps to complete assignment: \n", + "1. Load in Image Data into numpy arrays (`X`) \n", + "2. Create a `y` for the labels\n", + "3. Train your model with pre-trained layers from resnet\n", + "4. Report your model's accuracy" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CLdGdXCatCAb", + "colab_type": "text" + }, + "source": [ + "## Load in Data\n", + "\n", + "This surprisingly more difficult than it seems, because you are working with directories of images instead of a single file. This boiler plate will help you download a zipped version of the directory of images. The directory is organized into \"train\" and \"validation\" which you can use inside an `ImageGenerator` class to stream batches of images thru your model. \n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "moRVuHUqtCAc", + "colab_type": "text" + }, + "source": [ + "### Download & Summarize the Data\n", + "\n", + "This step is completed for you. Just run the cells and review the results. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "AR66H8o9tCAc", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 50 + }, + "outputId": "5205ec08-c8e0-4412-d85d-a23a43a51afd" + }, + "source": [ + "import tensorflow as tf\n", + "import os\n", + "\n", + "_URL = 'https://github.com/LambdaSchool/DS-Unit-4-Sprint-3-Deep-Learning/blob/main/module2-convolutional-neural-networks/data.zip?raw=true'\n", + "\n", + "path_to_zip = tf.keras.utils.get_file('./data.zip', origin=_URL, extract=True)\n", + "PATH = os.path.join(os.path.dirname(path_to_zip), 'data')" + ], + "execution_count": 1, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Downloading data from https://github.com/LambdaSchool/DS-Unit-4-Sprint-3-Deep-Learning/blob/main/module2-convolutional-neural-networks/data.zip?raw=true\n", + "42172416/42170838 [==============================] - 1s 0us/step\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "MNFsIu_KtCAg", + "colab_type": "code", + "colab": {} + }, + "source": [ + "train_dir = os.path.join(PATH, 'train')\n", + "validation_dir = os.path.join(PATH, 'validation')" + ], + "execution_count": 2, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "PrKeWLiKo4cg", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "7e785608-f8a6-4b8c-ed59-4718561ec359" + }, + "source": [ + "train_dir.shape" + ], + "execution_count": 42, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(50000, 32, 32, 3)" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 42 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "OsI9BQLotCAj", + "colab_type": "code", + "colab": {} + }, + "source": [ + "train_mountain_dir = os.path.join(train_dir, 'mountain') # directory with our training cat pictures\n", + "train_forest_dir = os.path.join(train_dir, 'forest') # directory with our training dog pictures\n", + "validation_mountain_dir = os.path.join(validation_dir, 'mountain') # directory with our validation cat pictures\n", + "validation_forest_dir = os.path.join(validation_dir, 'forest') # directory with our validation dog pictures" + ], + "execution_count": 3, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "NUs1e5-XtCAl", + "colab_type": "code", + "colab": {} + }, + "source": [ + "num_mountain_tr = len(os.listdir(train_mountain_dir))\n", + "num_forest_tr = len(os.listdir(train_forest_dir))\n", + "\n", + "num_mountain_val = len(os.listdir(validation_mountain_dir))\n", + "num_forest_val = len(os.listdir(validation_forest_dir))\n", + "\n", + "total_train = num_mountain_tr + num_forest_tr\n", + "total_val = num_mountain_val + num_forest_val" + ], + "execution_count": 16, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "ycI0lv0S8hdb", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "b39cf07b-7126-4b74-91ad-26156fcb5a1e" + }, + "source": [ + "? validation_steps " + ], + "execution_count": 54, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Object `validation_steps` not found.\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ZmklbgSMtCAn", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 134 + }, + "outputId": "0fb8191f-81b2-442e-e933-d58bcceb2402" + }, + "source": [ + "print('total training mountain images:', num_mountain_tr)\n", + "print('total training forest images:', num_forest_tr)\n", + "\n", + "print('total validation mountain images:', num_mountain_val)\n", + "print('total validation forest images:', num_forest_val)\n", + "print(\"--\")\n", + "print(\"Total training images:\", total_train)\n", + "print(\"Total validation images:\", total_val)" + ], + "execution_count": 5, + "outputs": [ + { + "output_type": "stream", + "text": [ + "total training mountain images: 254\n", + "total training forest images: 270\n", + "total validation mountain images: 125\n", + "total validation forest images: 62\n", + "--\n", + "Total training images: 524\n", + "Total validation images: 187\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dQ4ag4ultCAq", + "colab_type": "text" + }, + "source": [ + "### Keras `ImageGenerator` to Process the Data\n", + "\n", + "This step is completed for you, but please review the code. The `ImageGenerator` class reads in batches of data from a directory and pass them to the model one batch at a time. Just like large text files, this method is advantageous, because it stifles the need to load a bunch of images into memory. \n", + "\n", + "Check out the documentation for this class method: [Keras `ImageGenerator` Class](https://keras.io/preprocessing/image/#imagedatagenerator-class). You'll expand it's use in the third assignment objective." + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "67i9IW49tCAq", + "colab_type": "code", + "colab": {} + }, + "source": [ + "batch_size = 16\n", + "epochs = 50\n", + "IMG_HEIGHT = 224\n", + "IMG_WIDTH = 224" + ], + "execution_count": 6, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "B1wNKMo1tCAt", + "colab_type": "code", + "colab": {} + }, + "source": [ + "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n", + "\n", + "train_image_generator = ImageDataGenerator(rescale=1./255) # Generator for our training data\n", + "validation_image_generator = ImageDataGenerator(rescale=1./255) # Generator for our validation data" + ], + "execution_count": 7, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "ndsuM4L9tCAv", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "8586f0f1-ad1e-4b5d-c491-7ad59d5237a4" + }, + "source": [ + "train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,\n", + " directory=train_dir,\n", + " shuffle=True,\n", + " target_size=(IMG_HEIGHT, IMG_WIDTH),\n", + " class_mode='binary')" + ], + "execution_count": 8, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Found 533 images belonging to 2 classes.\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "9kxlk3optCAy", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "86d2e17e-205f-4226-acff-4edbd0387e03" + }, + "source": [ + "val_data_gen = validation_image_generator.flow_from_directory(batch_size=batch_size,\n", + " directory=validation_dir,\n", + " target_size=(IMG_HEIGHT, IMG_WIDTH),\n", + " class_mode='binary')" + ], + "execution_count": 9, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Found 195 images belonging to 2 classes.\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2l7ue6NutCA0", + "colab_type": "text" + }, + "source": [ + "## Instatiate Model" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "mKNIYOEItCA0", + "colab_type": "code", + "colab": {} + }, + "source": [ + "from tensorflow.keras import datasets\n", + "from tensorflow.keras.models import Sequential, Model\n", + "from tensorflow.keras.layers import Dense, Conv2D, MaxPooling2D, Flatten, Dropout\n", + "\n" + ], + "execution_count": 32, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "ajKnlblqEIsk", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "2b4101f7-1ff7-49bc-e6c1-1955440ab470" + }, + "source": [ + "train_data_gen" + ], + "execution_count": 51, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(224, 224, 3)" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 51 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "2Qb-Z05W7pXJ", + "colab_type": "code", + "colab": {} + }, + "source": [ + "import imageio\n", + "import matplotlib.pyplot as plt\n", + "from skimage import color, io\n", + "from skimage.exposure import rescale_intensity" + ], + "execution_count": 12, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "HbKHRvas5xBr", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 139 + }, + "outputId": "7bd321f8-1bb9-44b3-967f-72f259ad1279" + }, + "source": [ + "class_names = ['mountain', 'forest', 'mountain', 'forest',\n", + " 'mountain', 'forest', 'mountain', 'forest', 'mountain', 'forest']\n", + "\n", + "plt.figure(figsize=(10,10))\n", + "for i in range(2):\n", + " plt.subplot(5,5,i+1)\n", + " plt.xticks([])\n", + " plt.yticks([])\n", + " plt.grid(False)\n", + " plt.imshow(train_dir[i], cmap=plt.cm.binary)\n", + " # The CIFAR labels happen to be arrays, \n", + " # which is why you need the extra index\n", + " plt.xlabel(class_names[train_labels[i][0]])\n", + "plt.show()" + ], + "execution_count": 43, + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOMAAAB6CAYAAABN5+THAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy925IjSZKm96md3B1ARB66unt7hsvlG1DI1+MbUISXvOQrUCi85auQlCW5PbNTXd1VmRkBuLsdlBdq5kDWzHRVtlTKzkVZNQTZEQAC7mZq+uuvv6qJqvLr+HX8Ov7LD/df+gv8On4dvw4bvxrjr+PX8W9k/GqMv45fx7+R8asx/jp+Hf9Gxq/G+Ov4dfwbGb8a46/j1/FvZIQveXGMUad5RlXR1lAU6b8TQORu3c4J3gnS/+1EEAERwTlB+jtFjnf3/8nxgarQtKHN0i+PSRjnHCL9tY/pGVUUxf5nz8fnysPLxmtVAUVEEOceX2LvB0b6Rz77rdyv20n/LvYsIvfffvYVlc8TSfLZ//txmkmBv/zwiZfr+vkLf4Hx9PxG3//u92ir1JJtPlul1gLt8XsKzjmcD4g4nPd4H/r98jjn4GFuVZVasn2mNmot9pnjXh/XOO6QPNw/jvvovCelhPfB1lKI9rdUUW2gSm2VkjOttYc7Znd53O9aKqUUVBuq2l+rn93r8U9V0KY07etH7T6MzxqLyTnp121r+1iHx2seLrNf8vi/H374+J2q/vZfmpMvMsZpnvlv/7v/nlIy63qjlIIX8KI4UZKDxYMXOE+e5yUSvWOeAqc54r1jniKnyW6sE4cXMyrnA+KDWafYBKsqt3Vl23ab5AZNFec88zyRYjRjaPWYpFortVZUlVIKtVacd8QQEWc3rWEbSW2VnHdUGzEGpnmymzxuap+8WiooCM4eIp89Ukr2XUSIMRL7v50LOOdRoJRKbY1h1morz66VvohrpdUGCA1oCv/j//K/fskU/ezx/ne/53/4n/5nttePfPjuH9hvr+zXF15/+BNl21AVVO27zednzs/fEOLE5fkNz+++IYTEdLown5/NQIMQgqO1yvd//gd++PM/kvPGx+//xKcP39Fqpe2ZlrPdS3WICt550jwRU0Scw4WI857T+Ym//6//A2/fvWdZFt7/5reczmdayeT1RiuZl5dP/Onbf+R2u3ZjqyiN1hqtFpo2Pnz4gT99+y37vpH3lW290VqltrFOoDWhVaE1WNfMthZaU/ZcyMXWUlMzaOccp9PMPE+2npdESqGvqh0o9plV+zOUArXazP/v/9v/8f/8a3PyRcbYWuPl9YWSM+vtRqmFIEp04FBaEJsUB4VA9QXnHeoSJAUcWhrVVdQJDUcT6btjQLxnLNSGoK2x7jv7nrsxKk3BiUPLRokBQXHakL7blVrRpsfi1qY476BmxDlUG00bjUZrlVIyTRu0iJeG98O3m0G2agaOKoLHi3vAA30nLxs1eJw4aoyU0L2I84jzoFBboz54Be27qaqgSN9IWt+5+8/6Pfgqo3uw1hrBe4gRjZEUE04VxaEEQJjmE9OyEOJEiLHfnQbaANuoaILWiraKFjO6ljOtFLQ2pCnBOSQm27T8RHAB7wPLebGN0HtimvAxsiwnvnn/jqfnZ9I0cZonphhQB4FGqx6hUcp7ztvpboyqZmy12L1UePn0goijtYayoiqAQ0QNvKihIofiQyBEoTVFxbx2U6VWe3gvOO+6d5SO7B6Qk7jDA7emtIat258xjV9kjCVn/vTtn8h5Z71eqaUQHMwBgsApOWT2JC+EHDi1BMFTmSCekOBo6si1u/YDKQxvaAu3NLWFqUrOhVz6bqNKawYpfb8hAgRnntl2InvfHVIapAjeI8527tJ3zaa2Q0KjpgnKjPf+gNzI58boxYOzW2a77+d3WARiTIQQOnTziJhxa/eAxwvlYZL75tH65kGHgCDUmr9kin72aNpYbzcohRQiUQSvDfYzJUaQgEoCcSznNzy9eY8PCR88iPaFXxDNiDaoCq2htVC3G2W9UvJO2zdzDapMMTGFQPCB8/mZ03wmxMjT22eW84IPkfPlwjTNpJR4fvOWZVnw3UiDDzZvJaHaKE9n3r597h7ONlewTbiUQmuVaVp4fV15fX2hNeXl5ZXaGojDedv4RQSpDnWQ8Dhn8xFLsc9RZd+FXAyWhuAIwRnSErWNCfsd4tHWqDX37wC1QGs/rXT7Ms+oym29knfzjLUUogMJUB2E5igu4Lyj+krN4NWhxUHL0Dwq0OoIpGCEdeAx/wq5VEo3qpKreTtVqo5YAJz0GFXsbxu6NGO9v8bZjigCIeCco9ZC7t5wGCTdD9QgSAi2N/Qdr7VGKwUAEY86u/HDSLXHQqr952WnBoOpBkF7bCu+P9tjxCK51M+NUUdcYgb99TwjtFIQbebBUVoI3fOpTapLIGYIMU2EmB5iYMMvaLMfjXvQKloLrRa0FCy2aAhCcI4UIiFETvPC+XwhpsjT8zOny4kY7d/zvBBj5Hy+kFKykMYFnBh0rs6QRCLQpnTMQWt3oxwhyg8/fGCaZvY9431koA6jKh6enTkH5z3huD5AbOOt1VGb3D3ieO6vHdzH2Mi1r8XhHX9xYwRwIniB0A0geViiI3hYkueUIikIS0rMUyIGR4zRAvHhzfqEtmMhj6DbYqr2YHQ4wakZqXaYaoRLf3YdGvtO+HRjFMQWmUiHjO4whga2CLVRm8WIztlm0BrHzYb+b2//9s73hSso1fbDQUpoJ6XEHxBTsGtVwcgs74/rHgRBeyQVjrus1Fo6fP1K2uH+vbUZGUdTWjVEUmuz3T9NOBfwIdBao/RNScXQSSzRPGTTDvnskVLkfDpTp4lp8jw/n3HiuCwLy7QQQuByfsPpdMGHwPJ0YlomfAjMpzNTmvDe471Bf0EQHcSIGZu21kMX28DuOEhw4gnB4V1jnhZOpwutwfV6xbmISEOkf2fuiEt7COSCu5M2YsaUM0DDKMoGtD7PAK5zFPbzWiu12MMQ3ecc4782vsgYBSw+9IYom8KSHM+LQdPzHHl7nkjBcZoT59NMDJ55ScSU8P5Oeozo1m6m4eqqhvFtafZb2w1guBJp/SapwUvxQpwCU/R3Y2wg4jrr57r37eRNU8T7A9fXVlC1uLKpQOuwZXgy+v0XM8bQDUpdoWU5duUDc4uj/Zh3FcEFTwjRNppi1z0M0eDVfdjCt/hxeNyvMmqF1kmjVimlGhLJlRA9y3ImxAkVT6kVakcTrXayylPrjOA7ZDOksiwLXt4jqsToSMHhvedyunBeTngfmOcL07SAEyTYwzlHSokQPK57UtdhvhybR6XmbPevNSPFVHsoEhDn8M6RvK2H8+UN795+QwwL1+tKDH/qZEqjLyZUSw9FIMRADAkAH8AXqFXYdxgrU6ko1Tblsak2Nf6hVWpt7HuhlNo//ysZowO0e0dx5iGTF1JwDw9PCp4YAiH0Hc7dWUj7MLl7w8NDPHiH4fY7ba4K0gRxw2uOdIhNovfeUi0V1JkBeHeP2Zqx4TingDfvdHwHPSj14dUOeMn4M+ZpzYOCuNZh6H2HHV9ej3TN/VJG6sSpAn2SOk4fhPwRqzKgUf1Zk/i3jiPV0PQgvQa0QgTvAyEEqjrKmCOttFoP+GZoZpBeZpAheGSaEOC0JJbZUhRP5ydOywnvAtN0IqXZwhbXaKI4kft6UR683fGFu0E2Y2c7tG/Da3q9kzE9to8hktLENFViSIjziPQN9ZiFjkBUDj4CoKnDe4fSHoiax0d/f/eiNmd3Iu6ecvl5makvhKlGlIgoLhgjuCTPZY5M0bGkwBId0UtnWBtOgSrUjMVuD19s7MIGTVs3EMPtIq6TKAYzLefT0NrBQTeA4IQYHCl6WveuBz7Xe0zXqna4oA/eDLy4bi167L7ixEgJ/VEaA/3MU7ke+7WHmFGO/zjMGm1oK7QyclnF4ii1a3HyMLfQ4Vcj1/bVYKp2L2O7VDu27mPDUaXkbFfkAt4nvBN8CPhgZNjzmycuTxd88Hjn8CN1VCZa3hGBFL0Zpwh7g7pmnFT8rji/oyjNKU2aER95p9ZKdI7TNDOFQAyB87KQQuhw8g5ZS2dNSynsWz6uIQSLD9fbjqrvcDuRptm8WN3JeWXkEgdhpoqhAMzT5bJZykkrzoF0AifGcGycHE7lfivH4+57ftogv9gzBhriFImCBMfTHHh3mVhSYArCKXmCsxguSMVpQ3Mlq02soU0LlktplNxzb07h8GhC9JZrdN4hzveduzOYYnuxCEQnTDEwp0hrjUymlPtrB3QtpX0WiyoGO300YqB1g1FV88DNIbROHty95N0Y23CMSNVjF7T8qVmX2bjt1q1kqOWIfW1n6JRV36PG91Mae6ns+1f0jKpGTLWKVFtBoub9vQ/QIK8rNRfitDDFhPee5Xzi8nwxGHu5cHlzsTxjFwAImJH3Ba09vmpNua072+vNwpJ6pVWLP5s0mjTytvHDX/7M9eWFZUr8/t17nk5nLqeFv/v97/GXy2dz2Goj7zulVkpp7JsZpnMe70yYcLttgMf7iZROnJYnRALX2yduPecIhqxsVmEvlkrb95W8r4YGWkEceC+Wk55Cv47SiSNDF7a33RHGI/P/U+PLPKPcWUzvLD8TvRgsjb57RDMmL3fPMDzTozHahRhh0D/anp1BgEE8uu6VdDgwxu87oOyv8U4QFUqnV8aC0zZuUu0esxuVGIdqRi04NXa+/9b+xsGO9e/Qqd8BLD+/vZ9bzYEBurfX1miDiWx3tm687pG8GXCxdpXT1xravbOiA2jYvRjpmNZQMUP1YrGYiSMSMUamKRFj6sTc3RhRo/eN3S7UHt/nBlu2hHouUIv97SoVlca2rnz/4SOfPnzgPM8sIeEQvHMW0zEc+B1gjri71kLOmVqbeV5v+VpL2lvYYeGMEVKGth6814Oa654nHJxCV5vJfS04N/KJD2hr/PejGHG876fGF3vG5GyyvPOIwnTEh0biTMngihPusrdxsdINUcwYvTeDQG2HtEDhLo8ScUecagyk0eYAbaQ1ajXaWroHzOXA7KW0nnBt5Fr7TuUORlRUKc3SGuNmoo+GpkYwdMkXTU0ggCCYNzCP0gmGsUha/6zB7I7wFnOIpfbvpUquHKmN0nfTUpXrrbCX9rOSxX/LsA2yIIDH7rn3gRgnmgvMy8LT5Q0hRubTmfPTWyM35onptFjOVuHlevssVQPmcVsxlJHLTqlmJC8vN27Xrb/WAxbnazfG2/WVP/7xH/n+z9/xfD4TxR2KpFwsRhdRi/saBB+ZJiWEineFVsUYzQMmKuu68eHDR/Y9c72uaGdbvbc0TqvuPkGYp7P5M7JmpCxiDMbQeof30lndRs6ZnPcjnzg22mGA3jtCCIfn/Wvji4zRiTAHh1clYIzXkgJTCswxMEfHKTmCk2OXsOl5CHd7/s0WqWJkZ4/14DDGAQ2dSDeWBrXS6sj5SSdKlHXdaaXrFatpWWtr7DlTmzGAuVWqKt55YkgmU2uWrJY2SOr+rBaTDi8hXYtZtR5pCS+O0FleyyaO2LQdDPFQYdi3tOemyporudjmsJVGrtrlV2aktWGe42fmp/6m0WNC7xzBm9cgRNK8oK3x/PyG3/32t0zzzOn8xPPbd/gYaQhF7HpebisfP75QWiOXSu4LOe8bZd9prbFuK9u+mjG+rlyvGyKOGGdCnOzOSAVpvHz8yP/1f/7f/NMf/8i7t2+JEqjZ0gl7qSDB1gb1INR8NLi4bTtNHSUbi7ltmVorr69Xvv3Td6zrxp5XI/G8x4dITNMBMQefQKsWsugdoTkxbaz37jAw1UprlW1fWdcN8zTeUIEaAhvE4jR1IchPjC/2jF7EHkiHEPYYsZJ3zjxeD4vADLH9+IMQBndiYZOMf3CwqANyPsLHB/9vO2w3ADFqurXWYWk7GMmGdjlaY4iZB4Nb1TzjI7V0ICF95MzuelFUcUjfqUEwCr5TTPf3PpBF9qMOP2vtWlUzwFzNaLfcqJ1oKlVoKl8Vpn6G0wYS6blZHwNpmpgeHj5G8rivasqhbS+UWtlLZi8Ws+VtI287rVVu28q2rZRaeX1dud52RBzTLMQmiHRjpLKuK6+vVz69vJJiNF3yvpP3/LCpccyUOIfvQhHvmzHqI1bD1pR5r8K+5662Gsy4HDrk8frH2f78L/XCBz8IvYf5PVJTgqi7v6/D0pHj/sU9owjMyeNFiKJ4lHka7lssj9hzcu4x9lCORaoix9KvVVHR+yIe1tsaWpuRGv2GWMxXDapyXzymzKkHITIwfFOlqFL0HluoGiTMVXHNdJUqBehkDdVIoQJ7FWOOQ0ZWu/iSC2UvoDCHhbnrLL2belWD4kMBZ3FGy9lIIeQhL6bkXMm5Upqy7ZW9NEptBk13u76qQkOoX8szSq98cR6VQBNPc56K7ew7kdfSyHvhplc+FUWceagtm+j94+uVD59eKHVcWz3gb+u61z3v7GW33NtaqHvFeYP20TkUE+vXulPzhkNJ3uERWqnkPbNtGy+vr3z49AnnIISHjVx60OA98zLRUqTWxjRPtNpY942nDyd8FNbbqEopBC/2+tZsQ9kLQ4BiCLWHGdW8o/a1OphD6ZVIo9hBuecczXv649m5cKTY/tr4MpjqhNMcCNJITvBixhijJwTBecsDjgB36Dx/nEMcX9zoGGM5XbVshbFx5tnMZI16b133qK12jO+hQ+HcGnUAzSao2o64t5EbG8ZoN1eb5Y1KM6NraobnxP5mDMq0C06MwGhiLGjpkyYqXObIZbZc5rJE5jDimR3H3hcnUEc1iR7St22/L+h1LWydOf3waeO2mvE2cajIQXD90kOAEEwPrC6h4qleKc7m6SaRj7kRNFNvO/kvH2mqbFthXXsMeL3y6dOrzdWgBhiZWvvepRZKN9I9F0pp+OBxJ4heqA32vJO3V+p6w2sjeU90QiuFfd24XVc+fPzEsizEFDifF2IMxkt425h9jJxi6Ourb/ytsbedHz79QHx1iKys10LTnRAFl6wcsLJx2zvbPuBPG2ule2TXjVFBvKFA7aSW7/GzyXO1e1HL0Q7SaOSn/9r4cpjaYWlwDi+Kd64zrHKfkAcmdLzPXPudURUZ/3pgUh+fBwupiuhgFT/3EgO4HHhfoTV3pDPuRtj/tI6p0s70dfmXGhz2osf3ctIs5SFKEwUValFqMb9u4pUOpJtjbA7S9bB2zffd8Mg9tTt0Hd9x1NDZ97HrbtI60fW1PGMXYBxVKO5AFA2LV/dicXwulW23zWPfCrebebrbdeV222itdi/R5x3LR4NSWj1UMrWYOkWEQ0c6YF7rKQJbYz09NJjSVsk5s+07iFJrsrhNBJw/0lyuJ+ulX5+qGvESPSF6Y9zFUi0O03Mq96KFY46Gh+xraoQt2sm6x1Dps/pV+dHPRD5Li/3U+GLPeF4CQZTkG16UFD1T8sYayUhNjGHGN/C4cI/lzItZaQ0YZPHOBNYpBmLf5WrdaT3RH7xH0yix6nFcs8oOg0hyj7VU0U7mOMFg9Ui3OPOjuVb2LVO689F+z0LPc9oE+6Owdg6J5+cLTjyRGacRV8WU6oZ28cEIkeagemePg9q+s8S+I4gFT4zKVBXvJy6XSlPYm1Ia+NvLl0zRzx6CI4STqWuap6mwl8rrtlFq4zXAp48fcc6gfWmWs825kHs+b1031uuKajsM0fbc0sMJJZdC7on5XIxccd6z7q98/HQCbZSy0Uqm5EyKE2+f33JaFpMeaiNvK3/5y3doy6Rp4nZ9Ik2JlCKn06mzlY6YQq+csI0BRl7QRCFWcJKRtlPVhN9NoeTdYvhaqbnQdiu2rl1sLgLNi8WnfVOxJH/r6zVZaZ8TWrsb4ijjG0b5U+OLjNE74XKKBFGmbowheKbYq7GtCtEm+9gMtNP/d2V97bT3MFuH4EJAfMSJO3YzVNk1k0uz6rpghtEUcm2U1qgVrjfbNasKuQil2Qbgmxlf6iod08YqzlVEKmRl3ypb6d5gkAFY2sQJxBh7xbnnabrwmzffEHyg3KDcLKiVAmR7ow/2t5pCDY5aulEf3Q0EL47mFI/JvxB7/flsUKfUxmuPJf2fr18yRT97GKN5QgvUrJSiXNfKD59u7DnbBekK9LCgLyato9hayXsm98JvswHzhrVaOqNpY9t39rzT1NjtnDPOOT58OpHmCSemKXXO44FpmrmkhSlFUgjQKvt647s/fcvrp4/M88Tru3fM88RyOvH+vTJNEykFg6x4+6odqXnvSMlTiyd4xemONCueLtqNcc9HSqzsmbLuPTTKtFZsbas7UEopFenFwl4cPnlUBd/MGO+yQjPWUsovn2ekJ2C90/6gM6n3xSb6kHTv7/kcXurxGD5z7KhORlWHdKmZ3t2/jiqIUbbCUZpi8VijqZCrGaPrkMP1b+NFCP3znbeJCrV1DbjB0Kat/0XbKJqAc+bJh/jAexOLG3Qdj5ENeazXGFDdPud+rXJPAh8MpsP1e9cQXG3snez6uRDni4eAiJVOqZo3HnnanAuqGW0baOu51o5vWqP16veaS2+vYQTcMMZSd3KxDgp53y0Pp0ouVuMnzuHyjvgB4xwi3jTPPhDF7rH0v1drJe/74X33betzEcg5d5JEOgSWh/CgG2SP67y/s/8m5Hpk6v/lm3TPEY+5GGFG6x5YejjWCUs32PqhnrrL935qfCFMhTkFPJXQ/cgRNzrBiydIeKB+7bkhg+zEuUBK92D2+JoSbNJFMPlUx+jOHdUONVv+rdTGy62wdmX8y1pYd4t39uqoahe2OCNm5pj4uzfPvD0lQrA2IN4Lf77uzH+58roXXvfKh1sm116f1wkIET0WQauFst/ABfJuUj4UarSWDSJ05UkzdlYMtlLNY/oqSOs1nn2X0b6AxVnZmfOeXBshevZcD73nLz0svbBQ1ZjMfS+s643rywvbtiFkRHYEy7vVZrCzlkzJu6lzaj1qME1oUbqQIfea0aEVNmZcnG08IuCCx8dECJF5eWaaTgTnePKeyQnSKjmvlPVGmiLeKbVMlJIR70jryrqulFJIU2JZFt6+fSYlUwadzktP0AdOy4QXRd8+47ZvyNvKy5r5dN0ptaHVU4o3z0iluqE1jagmM+ie8G9qqZJRTuaOIojeH8iFh9zwUA096KX/yvjC1IYwp2ALqmWkWRw28ozeuV7+0j1WaceuacYoBG+vGUtsJEib9livu73W+sU6wflIrT1ZXht7Vl5eM6/Xndwa162y5e4Zm1IVkoPYW4DMl8Tfv3vLH95emJLn6ZJI0fGPH67gfuDDbecvrxtbeQUtneEd19wNEkVrZt9WmvPkvZJ3IxxqcbRqi+yRKHLOYBIdLnnvEKcEuthBR61bIwbP+ZSYpom9VELw7Nko+K8xnDNj3Iq1Ktn2jfV24/rpI7d1xUsl+tJj6509b70B1ErerqZEUkV6/L/vmS13aFoq2yjIdmLso3PMp4VpnqELzkOaiHHi9PSO5fSG6ITn4Dh5Yb9d+finT6wvHyxp7pVaZvZ9o2EdFWKMvF5vhBB4erqg2pjnicvlTJomRKzDw3mZmYIj6TMnfkvJG99/eMXJJ/Zce8G5wdTqrMhcaahmoBhOOiB4ZdtW9t1E6T7oXWYXrXb3Xvo2pHr1oYLjXx9fyKYaTBXExNQHnX1nkOyFozTpnrIZ8K87fDqB9RlEGAtZ5aHLzMMfacpRolKrycaGYmVorw8N9sNG5AWSd0zBM/dHCo45BKYQSKERfekwdmgW71WVxqiZFG50TyvVahIFqM3R1CRapru9A/VDvOB6bqwXLx/pnSM3xQH3TWhvVe1fDaZCZzPr0cGt1kKpBj2RimsVkU5k9DYWrbe4QDvEfyzClhGmjFXB0UHOeU8IiRgnk9WlmZSm3lcn2UME7zkS8qMaw3kh54L3ued7M/RGZoMgGY3IajX2diyAwerfHxY6eCeW5/SQgpIiVNfIzUPtjDwO1REW3fVkIyY0VnggqPZZNPY4b49VQn9t/A15xhktDqVYONFrBgWHNih9hdVe2zXkY2NyKhiDKnzmHWv3aIDFC52mHo6m0rhthQ8vK7k0XtfCug8liKBiLT0Q16sPIHpl8jD5xuIKJ7eT1BH3hq+OqVaew4SkQJ08H2cluEKpma0Y6eC06yOb8qo32m4sqO6Nlnu+0kdaNOjbUkJTtJ8r1vvGWY1f0mh0e7vnPWtXjHjvcFSoG07gnBwt/jy1/98yas18+PCf+eHTle8//JmX68rLp0+8vPyZdd3MEFu2mFKUUYgbo+N8OeOcsMwL59MJEeG2Fm5bplbluhdetwICabLObz4ELs9PnM5nq+6/PDMtJ1P8+AXxCa8NX1domVoa1+uNjx8/EmMkl0yaItO8kJswTZnL5cLzG4OoKSZa7cx6lxpqr6CgQ+r15YXvv/0nyrZSCTxNiTYJKTmWi6PUyscPwscPG60qW66UsveUTzvY0dL/hqJoNhFIiBEfJlwIPVXV27D0joUD1v618YXaVMcyzTQvlLbTiol2XZfdtC4PMqVL631suMvNhjt8MMax65SuJ0Us9nTeg0gXUUNV5boVPr6s5GrM37bbblVHYvOB4PHdGFM3xtllFtkJ6oi54rNjLo5nP+GTkIvneVaCq6x5hXZv6dB6vu11X3nVDQDpmlbnoMWEpkSIHimCq6PWrRuSc/gQmNzneazRzc4keYITqy/04phSNIbxK3lGM8Z/4sPHboyvQ4r2F/ZtR0tB9wytEWIgTZbAXtLM89OZFANv37zjm/e/wXnP9VZ5vWVKVT7dMp9uBURYzifmZSHEwPPbN1yenuxeLGfiPFMbXLfClitSM/5miutaK7frjU8fP+GD57rd8MGznM/gIstSiXEihshpORGCN0/ahQVHXWHra7IW1tdXvv/2W/J65fL8nue33+B85CSJZ5nIteJ0Y1s/kHP9zBgfC4ZLT9FYqw3zxDE1pjkTp6nD1FHBoZTed+mnxheXUHnvkeZp3iPqEd+LUYc0rQ54dxRQ9cTq/WP+mcN+UAscFfejQv8w1O5tu/j6SOQfzGRvoNyNcUA+5+5lXLVZ6iU3aKJYO1QrKnU4gohGvaUAACAASURBVHMEr4Rq8W/nVDuV81DV8ZAUtrhvNLe6616tWuUOcweDOjhi7fGlU2eFxgMp6L18bDB5X2O01tjWm8WCvVveyJtJr1QZXt17K9YNwTNNE/M0k1JknmemyTrq1VbJNeBqI1VPLAYpY7S4MIRAijMpzQZZ40QIE9IaLo+SODlg+9isWlNTIJRKRQm5kHMlxmLzKe7eL2dA4uNm2r0/yuhyIW8b+7pR5kyr1YiXADE4xNHVZB5VK5g+tMwoo1JDlf7z+5r9nAMZ3/3eOuUrwFTHvJxoKZCCQ1vpLJJ1Mqu19nISS6hXyV21cG9mAffk+sH9Y7KmgyIOESVSW+PlduW6ruRcuK4be8lWViRCCP1muIeyKws6mEQ5+cbcy7J+eN3wrSE4vDpQ4UbkEwu7eiqNFD3qBCfGotXRbQzLq43NAMWaN3UrOyR3zWIwOSR7dASg1m7jYFD7s3bIq67j8YfaTn14fIWR951/+E//ka05SnW44JmWmae376ilIbXhSkEUTqeF5zcXYgy8eT7x/v0TMQbmtDBPJwSHuIKKeaUqGxkrlYpxwnnriSpuQmSyNIYGWvPUKpSsRobVSqpWSaN9020dGm6r0hwUdaT5hVwa58sTznumNDNNicvTmRAC8zL1thsmCql7oayZ68cXfvin71ivL9StQlVimpjefcNyPtEIvHmzUNob9n1HXKU0EwTka2Hb90MeZ50EIGKbaugdEOipopx3SskdombjGn5ifBmB4xzTskC1hre0ypB/gZgrHs1iS6ZgcFX03tJi5PBEQF1nLaSXwzjfF3GgEahaLBH98UYuhetqyWRVQcR3FT2Enj+SXvzqgiehLK4xiRnMh9etKyucSeaa0OJEToHmjYJPyeFUcBJQjdTmu0Dd/JnpLA16l75hD6F7bQ3Xempj5DngsCyBXulhP7QwWo7uMQaHh1flqxoiQM47//QP/x+kM5zeI2EiTjMhLD2BrYRiHRWe3zzxzTfvSCnx5s2J37x/NrFCc2g1rkCl0LSQayOzsuuNpuCDJfTFBZzM3RgdEGnN05q1Vc17xbVGKM1E/F0Xqmp9dNe8U1Sp6pjmK6Uq77e9p8omlmXh+flNF2gYYaTYpln2StkKt49Xfvj2L6wvH9GsOBXSPPP+dGJKtpG/KTPIUxcr3LiuV2TPcG3s1pUKJ6Y7tU74vjsk15uVaYeyO3svIxtG+VPji9lU6bSfjDb4IgjugItOXG/l1+sR+/kI0tUI8GOmtJchOXfvKK7uoP1ru7e9F0yEe0eLxo6F3sPSe5M9+eCJKFGEINq7mrtDYTP6r47O4kpv5SGmvvHeEUPolR2mOzVpl2OvZnCC9Vw1ppQj4d1xEf2GHGmOA7LCkUT+bHQvyc+AM7/EUFVTw/hqLLIPptV0HroxRm+xbEpTf1jPU9ebOrWOEJo1lzuY5MfY/TMY9xiqdIb6aIp1wP87FX4vMDed7kiPPb5lfO69JefnsPFg+7mLSYz1tXm654VNTxuCI6UIYsqeeZ5wIrwGazU6ugeOT7c2Jf6A9qOv6mjXAhxdAX5qfHHfVFt5OroGf0bhihNccNCUIJ6AJet/3OlrvEWFA7L5kPA+ogp5b+Ss5Ax5zeSuf5xFmWaTHrUq1pKx5za998QYeHo6MU/Woj5pwasyecEHT+vtG0ouVprkG7gdpLcedL5L1BLLNKMI0xxZlglxcFs/8nr7gVIKL68bL68bqkpKRvx4p8jQxSFWQTKu/CEQHBvVWJSgqFg50X0BfcWAEesfc/14Y5Yz53Qmnd8gEnAyIVgj30ktj3y+LDy9uRCDJ0THnhuSG+utsL4WalX2DOveOxnk0tdGr4QRy6+OFofGsBdovTSuZus60Bt1mSpG8DERp8Vi6lKMiPEJfEJ8RFy4G6fdzX51xw9MR+wTGgrLtPB8fiKhnOaZ6KR3o684Mk6U8zkynZ4p1XK95/OJbd1xzh+FBfteKbn11pIz0zTZhuAtfzyMMKVkBchb/OVhar88M0Zx1ibjYYyGPSKOBgQ1AqYfF2OvechHmk8y7+rDhA8JbbDvBkdLUStb2naExhRNZ6jaW6ZX2wBC9MbQTZHfPC1cTrPlwYr1QfB9B1OxxlC5083iG85lRCoigdBV/M5HvF8Q8Tw9n3n77hnvhU8v3/Hhk8UDMQBqpUTeg3etd84rKBmT1zmLB+HoPjbg6tH2v+djh7fsPuFr2qHd+9a4vazEpZHiidPpGecmvD/hJJAEZteVTKfI+TKZAL4V9mIKnNfXjU/fr9bLSL3lW4GiBdNdGWKwsjpDB7bhKE0LNDHVzuhCPio5jCGxvGSajWhRB1RwEVxAXLAGvsf4FzyP0rWvEQ2JOc1clhNBq/Vn7WWAVjqXcQ6WFAnTQlMlxsT5fOF6W3m93nh5uZJzodWVvO+AEGNink92bWKbjRFK7kj4e+9/+dSGcYs89HvRz375WWX7XYpyvNJYzTvjVOmHwYggrvUDV8YC7aUuov2GQQowhd7MitHW35oY+5iYUmSKnhjcPbdZ75BhQB6ppsm0xsLW+LaJtySvOIO5MeJcYJoj02RVAXuOTHvCOUhxs5iVO0Tt280BieSgrfqje8a7Kvfx5urDu7/+kLEhdC/9EL0O3vgzb27E06gZ7bpgpf+sQx8ntk+L4Lqa6q7F5TMYau96LI3TYys6vqGzwgCHrQ/UNjVxvnfSdv1+3leafaZ7wLDHjHD0ru1w8iD9xjVrA629JtbWXPCeFALzNHE6LeS9UHtjY9/Xjxux4lEKP1a7/W0nd8j618aXGaOaYp1WDVL0cyrGrahlNOeplAalH4sleNrIRR5HcSm5H0KDwLwoS18Qte5o2xHNzKHBybrNPZ0c59lUEaNHTAiJ5fkt0/KEd8JpMnVNq5W8t35ykFHpzgd8DhCFWqqV4Cwn27l6B4AGzKczl6e3+BBZThOny2IGuDwxLXaN2qwnZ8ldtdJhiBOP89EMXx2ivWGufBYtH6HhaEGp7UeL8T6fX2WICDHNhJDwODzQamUv1rRJQyBIoDmHV0/uaYZWe0qogYrHTxEXoakQetc/aZaDVbBu4aafRqmUljujba1bzFNWBOuDY2kf6zsU0kJaLrRS8az4XHBxxoUZCTMSEur7GYTOSCRFoLPlohYbHkboPS7aw0dPSL73gFW0Wh+lvW2w942xmlorzJG/+/03LNPMngvf/fmFDx+vgB1l6HxAtR5NxkyKWw4eIMZI8L9wDxzo58z1iF16C74hqjZoOarcxYL6EQH1JqOl1KNXivVN2XtKIhBSRDDoQsuIFqbQ8JN1Ln97Fp4Wg5K5WIFvSJGn3zyxXN4hKIGCo1GqsEomF2utH+dkZ/9lq7hwpTJPM5fLhRQjeyms2cTNl+eFd795IqZETIFpMUVNjCditJaA15eVD1NiF2HblVqKLQDc0WZBsQZFYKQR2g7vAtxzYIfHGIbypbPyNwxx+DARfDRSQxVtvVGWgiOSo33Xoo3SDGYW6wtmxugcPpqqyOk9dtPSjGUFq2IQQw+q1vrQYZuUtRqrPWZv1jhaOIqGfZwI04ngC76C04wLExImXI8b8b1Swo1eOgLHfb/H67bGjNNwweGjnSTlo9h3a8XEHVqoaiVPPizWRNkHvnn3lsv5mX0vhPCBEO1Uq70f0qRa0ZrRWg0pdR8lYMcO/AyN8Rd3FB8KPTmg6P13AxoeVc5+HCByZ8BK094ZrZn6pJe9mK7Q8lq1ZGrNaC0Ijegh9ENXz6doxbdZyUUJyZNSsMJSgSAej+JrodGQ4nA+krox1g4tW6nMaWKaJ2II4MUaV2nrLUS0t/CvtL7D1l61Poqjj6r+3n5BennH4fX6RnSoGh9Yvh+jfMG85+OUfbUqf8wzhn4g0T2OH16aI6cqXQVV1b5g7Q9FrNWJjEYbownzaHI4Lu8B6j7A3dGoSb0e5U1OBNcM4g4hx+O9+jHcHKHA4+51X6EP0OJRQdGf7Ps2pDkeO6ofcFmBZoeuDn2q74z9lCKn02xrd8+43L1iE8s9j7xzX9tHP5mfGF8cMx6qlC4WHrnDcRnWBcsR8KjY7rRnZd2t7vC2Zj69XnsrhoZSrWfn7cZA/+W6Um8ropWTz8yzY548f/j9hd9/c6Y15dNt57YVXFhY3i6k09lOkk0ToQfM1+sre8744Jnm2ZQVrefzGkTvmWLCOWHfN663K7VV0uIJMeOCSdXKZovsett4fVkpubDvDZGA95Dig2oGTzEl2ANbPNQ82pU3D6zfWDICcljj59HT1xjOe57evGM+X/Ah9k3EhPClKWQOaNd8wBcICLWKndylguLMI6mVl432GkWh9msRcdYawzmas6ZXznlcjMRpwteCLpFAtvheCi73sxO1FwZYcGrIoxcjh36Oi/MjtUSPF3tPU7U+PIreRSFjsxSx4xNypurDYbg6OtWbtbacKdtogBYJEnDR8c37J56eniil8uHllevNCJ0fPt7Iuxlh2Q1BSScPR3O2vza+GKYe4f1jXqj/xqo6xsEwFjuBdCVLtmZMufJ6s/MUnBtUMGyb4HtFcLldqbfVThI+NZbJcVo879/O/P53F9MCfvJ8um64ODE9TYRlsnP/lgspTVZ0Oif2fScEzzybnMtiCWtiPFpNCsK2BcQ3Ss345HHB2rlb/xWTi61bFx5kgya2uwe8b4zDb8zL98UxmrM85HNGeA/3Lt6IER+MPONIT37F4ZxnPp1J89wJiE5/9FOpsghaPK4pvjS2ClWsx5AZY7/eboylNko/HqFp51I7m2lnZDpUPE0sR+l8wMdoFSzJ45q31iXVGGjXe+8Ob3tAeO7NzizfOZzenQi6r9Hj9pohOul9e62QvFTFYYfoDKcy0kqK0mom7wricEGOM0XSZQGf2HNGepH96uHlxebPWnZYCs26B/a18BPjy0+h8sbAaXO9Y7beCyfvK81cdbWdqpTSTyA2kW2rve0ivRK+dSnd3mOX2iyVKeZpY+hBcC+1oTWcLzhfOrNmcYN4T5gm4nSCsBNbtdjDC7H36bE5c8dNH0DSBSEkj1S1mKLHIg5wXkEdPkBMIK6Rpsq09JKd4qj1nsKwsqL7LtxvyLgtBwJQ+vEFPLxERsOux/d+pTGg24/1lLXafe0b7ji2rjY5/q1dlWHyMD1KrO6L2jaVUdNpR7ZZPasPjujtoTgk2M9U1CqC1BGjqaliDDSE4CuhKV7ucwbtId5+0Cvfb+b9Oju09d734wjGj+W4djoTbAenCk2gufE5ehCWqsVKCGnEXqwuRE7LTCmZzXm220buX6PVr1Fc7BzznNAqqGs9N9TIuYuMH6Br0WasqCqvrxsfPt7IpXK9rWzbRlO1/FxXNKwlU66md0naiArBWZXI01PidIqcnt4yP70l10LYBVcFCRMSAwSPnydO77/h/PSWkjPh5RN533BSSZLxci/6hF742QuggxdOce7HeN93MtfVQ01BveImI7HSfGY+Pxkk2W7k/Wb3Yi/s/WSt4QnQXvGud8ZUpHcid96YRr0LikdMRN/5v964K5Kk97Yp+04uRoSEGPpJ0zv7vuKrVcOP1v212FmJhxX09pqjvYWIY46ONBv9n5ZEmCLee85LZJ6DyRJloqWGlkINjbYJwTeuz2ccyrpnKoJfd+ZonSao/SChVtFWaervyhwZ/XYtLKAXOIcYmJYZNHf2tlrKCzFBQTO+YHT/dijB0Zn/Ri2b+c5aDHoLvDk7ns9n9n1iisLL84WX1yulC9JrbWy7Nfn6qfHFTYxj9KhTmtpEWaFvsUOHUJNDda9Ycm9hv25cr7fe8m8n98lE7zqT3KzBlAd88CRvXbpTTCzLiXmJVnaznKAUfNpwMVvrfe/Ni8VIOj+xvHlvrSG8Z982vGZCe8VpxqowrJq/VmvFSFO8N0ID7TK8kS+leww7fhhJpsX0fiKE2SDdGtlvgVYrr9cbta193+4ddkSgn4g10CjmbBndrVV7KdgBrR5Ih681jrzhg/cr1jDM+X78gRvHcucDwo6GYmXPlL03pMJQjq1939tRWG/UKZpCKkUj27z3LNEzRzseoRFQH9EqZC1UMbH1+TQd3+V1tRrH6J1tIVpRNVp3tHwc6glbWoM04/CMzlv+uMVoMa9ql3XaHNsJzENjasjXi97F5v0UMVMPWTy9LCdimsjZNrBpmgne821KOBGqWo1lzr+0AkctF6UPO9Kh7ZMeAx2JVjoI1ANHD2jqnWkYDbY8CL+6WsekbSZvi9NCXC6EKVKZWHPohBBsm+IC+MXhJSB4YwKrWtwRJiIeRyY0cFqMMNJsz7Whrh/nVVtXfhtTPCD4obcFXHNI9V0HPuH9gtbGniIxJWopFBVyHgd5cj8Ep9+Uz0qr6KVU3B/t/puvaYZ9PnvrDO5VNa63tAyOzmLTj/qz1hLmQXq5XHLU3TqQi947A455dc4xB5iCHXK0JCPifAjmGZdkRhUXqJ5WCtk76hSJMbD2owDSupOrtdpIMXA+T8TkOZ+smbG1NBkH48o/37+OqMEgqPMjzHo4f/P+Mh6je+3/dM7OZbQ5kq6/Vug9X1VbJ/M885R4upzZto1tyz3f/gvD1KaNbb0irSBlg37AyygeHme6w6gl7IybNnLeekDrmKLR6akfcipA2zNVM06EZZ56P5jE5Te/4/n3v8MHzyqBbz969l347nvhw4fKNCnTKZLOC4FEycK6Wm/O6fzOVPUUnG5Yc6VMqyuqtrvPo8HuXijr1sug+jmQYLnJlECEXT1bM22sNg/Vqrrz9kpZXyl5x//xP/VOaIV63dmz9VBRuZ/tMCSFlh64tyapwyRULHHeD3v9OkO7eMPyxU6tydgUTFm7RHiehBCFyznw9u1MDKEf92fw+ThnEmtUVXPuiKf08johTUJMlRCF5+fA05MVGj89XzidT92An6ETKXldKfvOett5c7nw8unKbct88/0r19tm50KeIiF63r175s3zhWWZmHo3gXHC2BGZD0bbgQRnJ0+ViFaL5Uab/sEfWJF3N0fp26NATI4otvGUWinNmnu2stLqjiJMIZL8RPKC/oe/57ffvOfjpxf+4//7D3z/w4efnJEv9IyNWjLSCq7mHiMOD9gn+LOYuast1c5fqLXgQyAGU6ikaD1XpbNxItXkbTGQpsS0TMznJ6an9zjnyK2ybdbV+uUmvL627oE8XiJCoFXIuRFdwKeFOE1YA8QMVLRlan2ltdxvfmdwtx3w1hO0KdpLXsI0EeYZcR6vDqcmTnZMOJlBoWyvlO2VvK18+viRj9//BXZB1nygAbw/POOx84p2eaEZ4WjMDBw1oF91dCGC9IcTtUbUHuuQEEyCeJ4cb5ZgVRtdHmaH/dznPu8bedtQrZS8U0v3khFCVEKEN7PjzTkSY+T5MnG+zEcyHulHMgxjXK1caZlnti0Tw8T1thOiY1oCPjien8/M80TqChcrLD6ab97JxG6QI/dt5VV23vyxTvVOesuDdx2oZhA/Frr0JtzaejoHRDwhJmPXnUPfCZfzmZQif/rz97xef7r/7d8gFL/DTlonKoZz75M7gqNBMTcs91T6r0dfXC+GyR2Ajp6kwrIsXJ6fmJaZ+fzMtDyhqqwvn1hvq8nRyqgM6Qee9iPjyr6jYQMRFhUrMhXtqhhFdbd8lubevarLlqrDhWZi32rJfoAmwah8BJWAcxPg8H4h+AWw0hvnbAf1KSCWajUhvWs9lJGuyuFgHAezKgwCp8MZ+wEPAPcXH945ni9n0unM5c0TcVrY951rtA3pdJp4czkRY+D5svB8WUgpEUOw3Gz3Jk6kx4+RfY+d1t8pdQKs9884evx8mlmmZG08ov/sdGrE0aSi3hkZ58TiwWYd6ubJTpgOKbB0mHo6dY/orZ5QHrevwRA/OIhhbY9M9TDax6ZRhz6ge1Tozw/vaQd1+4CJtXbG1SBrjJ6UIufzwvN2+ck5+WJjNMNpXfpT+pe77+aj8T56hwlZK1tT1gqugasWm9TS0GKB/+SFOUVSCrz/7Tf84b/6d0zzwjd/+Pe8+ebfsW073333if/8x++hZXyrzJKYJBCaIrXQ8sb100fYMsvlmcvbb3DB8mh+Sn1HLGizTtmtZOp2o9WKl52gC673Am29Oa8d22Yw08eZOD0jLjDNF9J8QUQo+0fKPrOtr0yXBZm7v4iN5ruCw3W4pGAeeni+u+JDh9a3V3zcc2e//JhS4r/593/HfL7w5pvfM80n1m3l5eWFkjOnZeL5+UQMnufnJ96/f2etNtLEaVlMcaR3TJT3jX3fUG39cNQMWBvDcd7msizM82yx5DwxpdCNMSDOW4WDVoKDmk2fnPMrTgLv3ixduJ94fvtMmpORbr2lvztqbO+dCemOYDCquG603iFNji6EgwDSag5FOhMl4V5hdOQpm33mUSx8lMMpNGtVKdihwSlFlBN/+MNvOT+df3JO/gbPSP/D1mLi8592sKVYXDWyQark7hlHozEn4FXJ2vAiTE4IwSDq6Xzm+d070nzi9PSW6fwWlSs5w8vHK47KJTWC9wSxE/qkVbQU8r7RihLSbEYgdmyAT2ckBKzew4iDmndac4hkRAOuSN8pKirB5FCldPYXXIw4P+N8JKYz0/JkHt5XfKjgFD9FJIjd2e4Z0dHTp6dNmm1aemxeHMWuj/fyoF+/wvDe8e7tM8v5iXfv3zLNC+s6MXnLCy9z4ulpIUbP89OFd88XUoos88zldLaTqxnEG+wPxlhbpva+t8NanbOeorGfzJRSIPguJ/RWieEFajFdqXlGq3X0QVhm66Ezn2bevX1iWiZDXcehSPbHDD52VDbYcAaryuEZ/xnJ85ikPIQaPHjG+8/GAUGMXwsddRm7K84fxzbMc+T5+Yz7pQ9LHfdWu1vvzWiOnjaPDKAxkn3hqR6b/KjIdyJEL6RglfrLErmcI9P/z967xFqWZndev/U99uOcc29kRGZWZvlRfshIgNSIQYtJg2ghD0FMeoQYMEYCCdFjaBgxZ4IAoZYsM0BCaokBEqOetITUNFjQqE0DjW257MpnxL3ntff+XgzW2vucKJerMrIz2gbXl7rKGxHnnrvP3t9jrf/6r/9/6Nm/OLB7fCT2Az5G44Rqt4hrBS+VLjhG46Vu7SlOiN5TY7fJw2OQdcmaE7GFI+oopaRn9UioqFRDroVlUVHbaZ44Xy7U1thXB74jhI6u7xXiFlFycVnsNCiIazgPvhNC762ccctl7JnZJLAbWpvK5lcNfVbPhvcVpjrnOOxHdocdLw57+t2OsY90thiHoWN/UArhbrej71Xv1Ac9zdqW/2ru60Mg2gbjmsffLcYVrVzttNfWrVtExfZcVPRJzWa6TvVsou+Ufxw8wTsaN+JB3YJ52VhQWxvffaB/dyO3eKMZwGN8Y6lyV+tdr/0+9dTks9ZGSQpelpLUXMk5hmEgxoimIMZfbvpZvoky/Lc7GS1nbJuA64o+rTtFU5lNJSmacZ0u5JXw7ZxjHxqH0IjB8eGHez768IFhHPjkl7/Pxz/4JZyPtOJZSmJJM63M+DoRg3AYRj449Piu09qnqCRHHHt8v6MbBgRHSQpONKlI1tpUsdpULVAWUai+BP17PNOy8HyaSSnx9PyGr19/SS2FDz+88D0yfdcTOsfusEOt4q7My5l5uVBbwkXwDfpDoIZO75f5LdK46cZYSaW1qu0700JNWSlnpdCKN5LCdz+6GPmFTz/m8PCCjz/9hGG3J+fMMk+UWohdVMkJr21GPmjO5pyGecYxY22SjsET6HR6bItgVcm7S9tWFkNjA69oFalaNivFpPOlsT/sjG6mp2LwwcoSibys7VZ3CaHV1m5dsT++MNeyxO0ExRhGJWluWoq2Q7U1p3C3TUcXI5SlMF8TORXOpwvTNDH0Hd/75CN2fdQNvSWruWeCq8T4s5/Juy/GdZuwD7mmQZsm6npPND69fdnPar1Gw5AYGjFWumAn42Gg3+3YP+7ZPT4i4pkuiXS1k9EgeI9TRfA+4kw8eCUEBx8IFg4JdjIXJSGsBeK176zVRikmj1EF9WJW45xpKSxL4ny98nx8ppTMMPYs8wGRQikzjYzg7GRMlJpoVOXbesFHITZnYdKa/DdlMFXLP0ozyZwKqdDK6jH/zXRTvu1w3vGw33HY73g47Bl3e2otpNRpzcxsxGXjrf7YuOvE2BqI70K57WV3P3IvY6ibULtbFLYpWQOBAF0MtHF4y6lK55m6j60bwrYgW9tKa3rNbYv078PV+1AW2LRVt2hkjeS4ocX3n6OWRkmq0TpdVG+2Fu3w0L7N25yXtUHebYDBnzreuei/Ik+1rhLnzlDCFaWyS5eGiJIBgg+MnUK+ffT0QelSQ+8Ze/Vj3D88cHj5imG3Y/fwgn7/QGtwOb9hup6Z5wsilWEIdF2g3+/oDw9IiIT9I34YcXEg9CO+65EQdD8wxJfFgWRKTSzLZGyThbwogNOq9aO1yjRdKXmhVe2NDEFM2TsxLyeQxDKfSPMF5x0lzVr4tfehqII1xWhb69SwfKPVVSJfc2xRNxxi1N23FrFWsp+Q23xHQ8C8J9VPMnhHFYCgi9E8D8XdddK3t6F/ty7C9Q3v5slPDLDXv7cFcvP80s18nb4NFNgJQeU1V3ogBqLYEtn2qg3UtLhz/R54azVa3LmSxW8/bPm6ndLrGwqrsoxs0jGtqvWbr/acUoO50HwhXxPLdTEtKLWoK87hRYx4+NPHu3dt1Grd3ppzKU87bIvylr/pJHPSGLvCy30mmchPjJo3Pu57XhxU8/LjTz/l0x/8Mv2449X3f4mHjz4hp8xXn3/N0+svKfMV7wqPj/r6xw9fcfjwJRJ6wv4DXL9HfMD3e1zocSFSqaS8UBFKnbXauMxczkdySqRlYr4cjd51W4wreaHRcJIZR1F5Ry6czplpDnRDr1L1LpCWM3VZqGmhpUzL2fQHZ2SZUApeBSSLEQAAIABJREFUMcn3WwiH6KbkBKpX9n8tSjpfglCS6X69h7F2oK+gShcCjUpn7V6rWp/cnST2kzaPZVuYtz2/bQtuJV7fT0H9sZvgxroY739+3czFad1u1blZf78R71hV6G4ivNwcvez3bLFwA0wcrIqnmR7SbQ2bvF1FVQyNUeSaoE0F2mPpmkAWYha6LLQEMhXaOVPLwvx05SKe0Ed2L/Z0fURqpnOQv8Gm+i3C1LVTw/Abp81IYB9QVpGgNY9seOfpLTT13m0nTRcDfd/RDz3Dfsfu8EC/29PvDnTjDpzyHpd5oiwTTip95+n6SBx64rjDhR4/7pFOF6ML/cZXbU1boGqDVNDO7Hliul7UN3CeuJ6PxhzRxQjqOR/6telWSeTVCVBI+UptnpSu5DTTfFGPQgOYNt1U1TG0OmYFzM1XtmmnPXZo6OUFdcWzvruSVwThnZ/QNxvG1VS92nUTXU8ODAC4hWirAvpbaoD3HSfr3OCWp92nNDee7e04W4Wn1gzv/qDbrs/+bhU01hNRF+HtUt4u8rfb39yditx9vre7aTa9pvv/N41OnL3HSi5TgFy2L8kNUtV5MGfSnFilIL1ttt6e788a746mWuE6F9Wv0QtWHRFtk1FKUamVQqNKQzzEzuOK7oo5ay+jiwPj40cM48ju8XsMj9+jH0ZC/4CWH6ptWip4IzgNS2NHN5q8YIgw7JG4o9bGdUmUOqOJ2xWcoxaVR6i1Mc8T5+cnUpopaSFdz/r+1C1E6fqIc70CQg78bgRpqnpuPMgu7nDS4cTjXUZcpkrAN5BSkVJoc6JcVWaytkJtazfH+oSF5q0+ZqijRvhqfR1CeG/GNyJCCNHQ0Vt9rjbuJqe+VsHeti2y20JrtxNrhfvvNuttom8F+Ntrc7U2OnufVUlhXmZSTlsYe4sujSCAs7KKLu4VQNJlbr/rPgqVG/d2becKwZGdvf+K4JaMSLO0ZvXovAtLNkJLxTkIQW0Ah74jjT2hU3+VtT1w7Q4S1t95r2T3k8c79zN6p8prSy4sc6K5BlnraKt26VtivNJwURibkoyvc2K6ZpzzxOEFH3zya+z2B158+qu8+OTXCF1Hv3tAZAc0ahbqnGg5qxdG19MNO4bHD9l/+AnNR7LfU31Pmiaen7/gcj5vXQiqdanS7KVk0jRxPj6Rl0VPsqJin26TmYRxP+JkT4jK9tg9vLB+u0BzCiTsxg/wfjTlr0bzDfGVgMOXQlkS9XIhPR1vRWVD9eoWXQnNGuvEC2GIptEiijibCsH7GCKObhiJnUYSOFP3rrdwmnLrV11FxFbd2a33cXNnuvlQ1Dtdn7oaz9De8p1IKZvHR6Pk5a33L63inIZ7PujpHbtOPRC9MoC8twbxO1CnrikGjdU1SxeEik57B13nkRZgFjKKrteSyNnTMJmNeotgWL2vG1t9OARRxUBpHA6D1rmtrTalhAtiDcvqoRKDp32DEOdbFv31hhdTNqtWt2gIuKqhFmuEpTWb4HXiySLbLupCRz8+0O8e6MdH4vBA6DpcGEA8qweFllE0+HBO23N816m+pQtU11El0mRhSYXrZABNSqZGl5mmSRfjPHE5PpGXGWhqfYYK0oagHea5E1rpaK7hpGfoIj4GmkSqRCvq6qnorINdxJvQEqwxfEuFmtJ2/RuaKFbqF6wDHlzwuOCMgqUtXX7983sYIrDa+a1BXTXGUd0ICHVbdDfXpUxa0mbqsjp1rb6Z66l4W4zrIjUVgbouRrUU137YeWvLqhbqOu/pa0/otP8Rt9YlHdXr5nkDle5Anbc+ZdvQ1TXy9k5Tjo01s11nUX7Gmtev0cHWiqSZLrTN37F5bYDuurClrzeTm1u1wZnGz88a77wYa9GJtV2QEy0tiMoLBLnJMK3HdHPaSQ/C44sD+xcf4GPHJ7/4KR99X2tchw9e0I2jUtZKYblcSNcLNS/rVKHWTKoJnxeW6cJ8OVLxTO3KQuByOvHVZz/k6euvWQ09MY2TZZkoNWtDdJq1U0G0RUiA2hLLYj7sruI7T4iR0HcsyawC+p5heMC5SBf3OGd1tSqUpE3WOTfTiYFk8o+rZwTm2VHucIUiGhRJbgQqEoWur8jQ6MOPT67vbixL4od/9Md03cCXb06E2JFLUmfgWlRc2FqDVnElmqYYybozUjJpzrVWZwab6q2xtie5zVB168ivlbSozGUpqlU0zxNrR0Q1NHd32BP77u5kVDesvtsRvLZZDX2vWjhOCFF/Z4iertO/k5JxVQE6Jxr9NO+IwVNipDntX3R3GqwqvK1zbuW7iruVOcKg6UPOntKKquhLo6mqC3HQrhKVqBRi9MjPjlLfVTdV5RVarVon9A5bgRrPCyZxvybBFioISFR4ev/iFYdX3yMOIz/4jd/gl/+pX6cfRsbDS4b9A9CYTs9M52fm65m8XBEpNCnmFT/B7LgcnxjejJQmnBJMGY7PT/zw//6HfPnZ5xrXe02eSy3krJMsBMc4BJN/8PSWl12nhevlQi6FVGeyVGXhx57do3JTu92Bw8Mn+NDh6NStqjZSEZa5MM+FtDRShiWrgt11U0EwEacmikQXJQ8umPO7a7AUxDeGnceNDYn3KOZ3O6Zp5nf/j/9LZfKjdqWktDBNZyu8L6T5asSOtTywnoB6wk3TxOVyvZ18Tfs/u35QbR3n6PteO2dEDOLX7ow0zaoqkBbevPma0+lEqYUlJXJR0sHjixf047DJZYhTdfC+3+F9YBx6Hh8fzZ6uY2/E9nHseXyxV4NaB700q/dVOnvuEgP0nUq8BM3bV9Go2qoiqzeum2q5ilYP+tDhRqGUSugD46ybU10R+OCIfTAgWOh9IN7nn3/KePeT0W68YGDbVvi10OctTMxOR7GmTif0Q8fhQdHS/cOB3WGvD6/vcMFv1KSSkpGFb0LJa1FYdWcSeZkpDdLcWHJjvpy5no5cTs84EbqouisaZs1K4u0CdIP6SqKCSfoQCqUkBZdS0t0fMTNMtSoX8ao1GjqoHtXmqFtDcy126jU7LZspBtQNWzaE1yJZVEUt2+1qVjzzuW2veV+j1MLz8QTiqX4C8aQ0c72edDGmmXm6qB4ONyjjfjFer1fOlp+vII+IY9iNDCYOPYyjLigRvGjNra2LcdbF+PT0zPH4TKmVeVnIORO7jtoawzyyWv2JiOaM3aSLcRyordJ3HcsygDTDLBrj2AHNTkK4NU8rWuycAkENuAni2Mxtb2O7a7plculqFxE8UhzRCOMa8ejPiRdDxfU0dVal/FnQ+Dvrpt5Ei28wsHZrKGizJb5tfa2WTNdcbxhHXrz8gGG3Z7/bb4n4PE1Ms3ZMXJ9fcz2+0VB0msyFyqQflhkR4Xo50R97CsKchFQhz1daSUgreBGCfTUHPmoO2nWBvovEqJLsOWlRPqV0505bzLxFWOaZ6/lCyZUQjoT4hPdx1XHWlqH5TJ4vpHlWtW3XU30l07FUE/ld87N1Y2r6oDp0kjaayk3QcE5V9d5nC1VKmR/96AskdFajjXoaLjO1Zq7nI8enr8lp0TpkjKb0tiofNOZp4XqdDHyx8M45XAjEQYvoxXy+NMRNtKyNAUOMPIw7Ssn0fWR6+QGlVj0ZczbwbE/XdyZ4ne3ZQDK7tZRm5vmqWqZ9x+H1jhADr159QKMwjgOHPtLvFBmXqptKQ+hCxI+j5vNOXcrECyF2my/KDfXYjgP9nE4XZnMNCU712Fvb2DeqJnAjRLhtJfz08S10U2/aI1soWtdEt203XktkNpWCgS4xcnh45HuffMKwP/Dw4lFpayJczmdOZ03kp+MbpvMTeZm4Xs6a6OfEMs/k60QthdPTa5w0Ko65eXLzpOsJ0oyvGS+OKI5oWpmIftS+j+yHnhg9S0pcLhdyySwruleayewpAfh6uXJ8eibGmVoitXQ4H0zxTsWYHErTy3lhrkLxO4oTZhm5tp7WGkHCtvEEH/BGkojmGaHOv4lcMt6k4Fdf+Pcx5nnhH/3eHxL6UZu340BrmVa0Qfjp69d8/sd/yDJdGYeBx8NB3Ye7jqHvQYTLZeJ4PFNyMeBFUVDXRfoy0hzqOyUKZszTRLpOdDHy8uNP+PjlK50z9ePN/TnlTC5FQ9OoHOYlLTwdj8zzzDTPvHnzzDzP5FJYlplSCl1Ut7AQPN//hU9xTnh4OMDDgccYkeBxDbypvkvstI+yQXVCtbqqs7xzRc7WzXAts2i1+VZ0lE7LeevrlYXWrFewbRHFN8Hh3p2Bc8dB2jh87dabt9Z6Vvxp0xcR2aQXlTsa8aL6m7Upz2+ZdTEuy0xaFkq6Qd43Rr92V5ecVXQKR9V+Ae2vNC2Wt79u/Em/eehZflBvYdda9G0GMlTR35OWBM2xzAvzPOFdYEkaYtGUORMchg42mgk46zI1U5SVoaRWXSrpKIIYe4nqFNJvtx7A94Wkrs9xXhaKi0iueGfcXeuySbkwXWem6xVBGLqe1hScKVGfr6YLKldZDehx3koY68FydzQoL7jQqgoQD31/+5wm/JRMDBlBQSAn+Dmoa7DlrGu5p9aijmV5BZvU8Wm6TixLsvJJ4WahsD5i01oSv8Gsba3xytsn4v0j2KIUuUWAmldardPmWMMymDsk98fe4SeOb9m1cTOZpNXbL7MPuk1oxZ0J4gixw8fI5XLlRz/8I7p+YHw4s/v6qOBBVrn+VgrL9UyeJ+OHatlB9U8jBDXs9M5b/K8xPCLEoGJAy9hvamXNLL2diRW3WkjzQi3ZQlOtRdKE4Lx27QPV5Dfmy5kjHu8Dy3VmPl1wzpNLUqfmpjC7c2rqc3x+w5JmckkQIOwCUHFRkGB8XV/ReaCgl8PhWsMX1dbpx8iLF3vGvbrwvo/R9wO/+uu/Qej3dA8f4uNArYlartSaEeD1F59RcqbvR8Zxv1HngtcWBO+jPhOcSXcUfPCMu5HHDx4IMTIe9vS7EVqjj4E09Ix9z6effo9f/vQXNKz1Gtat2kG5mmeGlSBSTpyvH7OkxLIsHE9HliWxpIXz5XJX99TV9urVK/b7R7puBALzlKiLOhdf3jxT0oSrFW9lmbkUpqxd7vuXO3YvR62Z94EuasuYYli2sTuvBPoGW+x7V0nUEnKz5mVFZts32Fi/Rc54A1N0ouvppnX+FUVVeyz1XgRxQux6fAicTyeu/8/vIT4w7L5g3L/A+UA/7OmHHQAlTZQ0WQFWQx/xAULEdT3RzFG1xucIztmOHdiNPTWNCs+nmZoLzgtUj3hHLZl5UiGiXArZmDmCyv2vZ2nNiYxQc2U+XxEcXfyarlNAoZSspRLLETBCdWkqOVlqhg7iQ9CHEiuEahFzUQs8BN8q3twrfXM4AuOu58MPD+wftNP+fYxhHPln/tm/hOt3hP2HuDhQ8kxezqp3Wyp//Ae/T1oSw7Bnv3+gi/HGP26NEHpi6NHMMFOaaq3uH/a8/OglXd8x7Hf0u1FLIbuBfJnZjyM/+MEv8hu//Cs66TuV5ahNc8NiyNWtw8IYO8bSWcxcaVmSapTmzHRdOD6fSSnTdT07wyMcjet5YW6V85sjT1++pixXpBRlSdXK8TrxfL0izvHxL3/Ex+EjYhd5CBE/dIjIdvKLFs0trzTUzUS53rJKvLUucXO1/unj3bs2tv/ZEdzu64prGHuPqK60Ja05lVxYyqS0uhZoTXvlNJfS0K3mtHVSANvPNvPlcz5sxeq7TjMrZK9+H82S5nZ3Ofq9hqS3/2PvhFjXxPYRmklT6ucQa8dSMxh1K4KmLJo1dLLTemWDuGA5RLAvaaYCYPie0cbEbNnUOdkRu3BjM72H4Zxjt98jccSPIxIGSnY4yZTiiLHbQCdnm93Kdlk7LDT8d3rar/+JUgbVCyPgo35RGy0ECNmaBSJdVIbNbTE2cJo/N30o26OLlrPVVulrpwDSvOC8J6VMCDO1aP00BDX0cc7Tquqdupr1NJ0TZUlIyZB1MV6vE5fzhHhhmlTRDxFKXoWn2WiCDcxHtK345d0ya9y6KeGbLMD78a3C1BtErOGfd3YyOiXHtoax4hU9XHPDVos2sOYrDaGkTJ4VppY8E2pGnNwWYjNR465HWtRQr4w4rxoqPgQtbeTMUhbNOcudDMMW/6+sH7R+JGsBVxfuasBcVz96u8Nv5wwmKVIXzS+qNS3TtEC+ThZXqQ4qBeJM7ApNKiUmatDu9+Kgiq7LWhyuClE8nd8RnCOOQj9GBquxva/hHeSamU5HChdynlnmEyUvnM+nzXPyLSZNrUbaVquGTVsWAWduXK3RSqJkyLOyiFqtzOczy/lKTQuff/4Zg9xatBB1vlryogCOEcW3vNEH4/A2E4fSxXg6nVmSAm85VaMXB7yxo+brxPH1l5Rl5vL6M06ff0lJk7a3GfD0dL7w5nzRaCkI15qJXeSDD17w+OJwp+MjFuVFNdmVu15O0XL7WiXBBMpWnELcd8xN1QcDNIW3sQv0As61W7UDwGhiiBCcU3pYgzQvXKdJi+XXM7PXXcyXib4lnPOmI7p2dnjiMOBE6P2O6FbQX0+emgvLcuYyL+S0qPmqjXXnBlPzqgYoyWqY4giGtto/b6+V+idWI0K+6f68dXpmlpr0qnymeFXekj4R+0JzBeICfqGh1gcN7W53yeEKNNex7zUf7kbHsFOn3G/iePtthhgbZV4Sz5eZOTdynpinE6Ukjsdn8gqemf9GdW5DkWtVBo7dmI2jq27OVaMbaSSH8T8r59OR6XhmjpE/Ch35cqWBqswbwXoyoriGr90m1dEPo6oNeCH22oc5TxPPzyeWZSHGnrHfKxIdOpyosPXxfOXzP/xj5suJ6elHXL/+nJpmE6BS4O7N5cqb85kmwnOa+ep0JsTIyw+PvHj5whoDAjGqX0vX99oeJc7U71RGpo/qHyIOXLjV1uNa/vgZ4x0X4w1pWhFSndQWM8v6qpUBYHW1FWISrBSyCgWLVUXs4ZWk4NCKgzYH0Vlhns0ER08jK7FYIfb++tqfuN4fq9fd0VpurUO3dbduvmvSvXWEtxuaXNvtz2V1Y5ZGlUJ12e5HxQVN3kVxdY0QrItj8+4DmgEBzqFFY6tHvk9EFbuPaVmYUyGnWRuriwIldVMdaBvqXLY8u24o6ltdHk1De63ftq1vcLUFKKVQREgpbQjpdV5YUqLUwjRPpJxw3tGlpBt1CORS8SGqIlzWdrxpnjmddDEOfcFJIIZG8HkDEksuTNPEdL2yTDPLvCgwWFSourZqYFCmCUzTgrtMhFjohgkfe6tjRrqosh/qyq24QzQZSueEVjzFFqEvzjp81GRnTWN+2vgWYapHnLLnpdWNd6itKiuLHpyP6i67NqlaPdIL9FGbQ5U+p9Q6VxfqclUVNVF3ouaE4itJGsE76Adc1+uDt4kQfGXvOrp9YZ6uLEtmWfTGVnEaLnLjP9QGLZsDllvLDfoZNiVpJ1vJvRpK11pjSYV50U4D7R/Wv08tkVtSgveu4Z1Kb/QeQqcEee8cVRSkqCmTsp6+ejIq4uxcR+wGvO+oTez93/0JfZPRamW5Xjg/n/nsh1/wfJ5Ylonz5UjOC+fnN5zPR9I8c0LV97zz2yJcOyxWRHmV/XPe4T/3tqC85oxBc/zog9ZVW6BKVIJ/q6SWWYp+3tN1YZqueo0ro0Uc3kXEOWorlLrQWuF6vfL09Jp5mdnvDnz48iP6vufjjz/lV3/gGIcdp+MTX37xOZfnJ9LlDel4oRYleCgPtnFJicuim+b8vPCUz3jv+PqYGT4/a4QWHJ1JN8YuECyf19NaN4e+M9WE4Bl2qpy+2x343icd+8P4M5/JO5+MgseJBxdw3nhe68noVL3LiSOEjtCNWgagmAtTJUjDRWVwWM82TootxosCNWbBLc6R/VrL0TYf3w2swAq1EkQ47BXdOp+OPD+faOereQQulDuAB96uK4YQ8E5Fh1dmCRiEbbzEtJ4OtXJdEs9ncxaaK/Oii7GQyCTEN0YRhk5DwBgCvvc0qXg8hUIzNY6UKlTB54YrQudFJSDjqIuxOnJ6z4txOnN88xV//Af/iK9ePzHNE8+nZ5a8QE7UNEOrlLSwXC/bZrVe0n1Hxm0OCCktPD896wlhzKsYIh999DEvX76CoB0wxXe6oNvCUhxLgtNl4Xy+ULJ236ykdIMQWNLM6fxk1L0zr998yTxfefHiBb/wyffZ7Xb8+q+f+fjlhwQHp+MbPv/sM45vXlOXK3W6aHmrFBZbjKlVUtPFWNJMPiaN6Nx5S2miq0RXjZKn5RhE5TXEatd99IQgdF3Hw4sXDMPIB69eIfF7NP8+pBpXUOTHTt110myW1FtyK6oAZsEnwrY4HKtQs4Vja7izkmZMWnEFDuw32I3S+pzafdkNsX63VaOy+mA82nXnNrB5IxHcNcuu4feGFN6Q4a1dyEK0XLQ7IxvaVqVSRJXF2+3it1BeEQy7D3pkQ9E/u+ZY//MbYulNLKtuYfH7GAJgjbW1ZOX75oWcknpwrNdP28LV+7h5uzL7rOu/1QY5W9TkjdjhlDAQzGfThWj8YLnr8nDb51Z2jdLeWuO2GBel4C3LzHW6cr1emecrXey4XFVCf5o01M1rWFxNhKw2tUBvQm5Om99pNLfORqwqqM+pFmFtNG6SKWI8XQNyEMH5rOLITkhR2/C6VHFhoFRPP2aWXMnlOw5Txe5tQZP6UorV2LxNYmOeiEOVGgtSb2CLoACKx9+SftGfDXHAx15DEkvaAQrVugei9sBVNfKM/YgLSiZeViOSMHD44ENc7JU+dz0riycnputZdU2LaqiWrHJ83lyo1AnXbXzCFaDOKXE9X8mlcDrPHJ9ncm2UanbagHQQYsAF6PvAOHh8FILyz3UhZg854kolXoFFKXGDj3QhsAs7HroHHoYHWnNMp8y1nSn5/VDinHPsd3sO+ysvHh/VczIGatPOibWILzRlTsl6b9bNBmO2GP/YFL1FBB9VQUCcY+gH+r6n63o++cVf4tNPP6WLHY+Pjxz2e0ouBNczdlculzNv3nyt5a9FkdLzRUkWMQ5KQ6RRUHJ9xdhMIZJq4c3xyGWaePn6NV+/fk1tcEkF2T0QqmOZrmTXqy6rc3RO52ETsQWJeqJ45ZqmpKBgq4WaLrSkqg0pp638ojmxlXRWtDV2XFNgGAvV7TieCw/zz34m7y7vL+boagibNOu0ttDythhFtzOpiLuBOzdzEnAu4pzyD0McVEjKCX0X6bpIrZXL9WqLUfOTVCrBBUK/ox/3uoNerqSyQBjYf/Ah4+GRkheW85GSZqbrRUGWa6VR9FQzP73kVKcUhBDuTsi7xXi5XEgpcTot22IUF8Gbm1YX8Bai9F3HOHQ43whSkGxuvlNAZkFKI04OvwSC9zzsRgYf2cUdj90Dh+7AvBSezzPTVCjfwGTz2wzndTE+7CdePDxSSyOGQDZVBBE7I+RWyoL7yAcDo/T9NCKJ+m9OPeCc8xz2B3b7PcMw8Okv/hq/9Iu/SAiBvuuIIdJKpvMdqbsQnCe4oP2nc+J4OvF8PBJjx/7B0/lAweR4Eao4fQahkErl6fiMc45Xr1/z1ZvXVIRLyroYXUcKAxklnsdOhaidqA7sWkMNMRCiuk1Nl2fmy4laEtNJmKt6kaa8MM8KPq3SwKClsQrEmHQxDgUJjxzPhet3vRjXI1rlMdfuDeXBrzC/1hj1Qa2aKFtbGBb6aUyzFfNvDTo3ksD64Nf3rBsfNQHKgKmmo6ItPyaXsIV1wsoFFXFbQfp+rGjorY7WUB2UmzxEtd1fL9v85Gnq1+DVMjyENXFXoGa1FVPQSrT2VSw0LeCqEpQ9nugine+IPhJcMGZRs7rZ+7SE06GGtJG+68mlMA7a4K2kottilLtQdONg3lW8nTc9HZS6iNNIox/0ZOy73jR9rHulwcaF1QTfdGRvfNds1gri/Aai1XY7jVbJxTW1WJUncs7M88I0z+rC7Lz2bYaMi52yumKPj7oY3boYTdM3RG/MLyOXrHNo+7qh9zeR5NVFTMHJUiGXRsp1A/5+1ninxeicsN8JC5VyXZB6pRk3ojVPLYG8LjBB3abA6i36IIOPEI1XasZ5gtOew5S0Y9ukEVbDUYpQauPp9ROn81Utxa5XdocDKReOp7PWLi252Lr8S4KqTrwaSnicC4SgDsVORDsOTDKilGIc00ozb8ScFoJzSIQXh8A47BQJjhEXOv1sEVyneWzfC7GB5KY9jYqB0GaVHJEGsal4UR86Xh1e8Pi4o+sGHveP7Po9LU2U5ch0noz9892PVhrTZUJEePXyQ3a7R5aS+GRZtHtdMCtwq1L9xJPxVupZGTpsZSl93t6oiyEEOidMZ+01XVBOcSmZ+XxhmSaej0+8ef2GpzdvuEwTx9OJ8+VMrpVud0BqR25NEXcf9FT00WrYbSVgcrxe+eGPPuPpdKE1T/MRP3b0sUeGA7VqFBBD1BDceeuoYUunSlF/lTmZwZOIqg0Ur4Laoou/pqzWEc4T44j4qMwf15Gq5zIVPvv8NU26n/lM3nExwjiCq5UlzpAnSivkJoDC3i1pyHpjtluPm+UTxLXuYqUFKzqorMNCFUfxnuLMoz0DxVFb5Tg9k4wdkfLMPD2Qcub4fOJyndYDdzvBo3l65JxtIuli9D7YRKp20xuSYa1hr34P66nsneBdYOw7XYCiQITvtIO9ukpzOhHwKtvYqORFWPn0LA5JOpk7O0nH0PNy/wEvP3ggho7DeKDrB5ZroSyV6aLmre9j1FaZrhOC4+XLV8o+WgEXUTTYe7ctxLVTYluMenduixG3YdZ1VUq3SZ2bheq1MJ+P0BqSK2JCV9fLlWWeeT4+8/zmiaenZ3XEOp84Xy8UYMwJX9UAvjnLFb2i+sqaqJsi3ek68ceff8H4fGa3f8HDi49UUbAJ0YrHXhzBIqbgvZZcUO+MUhY7IETLZDUTUOOe6r3WiJ3WGlMT9SsNkTgciN1OQcLayBUQu7K7AAAaqElEQVSuc+GLL9/wDQ7GdwdwohdKEC2+R8wjYXXluSX0rd01GnMjBaytUK2JFb+V1+hQISOxtqVsi7GUGyKWSyaZ1VhaZtIStRM/6wlY70K6JiD1thi1cdjavoxvqZe8yuhvlJr1026f2W9d5uo7oSCTV5EhK4lUY+5XC1fQspv1Ymtjq5IJdCeO3hG9JwavJZBVRt9Qv1rUf+N9jpVz4Nacf30+3KPjVvZx7i5M1X+r9y/cUhUwRSoD7NrGPVk3Z50nhZbVzqDmbBKHSipcw2Lvjd8abtL+b39peKn25isBQe+havVYC9X2bNcqwH26srE37I/NPsqPzQX7fYrMy/Z3ty9nviRxK/W02kCcEtxL+bHf+yfHOy1G7+Hx0TNET1c9aQ7Mi3C8Ktyv8THU6m7xM+bxbp0oORfELQoEtUxtM2sr0YrU5WXGSaQ2mOfMsuiNTmUm12ReHZmynPX+FQhWgliWZDSuFe1rupCTdgM4gRAiLka2zhNj0eRs/nri8L7ZZ9YFI1s4E7abv5ZhmkO7xWmU5ijFXK2yw2XdVKSAr2oe+jAM7HaRYex5OOw47EdaM1bKnDmfrlwuC9eL9ke+jyGiddBcUZ/MVpnmmePlTCpJBaaHDu+cCT91toHd2uOUGmeRxSrcrAm+orEixD6qHowTOu+IUWilKu3NumpqSqRFyytd7NjvD8ShR2LgIS+E2DMeDsRuuFOVgxC0O6PWgZQW5uvFFjVM80KrjtgtlJSsL0a2DaM0rFvH5DKdlcNqNkaVsnh0cTkCSuKX4kCmDWNozRaiCwzjnnH/gpt0ZWMYh61Z/O3N/k+Od84ZDwdHCp5YPWXxnCc1Q12S1u/qDKU4anOUqpbbRD0VndOGVMlWesOMRG0hOqvvzFUFYmuFJVVSMmJymSg1GVCSqOmM894ciiK5ZsoyKeWpNuver+tmR2vQdZF+p/70NJ000FjSYvVMC8tMNiHGuE1EzS8sFCuqU7P2bGJSFOmuhliyQ5LJN5qxaCeBfd/zsB8Yxp7DbmQ3DtrM+zwzTZnrZWK6JqbpPS5GVIi35QaYwPN05fXXXzHNE/3Qsd+pJdww9FB3eO/eKvTP88JkPGNKVjZDazi7r94Lh8cD0e+U3B87dl2k5EZjoaQLkis1FVJSaZUQA8M4EltPHAYtjzmPiwNiC8bPQQXDQiT4HVCZJ09eku0FwjJnWl0YFi1trXq/2MlbS8UeoPYnWpham861Yhxk74OxxeyEt5N1rVtqXVlr3F0/Mu4OGzup1kZv6nbfZLxzmOq8SjJ6j3Zp2PeugvcNS8dMhOkOZWUNTW/uShquGpLaVp2QmxhTq80Ktiu6ucZ9WPeAWX27otS3u4myWtatOVdbnZTr2vF978nAXejafgLqaiY/Wr9esYu7UOb2+ltL19sI7n17kQ+BGKNSqjbFgUZaMtOkfvbzklnMA/C9jC3CWntTK6Uk5vnKNF2pNSEt472n5AVqVhnNetNHXZaZaZptMSq1SGibPbz3nl0dNyV2bZ0KN3DIiCCr/mqxFMfZwgku4JpjUwgWwTttz2oNs4vrEKNiTtcrq1fiegpuEdKae6/lmKpRkbRV1Um2+vL6XJ3TZ0UT3AoKbmi/vvomR6nXFbuO1hrONHuilUrCd22WKiL0g1P3spBpJRE6oR8F1ymi2KJuOGlpzLOGDILe7ApKo6v6XqWoHoOIWIiji2YVHqqtsWTrL6PhXLNn0kw0SnvptMf51kmg5qoGIpTbkmlNwZa8ZFqp9mCtXOE9MXaEcBPc1VCsUquG0iGoI62sVci1ErMmiTbLnQvqteC5CcWLljv6vuPweODFq4PR8SI5wfWS+ezzZ77++sTxNPOjz4+8eb6S89vu0N/VECB4za1qWSi5cD4/8dlnP+T5qIhn8Jo+dF1U8Mo5zWerbhIpJZKRvaVqc62IMA4949DT9x27FyP92NF3kYfDjsdxoKREPl9Yzio1ksrCeZpZUqLScMHjXWAwYrZGOarAFzpPF1UCpB86Hh52xC7w1Rdf8HsVzqcTMXS6kPGKaeRCkWIbtOWQtalMI9B8oHlbjm7Nn4V+HAl9oNVMmo7kuVgbl+WQTtXfg4N+GHjx4pFXH70yFF57cvsu8uLxgaH/jtFUcRA7R8uC95XqinLxBkGKQGi0oJuk86pOrRRS7dAAcFWo1duDrZYUsyp4GKjD1imQysxSFg0Zndcyg3BjAKE6pBiVKpfVru4mPa+nyy28zMkcgkPTMgdi0DRbiLG1DtVKSoqsxqpGmBvdb5vW6nq7JvZOzLzV38ALb/J9cYiM+5HDw0EJ6c5RSmOeCq+/PvPZZ0+cLgtfvTnzdFz0s72HIeu8k0qriVIy1+uJr776nNdvXmuJyPLpLmgLkRNFtWup233KSUNDBch1Y3t8fODh8cC4G/mF8n3iEOn7nt1+5OGwI88LxyHqRpgbuSSu80Q2VsvanDyMgyLnqSBtYakF5wO7TkXB9ocd3/vehwzjQB86vv7iS2rS7g0nQZH6KrRcqaIgUbHP1NYcFxRsaqtek0e8GJmjx/ueWjOXulCWSZN/cduz9l4bGrquY3/Y8+LFo9Y6k3a9dDHy+LD77hcjsLX1iNV2pWnvloYm4FvT7oUCIWpoV4uwqp9ubBzBCqhmYiKw8kdZQwqrZ70FWt39YW0Qxupaelq1WzH6DiTTm77+31Df1VrJXnsLKC2HFdU+vXlFFHJx1t5k9K97vuZ6UsoKFMjKbISqCmPZ5POroa0VPVXmJXO5LpzPM5drYprXMPU91Rnbmia09VJV53QYGMedLUYV+Are0wWtDa8bnZK3CzkEQ07dthi7YSD2g/p4OK8F8FpZcmFeEjllUlG+6KauboX7XAwFrVoqWlKilkZKugmI1QR9CATjJIvNBWfItpMtdoF2M8Ztb80J2ZDbZtcg3Ogn0HDeE6KjFkV2nff4pilG1/WU1hTxXt2nSzFb8UoxtYoikHMif9ctVCKixizREXoPzdMFYYii+idFcElRpNA3fNTQIidHmmVLhFco3fuO6Hu2xbT6UdREq1l37+DxTU/DtQPAG3olosJIq4prbSYQXFbmzHbld7B72xqEmzRqZnOBcmvuZ3Ws1tTOWxFDzZW8EaBDuLXRODCH37YxNiqQpZHRyVCS5hC5Oc5T4WHRyaYhdeWr12f+8I/e8Pt/8CVzqnx9ylyX+t5OxtaacixX6F+E/eHAL/3gV/jwqvRDijoxr3IgGsEowLHCAWs5aWM7OVHWzdATu4gf9lyWwlKgpCOn04WSMs+nK+dcuZZGFoeEQCmZ4+XC8/Hp1qpVC95H+n5P8JFxLzx2HeNuR9d36l6VTOVeVKpfWNUaTAM334p8bdvA64b2F9jkY1y9dWaEvme/H1XJoahAWq0ZHxz7w54lJb5688R8ulDSwvX8zOlNr2DjotFGFyPkibl7Tyej8+62KBG6pieIFGhR7ZjFa62lFlX8FtEi6u0U0kK69z1gkLmzUkRZz5Smv6sZKOLWB66dDSKmCH2Xdm/NsO1uMd7NZ7HiMwitoJbeayJuqJfHgbudsMWaaaU2JaSL0KMLcG3EX8McWJFVzRczzoSW1HkJ75mT/lnlErUU83yc+PKrE5999sxS4ZTQCfyeiv5gjk+r3gzQDyMff/QxaXViKipFKbWqzZ3d32bhXbuPUpxFOSJKiIjadOvjwJyUO5qXhQt6ol6nhTlX5tIoJl9Zges083w8kXPmejmzzAvDMPLyg8o47rXLPkbGYcAHVWirK4prfaPbzmz1zFIKm4C0rGef04ZuMGX3O3lRq32HGBl2OxUxu4zE/kKrqhjXWmaaZ55PZ1rVWvc8XbhejrrpLiozWmIkUCjf9WJUEOpGY1vdPJytMY9qLhVphAAxKKm8VWWi1Cp2MumuJa7QKIDliSsP9M6tSJrmYIKYronfiOZ/crXZLs19p75NFmNeiLu1wIhJsK8HooYtwtooDavAlYO7xQ5rL1/RXyKrKrhdkqGv2XiJpRSma2KeF2jC5bpwNeawClvdocAW62oZhT+B7H6XY5WF8E6ohnZ2fY8rClq0opukaxW3ypBsRXEM4XS2/6ztXxbNbBblps692Xvr/12IhK5QpBK7Spch5USwZuTWsJa4urVYsXJQS1Z1cqo1WNh1Ov0Mrd4QUSWa2LOSDW27A3N0/haT8hSvxftavUUO+mxyKaSsVvGqnqdk8ZUTXUsmLzPzdNXnb6WN6hz3DmQ/bbyj8Y2wLJ6SA40IrsOJNl16aep1HqyfLSjtq1YhJVhmfY453cxzahFFPM0MZu2cLyVT8qoM5+ncyEpbCusOV6G29TWmatAUVheB5syD3e7BSijwBkI5m4jO24JsVn5ZJ4vXUy5UT1cCpTpjdegDTDQwTdfQqbI0YvQ30dzw+TzzfJlJqfD0+sTpOHHYd4x9j2sVHxyxF0IUljxTW2Y1Ze2dmqU4+QZ0/28xxAld58EJuYGrjdAHut1om27ZbNo8qhiiab1mhwiq1hdU58h7s7RDdKNdkwd7LpsKhPN41+j3Rh/LhRQv+HGhGzpeP73hOs3krA2+S1jouo7QdWrRIJXLdKFQiDGQi6qI55IU9d2NG6unNT0Ual2ATCm3HtFa8lbYVw1gm0vWDxtjpO8ju53Kfp4uF56OZ23HuxxZpgslZ04nVUOgVo5vviIvE845YlDXrEKllQHKd91c3DB2iaMRQFT7g6CNta4pilqb2Sc7R6vKvglB6W5paaSlWl0tkU3IO6/2aa1Rs5YlQIg+aje+qDSFd3ZKtXKrHbm1sncTjBWAtS64ho9oodsHtwkPi7v7eFv+Y1xMAR8cIXpDgW8nWMEQOOe2+hvOqE+ibKRpzpwuC8ucef3mytPrM9O15+uPTjzsIzE69o+RXgKlpM3n0gHRm9rBezoYBUV4A/r5KJXoPL2P2gJXb+pvHiFYrODFaxgvAj7QQtCO/qBUQaTRcqIZkLGY05Q+IZVsUexBP5gvhbE5WlCxrmEc6YcBl7yVdYQYIz6q7m0DlmWmUSkloFpngVqLWsGVSC2O4prVHButZUqRbTOtVTWXck52KGRzr27W9Kz1wmn6QD0+SmFaFi7TRE4L5+OJ6+mo+egy6Xu1yvV8oma1Z9jtdkjXKzOr3CRBf9p4p8VYm55wNQt5EVpxumVafaaBbYarf+MaKupobS30al4JGuZtERBNmy02yEsQURFYWdHW9ZXt7ui/bcDbRLO/5q3Wn7u/b9jvNenG7Vc2NQyo9kbN5D1W52HFgnTfL7UiTSl+zmvvZm560qRSmeeFZVpYlkJaMjlXrVuaY1ULOrmj1w6CIUaGLlJxVDwV/x7D1Eqr85YysGW5aOjZNPRafReXrJbfDlVdUzFfFZYWZ32AnYWprSg9Dj1xNgcoJ6pltIariHGTVe4SVA8pBK+15NJpSch7I603bX5eZkrN1NoRot9kI7dQfw37jbK2Gp6mpPQ9XYxmD9HuFyO4WnBeqZLTrEJWtZaN9tcaJisToDmiF1rtbAFqThtC1O9jR9d1jONI3/c/84m822LM8PwENQl58rQccJ0Qh4oEEKutOZOgCFFvuiLJGjKm3Mz5Vhnx87xQizDPwjw7FaKdG4slfYaNrfuy5gKN7YYjIPXm9MHKrOGWb92Ti3GK/JYqa/oCqGeImFVBLQWHwvolZ2pa7MGuIleaM6ZcEFGeYy4a4s65sGQ9GZ9eX3n9PJFT4XycmC4L0TnSXCipQRD6MHAYRpYRXj08cnmRddMrbosw3seotbDMX1npxoAtAiId4Mg5sVwWcq5czxPnZ1Ud8KIFeUFwXYfrVHSsHzr6QYkBQ/QMnbYk1aaAiBdHb/boIrdSlG5rmZwnal3oOsd+39Nax2E/bASMZTEyR848z1e1ixsGRSy7jnm6aqSVVT5kmWcFVqp5YTYlhCzzQq2NkpPZRzRazapMCNoC5aOBN9rbCXA5nyw/bKqb6vY4JwydqsaFoIux7wf7/mB1ysDQj1bP/unjnU/G6Qo1QZ4crThCbeAdvqmRpJO61USdZ6tJitkoh9wIXgEbM2BCCe1qFFOL5l1llatoFaEYAHOTYbw/FbcTy57wj09fzVfs52Wt/dmBYO8hNPxa/G2F1hZWfZhWsp0gq1jW2ia0smOy1e1gXjLTkkm5cjlfuZ4ncqrMUyIthZwKNVWq9TsGF+hDz9BldkPPw24gl0ZQweutj/C7H4VazsbttjC+ebQi7GhV/S9TKlzPZ958/YaUMl48wWlHv+t6fN8r73QcGFOvHS5jR8Bs2ORWt/Wmg6vPDwNjxE7hRGsFH4S+Nw6q1RBTSqoKN2dqzZZTamN512nnfjL08iYpme2rbqdhWhLzrG1puhhXi3d9xqCLUXwk5Mj5dGIYj3oNs3KXac3UyhUsejzsGAdtnN7vDvR9T4hxW4zOqduy+wbWxe+sDtck0qSnyU7lN6TQWDTcaIXWsvn03ZyWXFB/eo1VGuItfLC+xloVjcN5ZerYAlbxpgjVY81HYFzWtdXoNm6c01vu5zbgZu2nrAhlrbAYgqsAqoIOIg3XBNc0GG3NUZwirGoD7i0c1l5MQZAQcD4oMlcdoSps3nWVftB8GenousLDYWD/sGfc7xjHnsE8Roahsd/vORyUdeMXNQJ6XyLGNLacW5rpuNjmp5+u4UXdeEteuFxOzNNyE9USa7DuepxzHB725LxXH5S2o/NNvVDseSjftJCWCQ1P9flNy8KbN695fnrWrpHnJ46no802Xcg5Zy7Xy0b8T4vWZpfFc71etXtkMXkVCz+X+WK+noqEaluVLtDVtMlpEqx2EXGdqx0+dGraM/b0nTaQd0FoQ4fQCKJAYfCO/TjQ9x3BK8G9ix0+BLquJ5o3iffhVgH4KeMdSxuO6nZU76i+UZkprpJbxrVKawmpk55kXtRI0kHsPXFQsKca4bsBKTWWpVGrME+eeQrUKkxXYb6ay2/21Kz1h1aVaLDmphvEfkf+LmbLtvXgGX3OGTReKqSqDJhaKjnrg5KmmZrQCGQivS5Ol2hOu0BcFVrVV7mm9m0iSmgOXq+r9wUXC6FUHtoAXVrrAIg49rue7//S9/nok5eMY8/Lj17xwYsDPp759Cnh40BKleNZW8dC+IN3eUTv9DRbyVsxSP9ruKJKyr4VOtcQ10jXE1/86A85Hc+31KI2lR0JEec9H334io8+fkXXdeSPPyS0VzjntpanNSgFfU65VLLZhn/x+g1PpzPLMvP11685nc8b3U5zv6Yap7VqOc13IJ5lUWkN7zzZFmMt62K8bifZVupYwxdWKU2dH0OnnTneOWI/0plN+cMHrzg8HFTRvIv0MZhy+I2RFGO4kyc1D05bgOIsybqvn/2U8S1PxkZ1B1rraVKpKg9Eawu1BVwr2+kmXi2XYxesLNnQrtuGz+A7RVldCLgYqEXLDd4LrQopOUrW2mar3mpKN3RUd3hdlbVWxEKStRl4zRWdWau5CjU3Vj6aGOVLzVuKIbKeIF6BpuZITcsViGxhKqwyk/eSE1BRbw1xlX5wZDRHijFaLtHz8OKR/cMDw9Az7h8YdntyER4eHliWwrwUmixMU36/J+Oq7LYGkk2QplZ4rmm5qkmjppnz8YnnpyPX68TpbBPdohnvPdRE8DAMPQ9jx/Kg1gTzPJvcYtuafTfrt6KL8aunZ46XC8uy8ObNG86XyxZ6rhbl6/wTH4n9Hheiemeaw3ROy1Zu0JzxekPbbbbc+LhKONHGZBVA2+8GsxHY0w97fAjsdiNjr54uh/2O/TjgnWPodWGqNbrpHd3jErJWurff/N2jqd9gcX/jH/8G1/be3+fP+3ifBf93GX9+7vGfnyv5SePWLncbWzwgP/4vP+Hn34WILCJfAL//bpf48/GPOX6ltfbxd/2mP3+Wf2bjT32e77QYfz5+Pn4+3t94TwnJz8fPx8/Hu46fL8afj5+PPyfjL/RiFJF/S0R+4Ru87j8Wkd/8J3FNf9GGiPy7IvIPROS338N7/6qI/Bvf9fu+r/EXOmcUkb8N/PXW2v/0Z30tf1GHiPwu8JuttT/8Bq8NrbVvIAe8vf6vos/3X/3HuMR/YuPP1cloO9nvisjfFJF/KCK/LSK/KSJ/R0T+TxH5F0TklYj8LRH5X0XkfxSRf85+9m+IyF+/e6+/b+/3q7bz/hci8r+LyP8gIqOI/DXgLwO/LSK/Y3/3H4jI37Wf/c/FsGq7nr9m3/+eiPxHIvI/i8j/JiL/9J/Fvfr/wxCR/wz4deC/F5F//6c8198Skb8D/JaIfCwi/609p78rIn/FXvcv23P8HRH5X0TkAfhPgH/J/u7f+zP7oN90/LhP4Z/lF/CrQAb+ErpR/D3gv0ILNP868LeA/xT4D+31/wrwO/b930B3wfW9/r693/qe/7z9/X8D/Jv2/d8G/vLdz7y6+/63gH/Nvv+bwF+z738P+Hfs+38b+C//rO/b/5e/7H5+9DOe698DRvvzfw38i/b9D4B/YN//d8Bfse8PaA39r8L/2979s7QVhXEc//46ibQZ4guQ4lI6lbSDg4sgCB1rNwdXh1qcOmdsF0cRXHwDZhBFLLRLcSpCmhAHlw52zFQRC218HM6J3FwTehG1J/b5QMi5h+Th3ByenD+54bL9r8+x6OMatxG/dd/NrAkgqQV8MjOT1CQk1jgwB2BmnyWNSSoViFmP5YMYp59pSe+AUaAMtAidnFfLxHpV6Kzc30wxuF+3zOwslmeAp5kLIkqSHgL7wEpce9bM7EcqF00UlWIyZv/afp45Pie09/eA9/2hd9o9MiBmB7hyg3VJI8AqYaQ8llTNxejXxg5pfob3zWmm/ACYNLNfude8l7QDvAT2Jc3eWetuSFJrxoK+APNwuUBvm9lPwnSnEusrwOMCsU6AR7HcTbx2/KZ9fXNNdgUM6te8j8BS90DSs/g8YWZNM/sAfAWe0Nu/yRvGZKwCzyU1CAv0hVi/CZTj1PYNcFQg1gawJqlOGO3WCWvNPUKHurtTpX+/5r0FXsSNnkNgMdYvx423BmH2tAs0gI6kb8OwgfNf/7ThXEqGcWR07l7yZHQuEZ6MziXCk9G5RHgyOpcIT0bnEuHJ6FwiPBmdS8QFujmolDhda90AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [] + } + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "YC3zzpnz8Ukn", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "0126b323-8447-4157-e2ce-572a4f843686" + }, + "source": [ + "total_train/16\n" + ], + "execution_count": 58, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "32.75" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 58 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BVPBWYG7tCA2", + "colab_type": "text" + }, + "source": [ + "## Fit Model" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "H4XdvWA5tCA3", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 454 + }, + "outputId": "9ea1d502-033d-4860-f33a-ea47a27d750f" + }, + "source": [ + "history = model.fit(\n", + " train_data_gen,\n", + " steps_per_epoch=33,\n", + " epochs=epochs,\n", + " validation_data=val_data_gen,\n", + " validation_steps=11\n", + ")" + ], + "execution_count": 60, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Epoch 1/50\n" + ], + "name": "stdout" + }, + { + "output_type": "error", + "ename": "InvalidArgumentError", + "evalue": "ignored", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mInvalidArgumentError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m\u001b[0m in \u001b[0;36m\u001b[0;34m()\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mepochs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mepochs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mvalidation_data\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mval_data_gen\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mvalidation_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m11\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m )\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36m_method_wrapper\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 106\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_method_wrapper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 107\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_in_multi_worker_mode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 108\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 109\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 110\u001b[0m \u001b[0;31m# Running inside `run_distribute_coordinator` already.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 1096\u001b[0m batch_size=batch_size):\n\u001b[1;32m 1097\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_train_batch_begin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1098\u001b[0;31m \u001b[0mtmp_logs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0miterator\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 1099\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mdata_handler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshould_sync\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1100\u001b[0m \u001b[0mcontext\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masync_wait\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 778\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 779\u001b[0m \u001b[0mcompiler\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"nonXla\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 780\u001b[0;31m \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 781\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 782\u001b[0m \u001b[0mnew_tracing_count\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_tracing_count\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py\u001b[0m in \u001b[0;36m_call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 805\u001b[0m \u001b[0;31m# In this case we have created variables on the first call, so we run the\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 806\u001b[0m \u001b[0;31m# defunned version which is guaranteed to never create variables.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 807\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateless_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwds\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# pylint: disable=not-callable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 808\u001b[0m \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_stateful_fn\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 809\u001b[0m \u001b[0;31m# Release the lock early so that multiple threads can perform the call\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2827\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_lock\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2828\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_define_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2829\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mgraph_function\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_filtered_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# pylint: disable=protected-access\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2830\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2831\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_filtered_call\u001b[0;34m(self, args, kwargs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1846\u001b[0m resource_variable_ops.BaseResourceVariable))],\n\u001b[1;32m 1847\u001b[0m \u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1848\u001b[0;31m cancellation_manager=cancellation_manager)\n\u001b[0m\u001b[1;32m 1849\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1850\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_call_flat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcaptured_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcancellation_manager\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36m_call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1922\u001b[0m \u001b[0;31m# No tape is watching; skip to running the function.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 1923\u001b[0m return self._build_call_outputs(self._inference_function.call(\n\u001b[0;32m-> 1924\u001b[0;31m ctx, args, cancellation_manager=cancellation_manager))\n\u001b[0m\u001b[1;32m 1925\u001b[0m forward_backward = self._select_forward_and_backward_functions(\n\u001b[1;32m 1926\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py\u001b[0m in \u001b[0;36mcall\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 548\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 549\u001b[0m \u001b[0mattrs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattrs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 550\u001b[0;31m ctx=ctx)\n\u001b[0m\u001b[1;32m 551\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 552\u001b[0m outputs = execute.execute_with_cancellation(\n", + "\u001b[0;32m/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py\u001b[0m in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 58\u001b[0m \u001b[0mctx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mensure_initialized\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 59\u001b[0m tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,\n\u001b[0;32m---> 60\u001b[0;31m inputs, attrs, num_outputs)\n\u001b[0m\u001b[1;32m 61\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_NotOkStatusException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 62\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mInvalidArgumentError\u001b[0m: logits and labels must have the same first dimension, got logits shape [2704,10] and labels shape [16]\n\t [[node sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at :6) ]] [Op:__inference_train_function_1718]\n\nFunction call stack:\ntrain_function\n" + ] + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UPzsgS94tCA5", + "colab_type": "text" + }, + "source": [ + "# Custom CNN Model\n", + "\n", + "In this step, write and train your own convolutional neural network using Keras. You can use any architecture that suits you as long as it has at least one convolutional and one pooling layer at the beginning of the network - you can add more if you want. " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "hnbJJie3tCA5", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 420 + }, + "outputId": "4a26f9cd-3329-47be-d81f-015ba9b2774b" + }, + "source": [ + "# Define the Model\n", + "model = Sequential()\n", + "model.add(Conv2D(32, (3,3), activation='relu', input_shape=(32,32,3)))\n", + "model.add(MaxPooling2D((2,2)))\n", + "model.add(Conv2D(64, (3,3), activation='relu'))\n", + "model.add(MaxPooling2D((2,2)))\n", + "model.add(Conv2D(64, (3,3), activation='relu'))\n", + "model.add(Flatten())\n", + "model.add(Dense(64, activation='relu'))\n", + "model.add(Dense(10, activation='softmax'))\n", + "\n", + "model.summary()" + ], + "execution_count": 36, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Model: \"sequential_2\"\n", + "_________________________________________________________________\n", + "Layer (type) Output Shape Param # \n", + "=================================================================\n", + "conv2d_4 (Conv2D) (None, 30, 30, 32) 896 \n", + "_________________________________________________________________\n", + "max_pooling2d_2 (MaxPooling2 (None, 15, 15, 32) 0 \n", + "_________________________________________________________________\n", + "conv2d_5 (Conv2D) (None, 13, 13, 64) 18496 \n", + "_________________________________________________________________\n", + "max_pooling2d_3 (MaxPooling2 (None, 6, 6, 64) 0 \n", + "_________________________________________________________________\n", + "conv2d_6 (Conv2D) (None, 4, 4, 64) 36928 \n", + "_________________________________________________________________\n", + "flatten_1 (Flatten) (None, 1024) 0 \n", + "_________________________________________________________________\n", + "dense_2 (Dense) (None, 64) 65600 \n", + "_________________________________________________________________\n", + "dense_3 (Dense) (None, 10) 650 \n", + "=================================================================\n", + "Total params: 122,570\n", + "Trainable params: 122,570\n", + "Non-trainable params: 0\n", + "_________________________________________________________________\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "1P_mRtoutCA9", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Compile Model\n", + "model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])" + ], + "execution_count": 37, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "CwM4GsaetCA_", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 370 + }, + "outputId": "6293708d-5572-415b-87f9-037babffcd87" + }, + "source": [ + "# Fit Model\n", + "model.fit(train_dir, train_labels, epochs=10, validation_data=(validation_dir, validation_labels))\n" + ], + "execution_count": 61, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Epoch 1/10\n", + "1563/1563 [==============================] - 74s 47ms/step - loss: 1.4923 - accuracy: 0.4544 - val_loss: 1.1985 - val_accuracy: 0.5718\n", + "Epoch 2/10\n", + "1563/1563 [==============================] - 71s 46ms/step - loss: 1.1246 - accuracy: 0.6027 - val_loss: 1.0444 - val_accuracy: 0.6290\n", + "Epoch 3/10\n", + "1563/1563 [==============================] - 72s 46ms/step - loss: 0.9795 - accuracy: 0.6552 - val_loss: 0.9372 - val_accuracy: 0.6745\n", + "Epoch 4/10\n", + "1563/1563 [==============================] - 78s 50ms/step - loss: 0.8857 - accuracy: 0.6894 - val_loss: 0.9311 - val_accuracy: 0.6735\n", + "Epoch 5/10\n", + "1563/1563 [==============================] - 71s 46ms/step - loss: 0.8050 - accuracy: 0.7179 - val_loss: 0.9470 - val_accuracy: 0.6774\n", + "Epoch 6/10\n", + "1563/1563 [==============================] - 71s 45ms/step - loss: 0.7478 - accuracy: 0.7417 - val_loss: 0.8755 - val_accuracy: 0.7037\n", + "Epoch 7/10\n", + "1563/1563 [==============================] - 74s 47ms/step - loss: 0.7021 - accuracy: 0.7545 - val_loss: 0.8638 - val_accuracy: 0.7088\n", + "Epoch 8/10\n", + "1563/1563 [==============================] - 75s 48ms/step - loss: 0.6513 - accuracy: 0.7730 - val_loss: 0.8527 - val_accuracy: 0.7162\n", + "Epoch 9/10\n", + "1563/1563 [==============================] - 74s 47ms/step - loss: 0.6070 - accuracy: 0.7866 - val_loss: 0.8976 - val_accuracy: 0.7047\n", + "Epoch 10/10\n", + "1563/1563 [==============================] - 73s 47ms/step - loss: 0.5678 - accuracy: 0.7999 - val_loss: 0.9456 - val_accuracy: 0.6913\n" + ], + "name": "stdout" + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 61 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ceT3GrIq3r9I", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "fe90ae34-fe42-4c4d-b45b-7b0f2bdc7064" + }, + "source": [ + "# Evaluate Model\n", + "\n", + "validation_loss, validation_acc = model.evaluate(validation_dir, validation_labels, verbose=2)" + ], + "execution_count": 63, + "outputs": [ + { + "output_type": "stream", + "text": [ + "313/313 - 4s - loss: 0.9456 - accuracy: 0.6913\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FNTHjUddtCBB", + "colab_type": "text" + }, + "source": [ + "# Custom CNN Model with Image Manipulations\n", + "\n", + "To simulate an increase in a sample of image, you can apply image manipulation techniques: cropping, rotation, stretching, etc. Luckily Keras has some handy functions for us to apply these techniques to our mountain and forest example. Simply, you should be able to modify our image generator for the problem. Check out these resources to help you get started: \n", + "\n", + "1. [Keras `ImageGenerator` Class](https://keras.io/preprocessing/image/#imagedatagenerator-class)\n", + "2. [Building a powerful image classifier with very little data](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html)\n", + " " + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "XKioBv3WtCBB", + "colab_type": "code", + "colab": {} + }, + "source": [ + "" + ], + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "uT3UV3gap9H6" + }, + "source": [ + "# Resources and Stretch Goals\n", + "\n", + "Stretch goals\n", + "- Enhance your code to use classes/functions and accept terms to search and classes to look for in recognizing the downloaded images (e.g. download images of parties, recognize all that contain balloons)\n", + "- Check out [other available pretrained networks](https://tfhub.dev), try some and compare\n", + "- Image recognition/classification is somewhat solved, but *relationships* between entities and describing an image is not - check out some of the extended resources (e.g. [Visual Genome](https://visualgenome.org/)) on the topic\n", + "- Transfer learning - using images you source yourself, [retrain a classifier](https://www.tensorflow.org/hub/tutorials/image_retraining) with a new category\n", + "- (Not CNN related) Use [piexif](https://pypi.org/project/piexif/) to check out the metadata of images passed in to your system - see if they're from a national park! (Note - many images lack GPS metadata, so this won't work in most cases, but still cool)\n", + "\n", + "Resources\n", + "- [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385) - influential paper (introduced ResNet)\n", + "- [YOLO: Real-Time Object Detection](https://pjreddie.com/darknet/yolo/) - an influential convolution based object detection system, focused on inference speed (for applications to e.g. self driving vehicles)\n", + "- [R-CNN, Fast R-CNN, Faster R-CNN, YOLO](https://towardsdatascience.com/r-cnn-fast-r-cnn-faster-r-cnn-yolo-object-detection-algorithms-36d53571365e) - comparison of object detection systems\n", + "- [Common Objects in Context](http://cocodataset.org/) - a large-scale object detection, segmentation, and captioning dataset\n", + "- [Visual Genome](https://visualgenome.org/) - a dataset, a knowledge base, an ongoing effort to connect structured image concepts to language" + ] + } + ] +} \ No newline at end of file diff --git a/LS_DS_431_RNN_and_LSTM_Assignment.ipynb b/LS_DS_431_RNN_and_LSTM_Assignment.ipynb new file mode 100644 index 00000000..46fb95dc --- /dev/null +++ b/LS_DS_431_RNN_and_LSTM_Assignment.ipynb @@ -0,0 +1,1284 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "nteract": { + "version": "0.23.3" + }, + "colab": { + "name": "LS_DS_431_RNN_and_LSTM_Assignment.ipynb", + "provenance": [], + "include_colab_link": true + }, + "accelerator": "GPU" + }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "view-in-github", + "colab_type": "text" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZU45AwiEpPjg", + "colab_type": "text" + }, + "source": [ + "\n", + "

\n", + "

\n", + "\n", + "## *Data Science Unit 4 Sprint 3 Assignment 1*\n", + "\n", + "# Recurrent Neural Networks and Long Short Term Memory (LSTM)\n", + "\n", + "![Monkey at a typewriter](https://upload.wikimedia.org/wikipedia/commons/thumb/3/3c/Chimpanzee_seated_at_typewriter.jpg/603px-Chimpanzee_seated_at_typewriter.jpg)\n", + "\n", + "It is said that [infinite monkeys typing for an infinite amount of time](https://en.wikipedia.org/wiki/Infinite_monkey_theorem) will eventually type, among other things, the complete works of Wiliam Shakespeare. Let's see if we can get there a bit faster, with the power of Recurrent Neural Networks and LSTM.\n", + "\n", + "This text file contains the complete works of Shakespeare: https://www.gutenberg.org/files/100/100-0.txt\n", + "\n", + "Use it as training data for an RNN - you can keep it simple and train character level, and that is suggested as an initial approach.\n", + "\n", + "Then, use that trained RNN to generate Shakespearean-ish text. Your goal - a function that can take, as an argument, the size of text (e.g. number of characters or lines) to generate, and returns generated text of that size.\n", + "\n", + "Note - Shakespeare wrote an awful lot. It's OK, especially initially, to sample/use smaller data and parameters, so you can have a tighter feedback loop when you're trying to get things running. Then, once you've got a proof of concept - start pushing it more!" + ] + }, + { + "cell_type": "code", + "metadata": { + "execution": { + "iopub.status.busy": "2020-06-15T18:18:20.442Z", + "iopub.execute_input": "2020-06-15T18:18:20.453Z", + "iopub.status.idle": "2020-06-15T18:18:20.513Z", + "shell.execute_reply": "2020-06-15T18:18:20.523Z" + }, + "id": "dRrt23sZpPjg", + "colab_type": "code", + "colab": {} + }, + "source": [ + "import requests\n", + "import pandas as pd" + ], + "execution_count": 1, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "execution": { + "iopub.status.busy": "2020-06-15T18:25:49.778Z", + "iopub.execute_input": "2020-06-15T18:25:49.781Z", + "iopub.status.idle": "2020-06-15T18:25:51.467Z", + "shell.execute_reply": "2020-06-15T18:25:51.469Z" + }, + "id": "CBTyymNSpPjj", + "colab_type": "code", + "colab": {} + }, + "source": [ + "url = \"https://www.gutenberg.org/files/100/100-0.txt\"\n", + "\n", + "r = requests.get(url)\n", + "r.encoding = r.apparent_encoding\n", + "data = r.text\n", + "data = data.split('\\r\\n')\n", + "toc = [l.strip() for l in data[44:130:2]]\n", + "# Skip the Table of Contents\n", + "data = data[135:]\n", + "\n", + "# Fixing Titles\n", + "toc[9] = 'THE LIFE OF KING HENRY V'\n", + "toc[18] = 'MACBETH'\n", + "toc[24] = 'OTHELLO, THE MOOR OF VENICE'\n", + "toc[34] = 'TWELFTH NIGHT: OR, WHAT YOU WILL'\n", + "\n", + "locations = {id_:{'title':title, 'start':-99} for id_,title in enumerate(toc)}\n", + "\n", + "# Start \n", + "for e,i in enumerate(data):\n", + " for t,title in enumerate(toc):\n", + " if title in i:\n", + " locations[t].update({'start':e})\n", + " \n", + "\n", + "df_toc = pd.DataFrame.from_dict(locations, orient='index')\n", + "df_toc['end'] = df_toc['start'].shift(-1).apply(lambda x: x-1)\n", + "df_toc.loc[42, 'end'] = len(data)\n", + "df_toc['end'] = df_toc['end'].astype('int')\n", + "\n", + "df_toc['text'] = df_toc.apply(lambda x: '\\r\\n'.join(data[ x['start'] : int(x['end']) ]), axis=1)" + ], + "execution_count": 2, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "collapsed": true, + "jupyter": { + "source_hidden": false, + "outputs_hidden": false + }, + "nteract": { + "transient": { + "deleting": false + } + }, + "execution": { + "iopub.status.busy": "2020-06-15T18:26:12.630Z", + "iopub.execute_input": "2020-06-15T18:26:12.637Z", + "iopub.status.idle": "2020-06-15T18:26:12.643Z", + "shell.execute_reply": "2020-06-15T18:26:12.647Z" + }, + "id": "-K1rZBfQpPjl", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 195 + }, + "outputId": "7f13b527-d011-46f0-80c7-6585f0fd5e5e" + }, + "source": [ + "#Shakespeare Data Parsed by Play\n", + "df_toc.head()" + ], + "execution_count": 3, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
titlestartendtext
0THE TRAGEDY OF ANTONY AND CLEOPATRA-9914379
1AS YOU LIKE IT1438017171AS YOU LIKE IT\\r\\n\\r\\n\\r\\nDRAMATIS PERSONAE.\\r...
2THE COMEDY OF ERRORS1717220372THE COMEDY OF ERRORS\\r\\n\\r\\n\\r\\n\\r\\nContents\\r...
3THE TRAGEDY OF CORIOLANUS2037330346THE TRAGEDY OF CORIOLANUS\\r\\n\\r\\nDramatis Pers...
4CYMBELINE3034730364CYMBELINE.\\r\\nLaud we the gods;\\r\\nAnd let our...
\n", + "
" + ], + "text/plain": [ + " title ... text\n", + "0 THE TRAGEDY OF ANTONY AND CLEOPATRA ... \n", + "1 AS YOU LIKE IT ... AS YOU LIKE IT\\r\\n\\r\\n\\r\\nDRAMATIS PERSONAE.\\r...\n", + "2 THE COMEDY OF ERRORS ... THE COMEDY OF ERRORS\\r\\n\\r\\n\\r\\n\\r\\nContents\\r...\n", + "3 THE TRAGEDY OF CORIOLANUS ... THE TRAGEDY OF CORIOLANUS\\r\\n\\r\\nDramatis Pers...\n", + "4 CYMBELINE ... CYMBELINE.\\r\\nLaud we the gods;\\r\\nAnd let our...\n", + "\n", + "[5 rows x 4 columns]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 3 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "SqvPwfdFhNtl", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "945616e9-df99-49a4-e10c-b4f67daf4f75" + }, + "source": [ + "data = df_toc['text'].values\n", + "len(data)" + ], + "execution_count": 4, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "43" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 4 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "DedaROkRqRls", + "colab_type": "code", + "colab": {} + }, + "source": [ + "data=data[1]" + ], + "execution_count": 5, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "z3EyNF2ah7I9", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Encode Data as Chars\n", + "\n", + "# Gather all text \n", + "# Why? 1. See all possible characters 2. For training / splitting later\n", + "text = \" \".join(data)\n", + "\n", + "# Unique Characters\n", + "chars = list(set(text))\n", + "\n", + "# Lookup Tables\n", + "char_int = {c:i for i, c in enumerate(chars)} \n", + "int_char = {i:c for i, c in enumerate(chars)} " + ], + "execution_count": 6, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "46Ve7isxh7Vl", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "cb54dade-bc56-4a32-bfeb-dddc3331983c" + }, + "source": [ + "char_int['S']" + ], + "execution_count": 7, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "55" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 7 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "LrCKSdy3h7hT", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 35 + }, + "outputId": "88f6ab10-f559-40d4-cea8-92412360b8ca" + }, + "source": [ + "int_char[2]" + ], + "execution_count": 8, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + }, + "text/plain": [ + "'q'" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 8 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "AHDb5UZKjhXW", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "3ff05e50-129a-4a30-9619-291690f20c61" + }, + "source": [ + "len(chars)" + ], + "execution_count": 9, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "66" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 9 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "JFSdtnvImT1S", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "outputId": "6b6eab94-433e-46c4-c19e-921d3835532c" + }, + "source": [ + "chars" + ], + "execution_count": 10, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "['i',\n", + " 'z',\n", + " 'q',\n", + " '&',\n", + " 'F',\n", + " \"'\",\n", + " 'B',\n", + " 'D',\n", + " 'k',\n", + " 'h',\n", + " 'C',\n", + " 'o',\n", + " 't',\n", + " 'O',\n", + " ']',\n", + " 'u',\n", + " '\"',\n", + " ' ',\n", + " 'd',\n", + " 'f',\n", + " 'P',\n", + " '\\r',\n", + " ',',\n", + " ';',\n", + " 'L',\n", + " 'Y',\n", + " 'E',\n", + " '[',\n", + " '!',\n", + " 'A',\n", + " 'l',\n", + " 'G',\n", + " 'e',\n", + " 'g',\n", + " 'y',\n", + " 'K',\n", + " 'X',\n", + " 'N',\n", + " 's',\n", + " 'Q',\n", + " '?',\n", + " ':',\n", + " 'U',\n", + " 'b',\n", + " 'r',\n", + " 'W',\n", + " 'R',\n", + " 'I',\n", + " '\\n',\n", + " 'v',\n", + " 'H',\n", + " 'T',\n", + " 'x',\n", + " 'a',\n", + " 'p',\n", + " 'S',\n", + " 'w',\n", + " 'j',\n", + " 'c',\n", + " 'V',\n", + " '.',\n", + " '-',\n", + " 'n',\n", + " 'M',\n", + " 'J',\n", + " 'm']" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 10 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ylSEJcTlmj9T", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "outputId": "660f2bdf-e0b0-4030-b130-02d72183f337" + }, + "source": [ + "char_int" + ], + "execution_count": 11, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "{'\\n': 48,\n", + " '\\r': 21,\n", + " ' ': 17,\n", + " '!': 28,\n", + " '\"': 16,\n", + " '&': 3,\n", + " \"'\": 5,\n", + " ',': 22,\n", + " '-': 61,\n", + " '.': 60,\n", + " ':': 41,\n", + " ';': 23,\n", + " '?': 40,\n", + " 'A': 29,\n", + " 'B': 6,\n", + " 'C': 10,\n", + " 'D': 7,\n", + " 'E': 26,\n", + " 'F': 4,\n", + " 'G': 31,\n", + " 'H': 50,\n", + " 'I': 47,\n", + " 'J': 64,\n", + " 'K': 35,\n", + " 'L': 24,\n", + " 'M': 63,\n", + " 'N': 37,\n", + " 'O': 13,\n", + " 'P': 20,\n", + " 'Q': 39,\n", + " 'R': 46,\n", + " 'S': 55,\n", + " 'T': 51,\n", + " 'U': 42,\n", + " 'V': 59,\n", + " 'W': 45,\n", + " 'X': 36,\n", + " 'Y': 25,\n", + " '[': 27,\n", + " ']': 14,\n", + " 'a': 53,\n", + " 'b': 43,\n", + " 'c': 58,\n", + " 'd': 18,\n", + " 'e': 32,\n", + " 'f': 19,\n", + " 'g': 33,\n", + " 'h': 9,\n", + " 'i': 0,\n", + " 'j': 57,\n", + " 'k': 8,\n", + " 'l': 30,\n", + " 'm': 65,\n", + " 'n': 62,\n", + " 'o': 11,\n", + " 'p': 54,\n", + " 'q': 2,\n", + " 'r': 44,\n", + " 's': 38,\n", + " 't': 12,\n", + " 'u': 15,\n", + " 'v': 49,\n", + " 'w': 56,\n", + " 'x': 52,\n", + " 'y': 34,\n", + " 'z': 1}" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 11 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "hoYRqK1wmvY2", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "outputId": "947a9b63-0739-4bac-dd66-8589d558762d" + }, + "source": [ + "int_char" + ], + "execution_count": 12, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "{0: 'i',\n", + " 1: 'z',\n", + " 2: 'q',\n", + " 3: '&',\n", + " 4: 'F',\n", + " 5: \"'\",\n", + " 6: 'B',\n", + " 7: 'D',\n", + " 8: 'k',\n", + " 9: 'h',\n", + " 10: 'C',\n", + " 11: 'o',\n", + " 12: 't',\n", + " 13: 'O',\n", + " 14: ']',\n", + " 15: 'u',\n", + " 16: '\"',\n", + " 17: ' ',\n", + " 18: 'd',\n", + " 19: 'f',\n", + " 20: 'P',\n", + " 21: '\\r',\n", + " 22: ',',\n", + " 23: ';',\n", + " 24: 'L',\n", + " 25: 'Y',\n", + " 26: 'E',\n", + " 27: '[',\n", + " 28: '!',\n", + " 29: 'A',\n", + " 30: 'l',\n", + " 31: 'G',\n", + " 32: 'e',\n", + " 33: 'g',\n", + " 34: 'y',\n", + " 35: 'K',\n", + " 36: 'X',\n", + " 37: 'N',\n", + " 38: 's',\n", + " 39: 'Q',\n", + " 40: '?',\n", + " 41: ':',\n", + " 42: 'U',\n", + " 43: 'b',\n", + " 44: 'r',\n", + " 45: 'W',\n", + " 46: 'R',\n", + " 47: 'I',\n", + " 48: '\\n',\n", + " 49: 'v',\n", + " 50: 'H',\n", + " 51: 'T',\n", + " 52: 'x',\n", + " 53: 'a',\n", + " 54: 'p',\n", + " 55: 'S',\n", + " 56: 'w',\n", + " 57: 'j',\n", + " 58: 'c',\n", + " 59: 'V',\n", + " 60: '.',\n", + " 61: '-',\n", + " 62: 'n',\n", + " 63: 'M',\n", + " 64: 'J',\n", + " 65: 'm'}" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 12 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "HGGu5vAzjiS9", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "c7a43fdf-dfb2-4050-fbbf-016416fee377" + }, + "source": [ + "# Create the sequence data\n", + "\n", + "maxlen = 40\n", + "step = 5\n", + "\n", + "encoded = [char_int[c] for c in text]\n", + "\n", + "sequences = [] # Each element is 40 chars long\n", + "next_char = [] # One element for each sequence\n", + "\n", + "for i in range(0, len(encoded) - maxlen, step):\n", + " sequences.append(encoded[i : i + maxlen])\n", + " next_char.append(encoded[i + maxlen])\n", + " \n", + "print('sequences: ', len(sequences))" + ], + "execution_count": 13, + "outputs": [ + { + "output_type": "stream", + "text": [ + "sequences: 54663\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "ABVeXjAdjiV2", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "720e89ff-7949-4ec0-a6a4-3a59296d6227" + }, + "source": [ + "len(text)" + ], + "execution_count": 14, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "273355" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 14 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "pabKbsGljiYr", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 689 + }, + "outputId": "de957bb2-bf20-4020-fb54-65b9e7e8413e" + }, + "source": [ + "sequences[0]" + ], + "execution_count": 15, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "[29,\n", + " 17,\n", + " 55,\n", + " 17,\n", + " 17,\n", + " 17,\n", + " 25,\n", + " 17,\n", + " 13,\n", + " 17,\n", + " 42,\n", + " 17,\n", + " 17,\n", + " 17,\n", + " 24,\n", + " 17,\n", + " 47,\n", + " 17,\n", + " 35,\n", + " 17,\n", + " 26,\n", + " 17,\n", + " 17,\n", + " 17,\n", + " 47,\n", + " 17,\n", + " 51,\n", + " 17,\n", + " 21,\n", + " 17,\n", + " 48,\n", + " 17,\n", + " 21,\n", + " 17,\n", + " 48,\n", + " 17,\n", + " 21,\n", + " 17,\n", + " 48,\n", + " 17]" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 15 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "EpnzumtwlHpT", + "colab_type": "code", + "colab": {} + }, + "source": [ + "import numpy as np" + ], + "execution_count": 16, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "_fDwj2dDjicH", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# Create x & y\n", + "\n", + "# Padding!\n", + "\n", + "\n", + "x = np.zeros((len(sequences), maxlen, len(chars)), dtype=np.bool)\n", + "y = np.zeros((len(sequences),len(chars)), dtype=np.bool)\n", + "\n", + "for i, sequence in enumerate(sequences):\n", + " for t, char in enumerate(sequence):\n", + " x[i,t,char] = 1\n", + " \n", + " y[i, next_char[i]] = 1" + ], + "execution_count": 17, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "FP7w4L5Ul_1I", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "cfb1291c-c80e-43b4-89af-91ed711ffa5f" + }, + "source": [ + "x.shape" + ], + "execution_count": 18, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(54663, 40, 66)" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 18 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "btMfxnb7r9zH", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "outputId": "a8adbbb9-3280-466e-fda2-ee8e166f3725" + }, + "source": [ + "y.shape" + ], + "execution_count": 19, + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "(54663, 66)" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 19 + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "GfcxrqMKsaEu", + "colab_type": "code", + "colab": {} + }, + "source": [ + "from tensorflow.keras.callbacks import LambdaCallback\n", + "from tensorflow.keras.models import Sequential\n", + "from tensorflow.keras.layers import Dense, LSTM\n", + "from tensorflow.keras.optimizers import RMSprop\n", + "\n", + "import random\n", + "import sys\n", + "import os" + ], + "execution_count": 21, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "wrvoGYxWr92N", + "colab_type": "code", + "colab": {} + }, + "source": [ + "# build the model: a single LSTM\n", + "\n", + "model = Sequential()\n", + "model.add(LSTM(128, input_shape=(maxlen, len(chars))))\n", + "model.add(Dense(len(chars), activation='softmax'))\n", + "\n", + "model.compile(loss='categorical_crossentropy', optimizer='adam')" + ], + "execution_count": 22, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "C14p3ItDr95d", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 218 + }, + "outputId": "e06e2598-208e-47f1-fdc6-1e3799e47f92" + }, + "source": [ + "model.summary()" + ], + "execution_count": 23, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Model: \"sequential\"\n", + "_________________________________________________________________\n", + "Layer (type) Output Shape Param # \n", + "=================================================================\n", + "lstm (LSTM) (None, 128) 99840 \n", + "_________________________________________________________________\n", + "dense (Dense) (None, 66) 8514 \n", + "=================================================================\n", + "Total params: 108,354\n", + "Trainable params: 108,354\n", + "Non-trainable params: 0\n", + "_________________________________________________________________\n" + ], + "name": "stdout" + } + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "_FKbrEtLr98Z", + "colab_type": "code", + "colab": {} + }, + "source": [ + "def sample(preds):\n", + " # helper function to sample an index from a probability array\n", + " preds = np.asarray(preds).astype('float64')\n", + " preds = np.log(preds) / 1\n", + " exp_preds = np.exp(preds)\n", + " preds = exp_preds / np.sum(exp_preds)\n", + " probas = np.random.multinomial(1, preds, 1)\n", + " return np.argmax(probas)" + ], + "execution_count": 24, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "862wNjYar9-6", + "colab_type": "code", + "colab": {} + }, + "source": [ + "def on_epoch_end(epoch, _):\n", + " # Function invoked at end of each epoch. Prints generated text.\n", + " \n", + " print()\n", + " print('----- Generating text after Epoch: %d' % epoch)\n", + " \n", + " # Random prompt\n", + " start_index = random.randint(0, len(text) - maxlen - 1)\n", + " \n", + " generated = ''\n", + " \n", + " sentence = text[start_index: start_index + maxlen]\n", + " generated += sentence\n", + " \n", + " print('----- Generating with seed: \"' + sentence + '\"')\n", + " sys.stdout.write(generated)\n", + " \n", + " for i in range(400):\n", + " x_pred = np.zeros((1, maxlen, len(chars)))\n", + " for t, char in enumerate(sentence):\n", + " x_pred[0, t, char_int[char]] = 1\n", + " \n", + " # Predict the next step (character)\n", + " preds = model.predict(x_pred, verbose=0)[0]\n", + " next_index = sample(preds)\n", + " next_char = int_char[next_index]\n", + " \n", + " sentence = sentence[1:] + next_char\n", + " \n", + " sys.stdout.write(next_char)\n", + " sys.stdout.flush()\n", + " print()\n", + "\n", + "print_callback = LambdaCallback(on_epoch_end=on_epoch_end) \n" + ], + "execution_count": 27, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "NZo9WiZ0r-CQ", + "colab_type": "code", + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 + }, + "outputId": "929c8968-02d4-4eab-e87a-ad19b02a51ef" + }, + "source": [ + "# fit the model\n", + "\n", + "model.fit(x, y,\n", + " batch_size=32,\n", + " epochs=10,\n", + " callbacks=[print_callback])" + ], + "execution_count": 28, + "outputs": [ + { + "output_type": "stream", + "text": [ + "Epoch 1/10\n", + "1708/1709 [============================>.] - ETA: 0s - loss: 1.5407\n", + "----- Generating text after Epoch: 0\n", + " \n", + " b r i e r s i s t h \"\n", + " \n", + " b r i e r s i s t h e t h i l o l w e s t o r s h p o l n s p e ! z d i l y a c r f p s o u r i l m a v e g ; t h i g r y A . l u t e o r l ' g i d h i r a t s i s g i - G R I C . , M a r b e r t e i n t e c a b e t m a , t A J e \n", + " \n", + "\n", + "1709/1709 [==============================] - 20s 11ms/step - loss: 1.5405\n", + "Epoch 2/10\n", + "1709/1709 [==============================] - ETA: 0s - loss: 1.1569\n", + "----- Generating text after Epoch: 1\n", + " \n", + " C E L I A . W h \"\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " R R \n", + "1709/1709 [==============================] - 19s 11ms/step - loss: 1.1569\n", + "Epoch 3/10\n", + "1698/1709 [============================>.] - ETA: 0s - loss: 1.0597\n", + "----- Generating text after Epoch: 2\n", + "----- Generating with seed: \" e n o f g r e a t w o r t h r e\"\n", + " \n", + " U f a p t r l y a n p o l o s t h o v e a r i s g a t y o u a f c h a n g f o v e r d m e a n d z i d o o n h e r Y o v e I s a t h e G R o u f h a a d h a e n t h e c ' v h l a n d I y h o t i n m e a h - v e r e S a t M E R g a r . T C U L E . S o u d l a J y o t\n", + "1709/1709 [==============================] - 20s 11ms/step - loss: 1.0593\n", + "Epoch 4/10\n", + "1700/1709 [============================>.] - ETA: 0s - loss: 1.0038\n", + "----- Generating text after Epoch: 3\n", + "----- Generating with seed: \"r s e s w i t h r e a d i n g t h \"\n", + " \n", + " \n", + " \n", + " \n", + " S o r \n", + "1709/1709 [==============================] - 19s 11ms/step - loss: 1.0041\n", + "Epoch 5/10\n", + "1697/1709 [============================>.] - ETA: 0s - loss: 0.9651\n", + "----- Generating text after Epoch: 4\n", + " \n", + " T o \"\n", + " \n", + " \n", + " T o u t h a l l w i e y f e d t , w i l l t o h e a n y s i t r i s t h e f r a r o f p h e b u l l f u r g t h a y m e v e r ' l d b h a n t d a t b r s a l d w e r y f r i t s u r s i g t h e t a r g w o l t f t r o t I F e n t ; y o u h a u b o r n s o u g h i l s h i d s s , a n d t h l l ; t h e \n", + "1709/1709 [==============================] - 19s 11ms/step - loss: 0.9655\n", + "Epoch 6/10\n", + "1704/1709 [============================>.] - ETA: 0s - loss: 0.9348\n", + "----- Generating text after Epoch: 5\n", + "----- Generating with seed: \" \n", + " C H A R L E S . O , n o ; f\"\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " O O L I N D .\n", + "1709/1709 [==============================] - 20s 11ms/step - loss: 0.9348\n", + "Epoch 7/10\n", + "1703/1709 [============================>.] - ETA: 0s - loss: 0.9089\n", + "----- Generating text after Epoch: 6\n", + "----- Generating with seed: \"I V E R . W h e r e w i l l t h e \"\n", + " \n", + " c h e n t i n m e t o f o u m f c h a t o f y o u p a s m a v e h o r v e r a n d i s t r a s t o f l e f t h e ? W i l l h e m o n ! t e a n t y o u I t h e e n d e r o n , r i s t B u l l w o B u e d o n g h e l e t h a h a s k b e n e s ; o f d e n t t e \n", + "1709/1709 [==============================] - 20s 11ms/step - loss: 0.9085\n", + "Epoch 8/10\n", + "1705/1709 [============================>.] - ETA: 0s - loss: 0.8869\n", + "----- Generating text after Epoch: 7\n", + "----- Generating with seed: \"a g e , h e h a t h s t r a n g e \"\n", + " \n", + " \n", + " \n", + " \" w o t h l y y o u n g s h e r \n", + "1709/1709 [==============================] - 19s 11ms/step - loss: 0.8867\n", + "Epoch 9/10\n", + "1704/1709 [============================>.] - ETA: 0s - loss: 0.8661\n", + "----- Generating text after Epoch: 8\n", + "----- Generating with seed: \" R O S A L I N D . A y , b e s\"\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " A n d m e\n", + "1709/1709 [==============================] - 20s 11ms/step - loss: 0.8658\n", + "Epoch 10/10\n", + "1700/1709 [============================>.] - ETA: 0s - loss: 0.8470\n", + "----- Generating text after Epoch: 9\n", + "----- Generating with seed: \" s e h e r f o r h e r v i r t u\"\n", + " \n", + " \n", + " \n", + " E L I N e I S I L I K . \n", + "1709/1709 [==============================] - 19s 11ms/step - loss: 0.8470\n" + ], + "name": "stdout" + }, + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "" + ] + }, + "metadata": { + "tags": [] + }, + "execution_count": 28 + } + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "zE4a4O7Bp5x1" + }, + "source": [ + "# Resources and Stretch Goals" + ] + }, + { + "cell_type": "code", + "metadata": { + "id": "8EJ4PPR4uLF2", + "colab_type": "code", + "colab": {} + }, + "source": [ + "def print_text_from_seq(x):\n", + " INDEX_FORM = 3\n", + " word_to_id = imdb.get_word_index()\n", + " word_to_id = {k:(v+INDEX_FORM) for k,v in word_to_id.items()}\n", + " word_to_id[\"\"] = 0\n", + " word_to_id[\"\"] = 1\n", + " word_to_id[\"\"] = 2\n", + " word_to_id[\"\"] = 3\n", + "\n", + " id_to_word = {value:key for key,value in word_to_id.items()}\n", + " print('==================================================')\n", + " print(f'Length = {len(x)}')\n", + " print('==================================================')\n", + " print(' '.join(id_to_word[id] for id in x))" + ], + "execution_count": 29, + "outputs": [] + }, + { + "cell_type": "code", + "metadata": { + "id": "TQ5MnjEFyqGN", + "colab_type": "code", + "colab": {} + }, + "source": [ + "from __future__ import print_function\n", + "\n", + "from tensorflow.keras.layers import Dense, Embedding\n", + "from tensorflow.keras.layers import LSTM\n", + "from tensorflow.keras.datasets import imdb" + ], + "execution_count": 31, + "outputs": [] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "uT3UV3gap9H6" + }, + "source": [ + "## Stretch goals:\n", + "- Refine the training and generation of text to be able to ask for different genres/styles of Shakespearean text (e.g. plays versus sonnets)\n", + "- Train a classification model that takes text and returns which work of Shakespeare it is most likely to be from\n", + "- Make it more performant! Many possible routes here - lean on Keras, optimize the code, and/or use more resources (AWS, etc.)\n", + "- Revisit the news example from class, and improve it - use categories or tags to refine the model/generation, or train a news classifier\n", + "- Run on bigger, better data\n", + "\n", + "## Resources:\n", + "- [The Unreasonable Effectiveness of Recurrent Neural Networks](https://karpathy.github.io/2015/05/21/rnn-effectiveness/) - a seminal writeup demonstrating a simple but effective character-level NLP RNN\n", + "- [Simple NumPy implementation of RNN](https://github.com/JY-Yoon/RNN-Implementation-using-NumPy/blob/master/RNN%20Implementation%20using%20NumPy.ipynb) - Python 3 version of the code from \"Unreasonable Effectiveness\"\n", + "- [TensorFlow RNN Tutorial](https://github.com/tensorflow/models/tree/master/tutorials/rnn) - code for training a RNN on the Penn Tree Bank language dataset\n", + "- [4 part tutorial on RNN](http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/) - relates RNN to the vanishing gradient problem, and provides example implementation\n", + "- [RNN training tips and tricks](https://github.com/karpathy/char-rnn#tips-and-tricks) - some rules of thumb for parameterizing and training your RNN" + ] + } + ] +} \ No newline at end of file