{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-jlGXqM_t452"
   },
   "source": [
    "# P8 - Convolutional Neural Networks (CNNs)\n",
    "We have now learned about the Perceptron, Linear and logistic regression, Multi-layer perceptron and backpropagation, Auto-encoders. \n",
    "\n",
    "In this pratical session about Convolutional Neural Networks (CNNs) we will use the MNIST datasets.\n",
    "\n",
    "First, we will obtain baselines using a Logistic Regression and a Feed-forward Neural Network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ITJR4snhxdT0"
   },
   "source": [
    "## 0.0 - Imports\n",
    "We will need to import some libraries to be used in this session. Libraries include data visualizers ([matplotlib](https://matplotlib.org/)), neural network package ([torch](https://pytorch.org/)), and other helper packages for data handling ([sklearn](https://scikit-learn.org/), [numpy](https://numpy.org/))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "MWGjU3tDw4bD"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.base import BaseEstimator\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.utils import check_random_state\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms\n",
    "from torch.autograd import Variable\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from torch.utils.data.sampler import SubsetRandomSampler\n",
    "import time\n",
    "import copy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W-od7M6WMN0N"
   },
   "source": [
    "Then, other variable definitions are needed to be set. This includes the size of the dataset we will use, and the configuration of the GPU to be activated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ECqewHJ0MM62",
    "outputId": "e5377940-a224-4e98-b427-bad0a9579863"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu\n"
     ]
    }
   ],
   "source": [
    "# Configure Device\n",
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "odY0Ng9yycgr"
   },
   "source": [
    "### 0.1 - Create Dataloaders\n",
    "#### MNIST dataset \n",
    "Using torchvision we can easily download and use the MNIST dataset to create our train and validation dataloaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "snFv-Hu-zRnW"
   },
   "outputs": [],
   "source": [
    "# Define tranform - Convert data to tensor and normalize using dataset mean and std\n",
    "# mean and std are computed offline using the training dataset\n",
    "# tranforms.Normalize expects a value of mean and std per image channel\n",
    "mnist_transform = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "     transforms.Normalize((0.1307,), (0.3081,))])\n",
    "\n",
    "# These random translations will be added in the end of this notebook, for now we skip this. \n",
    "#mnist_transform_test = transforms.Compose(\n",
    "#    [transforms.ToTensor(),\n",
    "#     transforms.RandomAffine(0, translate=[0.1, 0]),\n",
    "#     transforms.Normalize((0.1307,), (0.3081,))])\n",
    "\n",
    "# Download and create MNIST train and validation dataloaders\n",
    "mnist_train_dataset = datasets.MNIST('../data', download=True, train=True, transform=mnist_transform)\n",
    "mnist_val_dataset = datasets.MNIST('../data', download=True, train=False, transform=mnist_transform)\n",
    "#mnist_val_dataset = datasets.MNIST('../data', download=True, train=False, transform=mnist_transform_test)\n",
    "mnist_train_dataloader = DataLoader(mnist_train_dataset, batch_size=64, shuffle=True)\n",
    "mnist_val_dataloader = DataLoader(mnist_val_dataset, batch_size=64, shuffle=True)\n",
    "\n",
    "# MNIST Dataloaders to get data into numpy for Logistic Regression\n",
    "mnist_train_dataloader_numpy = DataLoader(mnist_train_dataset, batch_size=len(mnist_train_dataset))\n",
    "mnist_val_dataloader_numpy = DataLoader(mnist_val_dataset, batch_size=len(mnist_val_dataset))\n",
    "X_y_train = next(iter(mnist_train_dataloader_numpy))\n",
    "X_y_val = next(iter(mnist_val_dataloader_numpy))\n",
    "X_train = X_y_train[0].numpy()\n",
    "y_train = X_y_train[1].numpy()\n",
    "X_val = X_y_val[0].numpy()\n",
    "y_val = X_y_val[1].numpy()\n",
    "\n",
    "dataloaders = dict(train=mnist_train_dataloader, val=mnist_val_dataloader)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vJANJSJ0hsqd"
   },
   "source": [
    "We can check the MNIST dataset properties such as:\n",
    "\n",
    "- shape of train and validation datasets - \\[number of samples, width, height\\]\n",
    "- number of input feature on the flattened/reshaped input for Logistic Regression or MLP\n",
    "- shape of train and validation batches - \\[batch size, number of channels, width, height\\]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KuluBqnnCbn5",
    "outputId": "fa915c87-a262-4a70-ea87-d55cf5985317"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datasets shapes: {'train': torch.Size([60000, 28, 28]), 'val': torch.Size([10000, 28, 28])}\n",
      "N input features: 784 Output classes: 10\n",
      "Train batch: torch.Size([64, 1, 28, 28]) torch.Size([64])\n",
      "Val batch: torch.Size([64, 1, 28, 28]) torch.Size([64])\n"
     ]
    }
   ],
   "source": [
    "# get batch to extract properties and plot example images\n",
    "# next(enumerator(dataloader)) -> creates an iterator of the dataloader and gets the next batchß\n",
    "batch_idx, (example_imgs, example_targets) = next(enumerate(mnist_train_dataloader))\n",
    "# info about the dataset\n",
    "D_in = np.prod(example_imgs.shape[1:])\n",
    "D_out = len(mnist_train_dataloader.dataset.targets.unique())\n",
    "print(\"Datasets shapes:\", {x: dataloaders[x].dataset.data.shape for x in ['train', 'val']})\n",
    "print(\"N input features:\", D_in, \"Output classes:\", D_out)\n",
    "print(\"Train batch:\", example_imgs.shape, example_targets.shape)\n",
    "batch_idx, (example_imgs, example_targets) = next(enumerate(mnist_val_dataloader))\n",
    "print(\"Val batch:\", example_imgs.shape, example_targets.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JnFAmoinjY1T"
   },
   "source": [
    "We can plot some examples with corresponding labels using the following function. This function can also receive the predicted labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 284
    },
    "id": "5ZWvjQOvC2ep",
    "outputId": "c77ced2a-931a-4fb1-db71-5354316f0e6d"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAELCAYAAADgPECFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmJ0lEQVR4nO3dedxV0/4H8M+3QfNcNNwGDboNqG5Kl5QhKrqUMqVEKD8zt7olicrQjQz5xS8y/IiSWa+QMaEfFSIq0UBz6qJZWr8/9n6WtZbnnGefYZ2hPu/X63m1vmftYT279Zx19lrrrC1KKRAREflSLNsFICKiAxsbGiIi8ooNDRERecWGhoiIvGJDQ0REXrGhISIir9jQJEhEHheRsdkuB+UfEXlPRC7Ndjko/+R73Ynb0IjIduNnv4jsMuK+mSqkiJwuIvNE5D8iskFEHhGRCnG2X2WUdWPYOJTPVHmNcgwQESUiQ53XfxSRzpkuTyblSt0Jy3KBiKwWkR0i8pKIVI2zrQq32y4ia0XkHhEpnsnyhuUYHZblHOO1EuFrDTJdnkzKlbojIp3D85vluSjO9qw7McRtaJRS5Qt+AKwB0MN47emC7USkhOdyVgIwFkBtAM0A1AHw7yL26RGWuw2AtgBGuhtkoNwAsBXA0HgN44EoV+qOiLQA8DCAfgAOA7ATwH8XsdvRYblPBnABgMsKOW6m6s6t2XizyqZcqTuhdWZ5lFJPFLE9604hkuo6C1v6H0VkmIhsAPBY+Ol9nrOdEpHGYbqUiEwQkTXhXcZDIlImyvmUUtOUUq8rpXYqpbYBmALguIj7rgUwG0BLo0xXisi3AL4NXztDRD4P75g+EpGjjN+htYgsEpFfRWQ6gNJRzmv4BsDHAG4oLDO8LveKyLrw514RKRXmFVznG0Vkk4isF5GLnX2TuqbZkum6A6AvgFeVUnOVUtsB3AygV5SGXym1FMAHAFqKSIOwTANFZA2Ad8KyXSIi34jINhF5Q0TqG79DFxFZKiI/i8gkABKxzAVeB7AXwIWFZYpIJRF5UkQ2S3DHNlJEioV5AyToBZgQlm2liHRz9n00rFNrRWRsrrwpxZKFupM01h1bKmM0NQFUBVAfwOURtr8TwBEAWgFojOCuZFRBZvgmf3zEc58AYEmUDUWkLoDuAD4zXj4LQHsAzUWkNYCpAAYBqIbg0+8rYQU9BMBLAP4Xwe/6HICzneNHKffNAK6TwrtsbgJwLILrcjSAdrDvvmoiuKOrA2AggAdFpEqYF/ea5rBM1p0WAL4oCJRS3yH4AzyiqJOKSHMAHWHXnU4I7qpPE5EzAYwA0AtADQRvLM+E+1YH8AKC/8vqAL6D8eFIROqF5a4XpwgKQd25RURKFpL/AIK60TAsV38AFxv57QEsC88/HsCjIlLwhvU4gH0IrmdrAKcCyIcxgEy/7xwaNlArRWSiiJSLUkjWHbc0SkX6AbAKwClhujOCP9bSRv4AAPOcfVRYGAGwA0AjI68DgJVRz2/s1wXANgBHFFHW7QD+A2A1gq6SMkaZTjK2nQxgjLP/svDinwBgHQAx8j4CMDZiWfU1ATADwF1h+kcAncP0dwC6G/ucBmCVcZ13AShh5G9C0DCl7Zr6/slm3QHwNoDBzmtrC65/IdsrAL+Edew7BF22xQA0CPMaGtvOBjDQiIsh6Jqrj+APd76RJ+H/+6URyz0awFNh+v8AXAGgRFiGBgCKh9exubHPIADvGdd0hZFXNty3JoIuxD0FfxNh/vkA3s12XcmxulMTQPPw//VwAHMBPBxne9adGD+p9BVuVkrtjrhtjbCwC/9oFCHhLxyZiBwLYBqA3kqp5UVsfpZS6q0YeT8Y6foALhKRq43XDkEwHqQArFXh1QytTqTMhlEAPhGRe5zXazvHXB2+VuAnpdQ+I94JoDzSdE2zJJN1ZzuAis5rFQH8GmefNkqpFeYLxrndunOfiNxtborgU3Ntc1ullBIRc99EjATwGII76wLVAZTEn+tOHSPeYJx/Z/g7lEdwR1ASwHrj9yoG+3fLVRmrO0qpDfjjGq6UYFLPawjelGNh3SlEKg2Nu+zzDgT/qQAAEalp5G1B8Mm8hQrGTBIWdnG9AuASpdTbyRzDYJb9BwDjlFLjCjlnJwB1RESMxqYegk8riZ1QqaUi8gKCrjLTOgSVrqArsF74WlFSvqZZlMm6swRBl2TBsRsCKAWgqA8qsRRWd552NxKRJgDqGrGYcUInVGqOiKwA8F/Gy1sA/Iag7nwdvlYPwd1aUX5A8Km0uvMhJh9k9H2nkHOnMtxw0NaddH6P5gsALUSklYiURnD7BgBQSu1HMIA/UUQOBQARqSMip0U5sIi0RDC4dbVS6tU0lhlhuQaLSHsJlJNgOnUFBIP4+wBcIyIlRaQXgjGUZN2KoB+0svHaMwBGikiNsG92FICnijpQqtc0x3irOwCeBtBDRDqG/eu3AXhBKRXvjiaqhwAMl2BmW8EgaZ8wb1b4O/WSYJbRNQi6HpJ1EwA9TV4p9TuC7thxIlJBgoHkGxCt7qwH8CaAu0WkoogUE5FG4QerfOPzfedEEakfvi/URTDe83Kayn1Q1Z20NTRhV9ZtAN5CMJtrnrPJMAArAMwXkV/C7ZoWZEow97xjjMPfiOA2+FH5Yz57pMkAEcq9AMEUxEkI+lZXIOijhFJqL4LBugEIpguei2CQTiui3O65ViK4hTUHFMcCWABgMYAvASwKX4si7jXNFz7rjlJqCYDBCBqcTQAqwP50l0q5XwRwF4Bnw3J9BaBbmLcFQB8Eb04/AWgC4EOjzPXCcscb0DXP9SGAT5yXr0bwif57BNdsGoKJLVH0R9BF/DWCej8TQK2I++YMz+87rRGMye4I//0SwZt+Osp9UNUdsYcfiIiI0otL0BARkVdsaIiIyCs2NERE5BUbGiIi8ooNDREReZXwFzZFhNPUcpBSKtFF9zKK9SZnbVFK1ch2IeJh3clZkesO72iIDm7JLqlEFLnusKEhIiKv2NAQEZFXbGiIiMgrNjREROQVGxoiIvKKDQ0REXnFhoaIiLxiQ0NERF6xoSEiIq/Y0BARkVdsaIiIyCs2NERE5FXCqzfngrZt2+p0t27dIu/317/+1YovuOCCmNvedtttVnzLLbdEPg8d2CpUqGDF5cqVi7zv5s2bdfr3339PW5mIchnvaIiIyCs2NERE5JUoldgzhbLxEKIZM2ZYcc+ePXW6ePHiXs7pXpfVq+1HL4wZM0anH3vsMS9lSAQffJaahg0bWnH79u11umvXrlZemzZtrLhZs2aRz1O7dm2d3rRpUyJF9GWhUqpt0ZtlT67XnYNY5LrDOxoiIvKKDQ0REXnFhoaIiLzKyTEad0ymd+/evk+ZsLVr1+r0iSeeaOWtWLEi08XhGE0EpUqV0uk77rjDyrvwwgutuGrVqjGPI2Jf6kT+hu6++26dnjx5spW3atWqyMdJI47RZFHp0qV1evfu3VZe2bJlY8bHHHOMlWeOIbv1s27dula8YMGCyOXbsmVLvGyO0RARUW5gQ0NERF7lZNfZnj17rLhkyZK+T5mSc845x4pnzpyZ8TKw66xoV111lU7fe++9SR8nla4z08aNG634vvvu0+nx48cndcwksOssg8477zwrHjJkiE7/8MMPVl6jRo2suHnz5pHOka76CRT59RF2nRERUW5gQ0NERF6xoSEiIq9yZvXm66+/XqdTGZNxp0Zv375dpydNmmTluX3k5jIk7nFq1aqVdJkoO6ZOnWrFF110kU6n0m89YcKEmHnuVPz69evH3LZmzZpWbK4Y7q7sbE6LpuwyV48H/rwE1a233qrTvXr1svL69OljxcWK/fFZv3Xr1lZevDr67bffWrFZXxIZo3GPM2vWrJjbpoJ3NERE5BUbGiIi8ooNDREReZUzYzQrV67U6f3791t5Zj8mACxfvlyn3T7xpUuXWvG+ffsil2H9+vU6vWzZMiuPYzS5yVxW5uGHH7by+vXrZ8VmPXLr2N69e6140KBBOv3kk09GLs+wYcOsuEWLFlZ888036/Tpp59u5ZlLjLjfo5kzZ44VL168OHKZKHWVK1fW6YkTJ1p57v9xnTp1dLpatWpW3sKFC634yy+/1OkPP/zQyjPf51yffvqpFf/2228xt80FvKMhIiKv2NAQEZFXOdN19tJLL+n0F198YeW50/7eeecdnf7qq6+8lotyS5kyZaz4/vvv12l3BWZ3WqfZXbZz504r75prrrHiRLrL4lmyZIkVm0uQuEsXTZs2LeZxzCe6AkD//v11+ueff06liBSB2UV/3HHHWXlul5e5lJCZPpjxjoaIiLxiQ0NERF6xoSEiIq9yZozG1LVrVyt+8MEHrfiee+7xct4aNWrodPXq1eNu+8orr+i0O/WU/OnSpYsVX3zxxZH3Nacwu2Mw7jIivjRo0ECn77zzzsj7uVOhmzZtqtOffPJJyuWi+Bo3bqzT7thfiRL226i5tL67lNDBinc0RETkFRsaIiLyig0NERF5lZNjNJs3b7Zi9/sGvphjQS1btrTy3O8qmP3r/B6DXyeccIJOu0v/J8J8fPPw4cNTKVLSmjVrptM7duxI+jjmUjY9evRIqUxUNPcxy6b27dtb8bhx43T6X//6l7cy5RPe0RARkVdsaIiIyCtJ9EmDIpL8owlzzLHHHmvFb7zxhk5XqFDByluwYIEVt2vXzl/BkqCUkqK3yp5U6s27776r0x07doy83+TJk63Y7MZIpdsqXRJZgiYed3ptghYqpdoWvVn25MJ7TtWqVXX6rbfesvJatWplxeaUZvfJqDfddFPMbfNQ5LrDOxoiIvKKDQ0REXnFhoaIiLzKyenNmXL00UdbsTsuY1qzZo3v4lAM5viYOdW5KGPHjrXiXBiXMb333ntWvGfPHp0uXbp03H0nTJjgo0gUw9atW3W6W7duVt77779vxU2aNNHpIUOGWHnueM706dN1OlPLIGUD72iIiMgrNjREROQVGxoiIvLqoBqjMR/HCgB33XVXzG23b99uxewTzx7zu17xvvflPlI315cG2rRpkxXPnDlTp/v27Zvp4lBEGzdutOJOnTpZ8ciRI3X6yiuvtPJOPfXUmLH7KIh+/fpZ8a5duxIvbI7gHQ0REXnFhoaIiLw64LvOKleurNO33367lVexYsWY+w0dOtSK58+fn9ZyUfqtXLnSinfv3p2lktDBxO1Ku+6663R66dKlVt6IESOsuGbNmjrdq1cvK+/rr7+2YnO6vvm02HzAOxoiIvKKDQ0REXnFhoaIiLw64MZoevbsacXmWEvjxo0jH2fZsmVpKxNlRsOGDa24TJkyVpxr00Pd5XTc6a2Un8yl/82n9gLAE088YcUff/yxTrdo0cLKcx8pYE6HnzRpUsrlzCTe0RARkVdsaIiIyCs2NERE5FVejtHUr19fp83H8wLAhRdeaMXlypWLfNwlS5botDuHPV2KFy9uxXXr1tXpVatWeTnnweLvf/+7FZ900klWPGvWrEwWp0i33nqrFZvf+XK59dF9RDDlB3dpq+eff16n3TEal1mfOUZDRERkYENDRERe5UXXWZcuXaz4lVde0elSpUql7TxVq1bV6cGDB1t5jzzySOTjVKlSRaevv/56K899cqK5orS7WuuMGTMin/NAZq6c7a6GG+9JlGY9AYC5c+fq9OzZs628KVOmWPG2bdsil69SpUo67U6pbtasWcwylS9f3srbv39/zPOfd955Vuyu/Ez5yXx67IGMdzREROQVGxoiIvKKDQ0REXkl8Z5YWOgOIontkAbmstsAcM8992S6CBnhLv0db/zBpZSSdJcnndJVb9yxiqeeeiqRMui0W+/Xr19vxe+8806k4wBAhw4ddPrwww9PqjwAsGbNGp12p2q75UujhUqptr4Ong6+3nM2bNig0+746Jw5c5I+7l/+8hed7tixo5Xnfh3jyCOP1Gm3PrjvByeffLJOz5s3L+nypVHkusM7GiIi8ooNDREReZUX05uJCrjf7jenoZ999tlWXqdOnaw43lT4WrVqWXHfvn1jbut2cSTa/Vxg+fLlVjx+/Hid9thVRqHVq1frtFuvVqxYEXO/b7/91ord+tCuXTudrlGjRtwymHXnu+++s/LOPfdcK160aFHcY+Uy3tEQEZFXbGiIiMgrNjRERORVXozRmE+sA+xpf4cccoiVZy7j4cYlSuTer/vrr7/q9OjRo7NXkDxhXi/AXhrIXSaoefPmVlyxYkWddpcYSkS8MRp3Be6ZM2fGPM7GjRutePPmzUmXiRLXrVs3nXa/MmHWFXfbpk2bWnmJjNmZK8QDwP3336/Tb775ppVnTnfPd7yjISIir9jQEBGRV2xoiIjIq7xYgiaea6+91ordPnJzznvPnj2tvCZNmljx8ccfr9Nuv2siS4uYS7y7fb87d+604okTJ0Y+bjwHyxI0lHYH7RI0iTjttNN0+qijjrLyPvjgg5j7ud+5cf/+d+3alYbSZQ2XoCEiotzAhoaIiLzK+64zCrDrjJLErjNKFrvOiIgoN7ChISIir9jQEBGRV2xoiIjIKzY0RETkFRsaIiLyig0NERF5xYaGiIi8YkNDREResaEhIiKv2NAQEZFXbGiIiMgrNjREROQVGxoiIvKqRBL7bAGwOt0FoZTUz3YBImC9yU2sO5SsyHUn4efREBERJYJdZ0RE5BUbGiIi8ooNDRERecWGhoiIvGJDQ0REXrGhISIir9jQEBGRV2xoiIjIKzY0RETkFRsaIiLyig0NERF5xYaGiIi8YkNDREResaFJkIg8LiJjs10Oyj+sO5QsEXlPRC7NdjmSFbehEZHtxs9+EdllxH0zVciwLDVEZJqI/Cwi20Tk6TjbrjLKujH8Ay+fyfKG5RggIkpEhjqv/yginTNdnkxi3UkN607O1J0LRGS1iOwQkZdEpGqcbVW43XYRWSsi94hI8UyWNyzH6LAs5xivlQhfa5Dp8gBFNDRKqfIFPwDWAOhhvKb/WEUkmQeoJeoFABsA1ANwKIAJRWzfIyx3GwBtAYx0N8hQubcCGCoiFTJwrpzBupMWrDtZrDsi0gLAwwD6ATgMwE4A/13EbkeH5T4ZwAUALivkuJmqO7dmo6ErTFJdZyLSOfxkNUxENgB4LPwENs/ZTolI4zBdSkQmiMia8JPiQyJSJuL5TgVQF8AQpdTPSqnflFKfRdlXKbUWwGwALY0yXSki3wL4NnztDBH5XET+IyIfichRxrlbi8giEflVRKYDKB3lvIZvAHwM4IYYv1spEblXRNaFP/eKSKkwr+A63ygim0RkvYhc7Oyb1DXNFtadhLDuGDJddwD0BfCqUmquUmo7gJsB9IrS8CullgL4AEBLEWkQlmmgiKwB8E5YtktE5BsJ7rLfEBH9xEoR6SIiSyW4C58EQCKWucDrAPYCuLCwTBGpJCJPishmCe7YRopIsTBvgIjMC6/bNhFZKSLdnH0fDevUWhEZK0U0aKmM0dQEUBXB4zwvj7D9nQCOANAKQGMAdQCMKsgM/1CPj7HvsQCWAXhCRH4SkU9FpFOUQopIXQDdAZhvLmcBaA+guYi0BjAVwCAA1RB8gnklrKCHAHgJwP+Gv+tzAM52jh+v3AVuBnCdFH7bfVP4+7UCcDSAdrA/QdcEUAnB9RoI4EERqRLmxb2mOYx1p+hyF2DdsWWy7rQA8EVBoJT6DsGb9xFFnVREmgPoCLvudALQDMBpInImgBEAegGogaBReibctzqCu/CRAKoD+A7Accax64XlrhenCApB3blFREoWkv8AgrrRMCxXfwAXG/ntEfzdVAcwHsCjIlLQ2D0OYB+C69kawKkA4o8fKaUi/QBYBeCUMN0ZwQUvbeQPADDP2UeFhREAOwA0MvI6AFgZ8dz/Ex5rIICSAM4D8B8A1eOUdXu4zWoEt7tljDKdZGw7GcAYZ/9l4cU/AcA6hI+8DvM+AjA2Yrn1NQEwA8BdYfpHAJ3D9HcAuhv7nAZglXGddwEoYeRvQvDmktI1zeQP6w7rTp7WnbcBDHZeW1tw/QvZXgH4BcC28P9mLIIP8w3CvIbGtrMBDDTiYgi65uojeNOfb+RJ+P9+acRyjwbwVJj+PwBXACgRlqEBgOLhdWxu7DMIwHvGNV1h5JUN962JoAtxT8HfRJh/PoB345Uplb7CzUqp3RG3rREWduEfjSIk/IWj2IXgD+jRMH5WRG5C0Mq/HGOfs5RSb8XI+8FI1wdwkYhcbbx2CIDaCC7uWhVezdDqiGV2jQLwiYjc47xe2znm6vC1Aj8ppfYZ8U4A5ZH6Nc0m1p3EsO78IZN1ZzuAis5rFQH8GmefNkqpFeYLxrndunOfiNxtborgjqu2ua1SSomIuW8iRgJ4DMGddYHqCD50uXWnjhFvMM6/M/wdyiO4mywJYL3xexWD/bv9SSpdZ8qJdyD4TwUAiEhNI28Lgj/4FkqpyuFPJRUMmkWxuJDzuXEizH1/ADDOKFdlpVRZpdQzANYDqGPcMgLBgHLiJwz6bF9A0N1hWoeg0pnHXxfhkKle02xi3UnkhKw7pkzWnSUIuiQLjt0QQCkAy5Mq+Z/rziCn7pRRSn2EoO7UNc4rZpzQCZWaA2AFgP8yXt4C4Df8ue6sjXDIHxDc0VQ3yl1RKdUi3k7p/B7NFwBaiEgrESmN4PYNAKCU2g9gCoCJInIoAIhIHRE5LeKxXwRQRUQuEpHiItIbwF8AfJiGck8BMFhE2kugnIicHg74fYygL/IaESkpIr0Q9IMn61YE/aCVjdeeATBSgim41RF8en2qqAOl4ZrmEtadorHuFM5n3XkaQA8R6Sgi5QDcBuAFpVS8O5qoHgIwXIKZbQUD7H3CvFnh79RLghlq1yDotkrWTQD0NHml1O8IumPHiUgFCSYh3IBodWc9gDcB3C0iFUWkmIg0KmrcM20NjVJqOYL/iLcQzMiZ52wyDEHLOl9Efgm3a1qQKcHc844xjr0VwD8A/BPAzwD+BeBMpdSWNJR7AYIpiJMQ9K2uQNBHCaXUXgSDdQMQTBc8F8EnSy1euQs510oEt7DljJfHAliA4JP3lwAWha9FEfea5gvWnUjnYt0phOe6swTAYAQNziYAFWDfGaRS7hcB3IWgK/cXAF8B6BbmbQHQB8FEhp8ANIHxwSicDLC9iMkA5rk+BPCJ8/LVCO4Gv0dwzaYhmNgSRX8EXcRfI6j3MwHUireD2F3IRERE6cUlaIiIyCs2NERE5BUbGiIi8ooNDRERecWGhoiIvEp4ZQAR4TS1HKSUSnTRvYxivclZW5RSNbJdiHhYd3JW5LrDOxqig1uyy+IQRa47bGiIiMgrNjREROQVGxoiIvKKDQ0REXnFhoaIiLxiQ0NERF6xoSEiIq/Y0BARkVcJrwyQb1q1aqXTY8aMsfK6d+9uxTt37tTpTp3sB8YtWrQo/YWjnHXIIYfo9DXXXGPljRw50ornzp2r0+eff76Vt2PHDg+lI8ovvKMhIiKv2NAQEZFXbGiIiMgrUSqxhVHzbSXV119/XadPOeWUuNtu2bJFp+fMmWPl9evXL70FSzOu3pya4sWLW/HFF1+s0w8//HDcfUX+uPRnnnmmlffqq6+moXReLVRKtc12IeLJ9brjqlatmk6XLFnSyvv111+tOM/H8CLXHd7REBGRV2xoiIjIqwNuevOJJ55oxW3atIm57YQJE6x46tSpOl21atX0FoxyWtu2dg9AvO6yJUuWWPH48eN1+v33309vwSjnlClTxoq7dOlixeb7SOXKla28Dz/80Iovu+wynV6+fHmaSph7eEdDREResaEhIiKv2NAQEZFXeT+92ZxKCADLli2zYrOP9LXXXrPyevfubcX79u1Lb+EyiNObE3Psscda8ezZs624YsWKMfdt2rSpFa9YsSJ9Bcs8Tm+OoFSpUjo9adIkK8+cCl8Ucyo8YE93HjJkiJX37LPPxtw2R3B6MxER5QY2NERE5FXeT2/u0KGDFbvTCU133nmnFedzVxmlZuLEiVYcr6vs66+/tmJzlW86MPXp08eKR40apdPNmjWz8t5++20rNleDuOOOO6y8smXLWnH58uV1evLkyVZe165drfj222/X6YULF8Ysey7iHQ0REXnFhoaIiLxiQ0NERF7l/RiN+yRMd/rgSy+9pNPz58/PRJEoR5199tk63aJFi7jbfv/99zp93HHHWXm//PJLegtGWeeO395www1WbK7u/fjjj1t5V1xxhRXv3btXp92lrMqVK2fFNWrU0On+/ftbee5K4OZSN8OGDbPy3PGdXMM7GiIi8ooNDRERecWGhoiIvMrLJWgOPfRQnXbnsLtz3Lt166bT7lMzDyRcgqbQc1rx9OnTddocrwH+/N0Ys97MmzfPQ+lyxkG7BM2YMWN0esSIEXG33bRpk06ffPLJVp77PatkNWjQwIpvvvlmKzbHcEqUsIfXzeW1brzxRivP4+MHuAQNERHlBjY0RETkVV5ObzZvIZs3b27luSuc/vTTTxkpE+Ues4sV+HN3menee++14gO8u+ygVK9ePSs+6aSTdNodQti8ebMVm91l6eoqc61atcqKBw4caMVm993QoUOtPLOrd/369Vbetddea8W7du1KpZhJ4R0NERF5xYaGiIi8YkNDRERe5eUYjTuF2WQuHQIAixYt8l0cIspR5pMxH3nkESuvffv2Ou2Oa/To0cOKfY3LJGL48OE6/dlnn1l506ZN0+lLLrnEynN/t1tuucVD6eLjHQ0REXnFhoaIiLxiQ0NERF7l5RiNOWfc9dBDD2WwJESUy6pUqaLT5vdmXO5SVp9//rmvIqXFiy++aMX//ve/dXrIkCFW3pVXXmnF999/v05n6nuGvKMhIiKv2NAQEZFXedl1Zq7KW6yY3Va60xIbN26s0+606O7du1uxeaz9+/dbeatXr7Zic+XXJ5980sr7/fffY5adssddzTldWrVqpdPuirs9e/aMud+GDRusuGvXrjq9ePHi9BTuIPfoo4/GzDP/ps2/53zw22+/WfGoUaN02py2DQAnnHCCFZtPC80U3tEQEZFXbGiIiMgrNjRERORVXo7RmEt6u2Mp7tTneFOh3aXBv/rqK512x3PcJcanTJmi09WrV7fyzKmGlDsSeZps+fLlddrt8x4wYIAV9+3bN+Y54p3zsMMOs+Jhw4YVekxKnjk+4Y7RmbGv8btMMcds3PEb93fr3LmzTs+YMcNruQrwjoaIiLxiQ0NERF6xoSEiIq8kkX5rABCRxHbwYN26dTrt9nO7j3L++OOPddr9vsuWLVuseO7cuTrtzj2//PLLrTje9yPOP/98nX7uuedibpdOSqmc7mTORr1x64ZZb1xuXfj55591ulGjRnHPY/aBr1271sqbPXu2FV900UU6XaKEPURqntP8/hcAbN26NW4ZUrBQKdXW18HTIZW6c+KJJ+r0nDlzYm63cuVKKzaXaQGABx54INkiZNzrr79uxaeccooVf/PNNzp95JFHpnKqyHWHdzREROQVGxoiIvIqL6c3P/HEEzo9dOhQK2/69OlWPGjQoKTO4d5mz58/34pbtmyp002aNLHy6tevn9Q5Kb02bdpkxTNnztTp3r17W3nuFHU3Nrkr3prdqAsXLrTydu/ebcWvvfaaTj/77LNWXqVKlXTa7Vaj5JirMLt/03/72990+vDDD7fyJk6caMVmF9z48eOtPPe9Idvc5bJc8Z5Q7AvvaIiIyCs2NERE5BUbGiIi8iovO4LjPRXumGOO8XJOd9r0vHnzdNodo6Hc4E7dd6ceRzVu3Dgrdvvot2/fHvlY5nTnDz74wMpr3bq1TrvLiFBytm3bptPuclTm+K471lu5cmUrPvPMM3W6S5cuVp77dM7nn39ep81xQQDYtWtXhFKnxjw/AAwcOND7OYvCOxoiIvKKDQ0REXnFhoaIiLzKyzGaHTt26LT7KOeSJUtacalSpXR6z549SZ/TfFwvAPzjH//Q6XxfYvxgYT7u1u1nb968ecz9zLGTVJlL0LhLg5jfqzHHFsgPc6zNfeTz1VdfbcXmI7rLlClj5Z1xxhkxY/eRIe62CxYsSKDE+Yt3NERE5BUbGiIi8iovu84mT56s0+3atbPy+vXrZ8XmqqvXXnutlRdvqqH7RM0HH3zQiqtVq6bT7jTazZs3xzwuZY85DdldjsRdlsPsDu3evbuV9+KLL1rxe++9p9PuE1/NpYoA4LzzzotZPk5pzh73KxOjR4+OGbvTpM866ywrvvTSS3W6Ro0aVt6nn35qxRs3btRpcwo1ACxatMiK49UPc8hg+PDhVp7btb9z586Yx/GFdzREROQVGxoiIvKKDQ0REXmVl0/YNFWpUsWKFy9ebMW1atXS6alTp1p57vIQ5cqV02n3CXvmcQBg/fr1Ov3YY49ZeeY02kzhEzZT8/TTT1txvLEUlznF3h2jiWfKlClWbNYb9xEHHh3QT9jMBPe94aqrrtLp/v37W3m1a9e24njvv7NmzbLit956S6fdsZ4RI0botDum6DLHcNzp1wniEzaJiCg3sKEhIiKv8r7rzNWmTRsrfvnll3XavcV1mdMA3evirtBq3n660xCzgV1nqWnQoIEVm6uA33jjjTHzgPj1Zs2aNVZsPvH1iy++sPLMqa4ZxK4zj9x6dfTRR1vxP//5T53u0KFD5OO6U5bNeudO1X7qqaes2Hzv2rt3b+RzFoJdZ0RElBvY0BARkVdsaIiIyKsDbozGZa66PGbMGCvPXUpi7ty5Om0+CREA7rvvPitOsW8z7ThG40/ZsmWt+JlnnrHiSpUq6fRnn31m5T300ENWvGzZsjSXLmUco8kicyXoRo0aWXnmSt8AMHjwYJ126+TWrVt1uk+fPlaeuURSmnGMhoiIcgMbGiIi8ooNDREReXXAj9EcLDhGQ0niGE2eMB854D5Z2HyEgDle4xnHaIiIKDewoSEiIq/y8gmbREQHm3x+ci/vaIiIyCs2NERE5BUbGiIi8ooNDRERecWGhoiIvGJDQ0REXrGhISIir9jQEBGRV2xoiIjIKzY0RETkFRsaIiLyig0NERF5xYaGiIi8YkNDREReJfOYgC0AVqe7IJSS+tkuQASsN7mJdYeSFbnuJPwoZyIiokSw64yIiLxiQ0NERF6xoSEiIq/Y0BARkVdsaIiIyCs2NERE5BUbGiIi8ooNDRERecWGhoiIvPp/OxYtLcz5y6wAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_img_label_prediction(imgs, y_true, y_pred=None, shape=(2, 3)):\n",
    "    y_pred = [None] * len(y_true) if y_pred is None else y_pred\n",
    "    fig = plt.figure()\n",
    "    for i in range(np.prod(shape)):\n",
    "        plt.subplot(*shape, i+1)\n",
    "        plt.tight_layout()\n",
    "        plt.imshow(imgs[i][0], cmap='gray', interpolation='none')\n",
    "        plt.title(\"True: {} Pred: {}\".format(y_true[i], y_pred[i]))\n",
    "        plt.xticks([])\n",
    "        plt.yticks([])\n",
    "\n",
    "plot_img_label_prediction(imgs=example_imgs, y_true=example_targets, y_pred=None, shape=(2, 3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Mj3utDDuzDCj"
   },
   "source": [
    "### 1.1 Logistic Regression\n",
    "\n",
    "We can use a very simple Logistic Regression that receives our input images as a vector and predicts the digit. This will be our first baseline to compare with the CNNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "TniyY4bQzBMS",
    "outputId": "54a7e07e-3078-4a71-95f6-5670a051b4b2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score with penalty: 0.9002\n"
     ]
    }
   ],
   "source": [
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(np.reshape(X_train, (X_train.shape[0], -1)))\n",
    "X_val = scaler.transform(np.reshape(X_val, (X_val.shape[0], -1)))\n",
    "\n",
    "clf = LogisticRegression(C=50., multi_class='multinomial', solver='sag', tol=0.1)\n",
    "clf.fit(X_train, y_train)\n",
    "score = clf.score(X_val, y_val)\n",
    "\n",
    "print(\"Test score with penalty: %.4f\" % score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A8rylkCnrwIy"
   },
   "source": [
    "We can select the coefficients for each class and reshape them into the image shape to plot them. This allows us to visualize what are the pixels that are contributing more to the classification for each of the digits. \n",
    "\n",
    "But what happens if the digits are not centered? Will we still get such a good performance? Lets test that out later!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 339
    },
    "id": "2pucfjpaDF9_",
    "outputId": "3d370f1b-27ce-4a5a-a05a-62e25f560876"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj8AAAFCCAYAAAAe+Ly1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABv1ElEQVR4nO29ebRk2VXe+e2YI96c7+U81jyXSqXSWJIQQsJYgATGtCUZyVoW2PTy0BZYdoNZ3dhuDLbbGC8DjRcYhKGRbKDREkY2AiQhqTRLpZKqJNWclVk5D28eYjz9R0TG2XtHxs14ryLyZdb9fmvVqnvfuXGnM9yT59uDhBBACCGEEJIWMtt9A4QQQgghVxNOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5Idc9IvKzIvK7Izz/YyLyhs62iMhvici8iHxRRF4nIo+P4JqHRGRFRLLDPvcoEJHbRORrIrIsIv9QRMoi8scisigivy8if1NEPjbAeX5aRH7jatzziwER+V9F5Eynrcxu9/0Qcr0gjPNDrgdE5J0AfgLA7QCWAXwNwM+FED4jIj8L4OYQwo9chft4HYAPArgthLA6xPMeBfCjIYQ/H9Y5ryYi8p8BLIUQ3tfZfxeAfwDgNSGExjbczxEAzwLIX83rX+W2mAewBOBVIYRHRn09Ql5McOWHXPOIyE8A+CUA/wrAbgCHAPwqgLdtw+0cBnB0mBOfFwmHATzm9p/YjonP9YyI5DZx+G4AJdj3Puh1REQ4/pP0EkLgf/zvmv0PwBSAFQA/nHDMzwL4XbX/+wBOA1gE8CkAd6mytwD4JtqrRycA/OPO3+cA/HcACwAuAvg0gEyn7CiANwF4L4ANAM3OPf1zAG8A8Lw6/0EA/x+AcwAuAPjlzt9vAvDxzt/OA/h/AUx3yn4HQAvAeue8/wTAEQABQK5zzD4AH+nc21MAfsw9/38D8F86z/UYgAcS3tddAP6sc64zAH668/ci2pPMk53/fglAUf3u+9BecVsA8FkA93b+/vHOO9no3P8HAdQA1Dv77wXwHgCfGeAefF2+qnOtBQCPAHiDKvskgH8J4KHOc38MwFyn7Fjn/a10/nu1ewf7Ou97h/rbSzt1k+/s/20A3wIwD+BPARxOun8A3+Oe+5EB6+4PAPwu2qs4PwrgFQC+3Nk/A+AXL1OHtwJYVc/48c7fXwPgS2i3/S+hvfKm39fPdd7XOtorVNvex/kf/9uO/7b9Bvgf/0v6r/NBaaAzCehzjP9g/m0AE4gf86+pslMAXtfZngFwf2f75wH8GoB857/XIcrCRwG8qbP9HtiP+BvQmfwAyHY+0P8ewBja/yp/bafsZgBv7tzTTrQnZb+kztO9Rmf/COzk51Nor3aVANyH9uTqjer5N9Ce2GU7z/L5Pu9qovMOfrJzrgkAr+yU/QsAnwewq3OPnwXwLztlLwVwFsArO9f4W517LnbKP4m2bNevTrrv7Qr30P0dgP1oTxbfgvYq9Zs7+zvVNZ9GeyJQ7uz/wuXeX5938XHYici/BfBrne23oT1RuQNADsDPAPjsZu5fnfdKdVcH8AOdZywD+ByAd3XKx9GWtS53/+YZAexAe6L2rs49v6OzP6ve1zG0J245dCZ5/I//pfE/LnuSa51ZAOfDJuSTEMJvhhCWQwhVtD8uLxGRqU5xHcCdIjIZQpgPIXxV/X0v2v+6r4cQPh1C2KxB3CvQ/lf++0MIqyGEjRDCZzr39FQI4c9CCNUQwjkAvwjgOwY5qYgcBPAggH/aOefXAPwGgHerwz4TQvhoCKGJ9krSS/qc7vsAnA4h/LvOuZZDCF/olP1NAP8ihHC2c4//HO0PKQD8HQD/KYTwhRBCM4Tw2wCqaK/MbJake9D8CICPdp6rFUL4M7RXRN6ijvmtEMITIYR1tFe/7tvEffwe2hMEiIgAeHvnbwDw4wB+PoTwrU7b+1cA7hORw5u4/0Hr7nMhhA93nnEd7bZ4s4jMhRBWQgifH/B5vhfAkyGE3wkhNEIIHwTwbQDfr475QAjhsU55fcDzEvKig5Mfcq1zAcDcoLYQIpIVkV8QkadFZAnt1QmgLWsBwA+h/fF8TkT+UkRe3fn7v0X7X/ofE5FnROR/38K9HgTw3OUmaiKyW0Q+JCInOvf1u+qersQ+ABdDCMvqb8+hvTJyidNqew1Aqc87O4j2akm/6zznrrGvs30YwE+KyMKl/zrn2ofNk3QPmsMAfthd87VoT1Iv4Z97fBP38YcAXi0iewG8Hm3p8dPq2v9BXfciAEH7nQ96/8BgdXfc/ea9aK9mfVtEviQi37eJaz3n/nalaxGSSjj5Idc6n0N7heEHBjz+nWhLFm9C217oSOfvAgAhhC+FEN6GtrTzYbRXC9D51/tPhhBuBPBWAD8hIt+1yXs9DuBQn0nHv0JborgnhDCJ9qqGqPKkVaaTAHaIyIT62yG0bZY2y3EANyZc57C7xkn1u58LIUyr/yqd1YVh3oM/7nfcNcdCCL8wwG+vuGoXQphH207ob6Ddbj6kVvuOA/i77trlEMJnr3D//rqD1J35TQjhyRDCO9Buo/8awB+IyNiVnge99XfFaxGSVjj5Idc0IYRFAP8HgF8RkR8QkYqI5EXkr4rIv7nMTybQnixdAFBBe9IBABCRQifezFRnyX8J7X/tQ0S+T0Ru7sgfi2gb8LY2ebtfRNsW5BdEZExESiLyoLqvFQCLIrIfwPvdb8+gzwc1hHAcbfubn++c8160Vwe2EtvovwPYKyL/SESKIjIhIq/slH0QwM+IyE4RmUP7vV+6xq8D+HEReWXHU2hMRL7XfdSHcQ+a3wXw/SLyVzoreiUReYOIHBjgGufQrr8rTbJ+D20J6q8jSl5A2/7rp0TkLgAQkSkR+eEB7v8MgCOXPKm2Unci8iMisjOE0ELb0BsYrC1+FMCtIvJOEcmJyN8AcGfnfgkhCk5+yDVPCOHfoR3j52fQ/qgdB/D30V658fwXtJf6T6Dt1eXtJd4F4GhHevpxtO1cAOAWAH+O9gTlcwB+NYTwiU3eZxNt+4qb0TYsfR7tVQWgbT9zP9oTqz9B2yNM8/NoTzwWROQfX+b070B7FeskgD8C8H+GLcQE6sgvb+7c52kATwL4zk7x/4W2Tc3XAXwDwFc7f0MI4csAfgzAL6NtRPsU2kbMm+YK96CPO472Kt5PI9b7+zHAuBVCWEPHs6nzTvvZJn0E7bo/HVSsnBDCH6G96vKhTlt5FMBfHeD+f7/z/wsicsmebLN19z0AHhORFQD/AcDbO7ZAV3rmC2jbI/0k2pP/fwLg+0II5y93vIj8DxH5abW/0oljBWkH71y50jUJuV5hkENCCCGEpAqu/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUwckPIYQQQlIFJz+EEEIISRWc/BBCCCEkVXDyQwghhJBUkdvMwXNzc+HQoUOjupehIX3+Hq7qXQyfY8eO4fz58/0eb1OwLreXYdYlcH3WZ0tvJ1RodmhvaXSks28GszcI10O/ZV2+eOoSAB5++OHzIYSd/u+bmvwcOnQIDz300PDuakRIq6F24uJWkOt7oevBBx8c2rmuy7pUhMymmu41xzDrErg+67Mast3t9Ub/oXSqeO3321T2zWZN7ag6Suib18MHk3X54qlLAKhUKs9d7u/X/qhCCCGEEDJErso/n5MW0RIX2/r8q78HNzutqcfSszv/sPra9ZYt00vtm1l2v15mw1cb/QqT3lHPq1Z1m1RfmgJcu9niKlEa6nLgph0SXrhbUU18b6ouCurPTfejmvrD8eW6KZsrx3NUsu6H+j79Su+AK7/Xa71vWadJqluNH4/1+9Tb/nxqX5LqwJcl1eWAvCjrUr+XhHed9D6l2f/bGrJqvHTnl/pG3HZlIad69GbqOQmt3Az+q4Hgyg8hhBBCUgUnP4QQQghJFZz8EEIIISRVjMTmZ8MJ+OWM2nd6Xz/PLAADa5RorZndYp/DpNV0f4jXy2SyGBhjDe/ueVD7Eq+lqv1WrjT4vVxlBrXd8STadiXYHGgPPW3nE4K9um5ymaytg1ySRp5Aon2CucmE+78GvNIS7QeUl0fIFkyR1/Q1DfXvpnVngJWVeMWWq6d+babmxgx9zzMl2zeNDZ7vR7U4Foi/tu7jrm4b+Up3+/SqHWuqzXiNG6bsO7raDGwLktR2ve2OHmc30T+gx2R1Tv/ek+4rSMIT9bMpgrNLSXjWpLHGt/ftpOe9D1iXPd6wLfU7VybNutqumbK+1/P3pW1w3NgmjWrfew7ZfNxx43NIqGfTrtCfrdQlV34IIYQQkio4+SGEEEJIqhjJmnwh4xeo1DKoX6ZOcmdvadmr7soGk8sS3QLV7yS4V6GX29wSoVnu8zJeQoCokI+CXHDSlr6zhLd3VdAKRJKb/6Zca/X79HXe1O/a1rOWKgp59c78ew/qHBtVW7axFLcbtsy40ufy6EdQsgjglnH9scrls6ftbMNSu5Ec3bvX9+PdzVshvuNa0/adtXqUkBeqVk5eq8X9M6v2fS9V4/U3GvGcEwXbV8YLsd4reSt77RqP96zd3gFgqjTZ3c5UV0yZkb2b66Ysp97LvvFJU7ZY7S8LbWvg1ASpZMV1sVIu3mfLDfsNJVPVXajthtpfqtrr1VvxnEXVH/yYUcjG4/y3Qd2WuUcAyKH/2J0ob+q+6cwS9Ni93eOsuX6S7OV/p/twy70XPX42NmyZHoPrbozU45SXxDR6vPRjmS7zZiTF8XjLOfe7pDHR3LN9HjPO+nFtAHMDrvwQQgghJFVw8kMIIYSQVMHJDyGEEEJSxdBsfrR+2WMnom13Euw9etBuic4lUpL0ReVK18hGO5Ga07PXlItuzmnRBfUQxlUfTh91mqvWmHtch9XvxOmxGMCO6Golt9Zux5Wcu2qSK2yCfZWxlUio81Z5yuwvqFe9vtH/2mWl/06UXKiAUjznWsOe4+J6tANZqVnbFW1rMu7qZEzZMYw1V01Zphr3W6UJV9a2Q9mUS/EmSbRl2MR119W7WvZ2PSoR6bPzNtSEtus5Nm9ta547H9/NhZX+tgWzyq5nx7gNXnHPvmiT89K99v1W8vF3JWc/pm2/MnV7X7qveruz6cpM3/u8NJ5dtb6pqs81VwT1b9lVF35gYSMe7MfBqqpLbcsFAOfXauo4e86istGZq6j6KufdcXG77MaTosTreZsOqS53t3Wfau/HslZhzJTp8dOPJyjGY7291qjH2cTzuntZV0l/mz50jLIXzWb7f3+8vaG28+lxda/GPtxai/0kVN13SpGdmrV/KMe+GNy4Fwrl7vZKy047qqrN+e9w1tjQWrvLsnrWbLDflIz/vl4GrvwQQgghJFVw8kMIIYSQVDE8V3clZ3k3s0yCi53BL8kP6HIHJ3vpJeyLarl30S3dr6j9pnOX1G6XE0V7/plSPP94oX/E6qRlR0/Q59lO91lYya/mqiTvI1or9IKlNPyzr6sytySppMPgXNGnx+LSqvaE9W7ZZVWYq1sZxkSwdUunut43Gv0loax77JJavg95J7MlSUuXJNmkyLYvEO+yqyW2KqwksaEq2Eeo0LKX7zvLSnPxEZ4XN2IfWHPaTEHVU1m5sxecm7Pe92V15XbvFBzbjxOyuvdIHqrNZHx02pwKUeHr+ir3VT1MhGArTI91yzUv70Ypr+7CFiwqmXJpw8oHy7W433Qve0LpWd5FXtNo9f/M7Cyovun6vhlL3ZjRWlcy2LqTxMpK2vLfBlWXPdH4R1yXPf1SPV9T3DdTHe1NPnS/LLiBKavMPLJjzoVcjQPZVfs+m/Nn4/aF0/EnG/bdBhUuItQO2WsfvjOeY3ynKVtoxOfzY4k2s/AhNeqqbM+EHbsy6r1UxIWgGCCkCFd+CCGEEJIqOPkhhBBCSKrg5IcQQgghqWLLNj89rromVYQr0zYd3mXdhJz3diJRl9xMhmat52eUK2XWXbuu7ktr4oC1JZirOK0Ryi0Wlry5L6s3F5Vgn1lftOfU3vPKLRAAmpn29UYVft3Xl86C7rVo8zufNVhr9DWnw6vnlfUlU6Z15UzB2lRk1ua725Pa5djVeWbxQixanTdl2gVzfPftpmhNubP70PvlfLzGeN5er4jYXjLry+hH8OkQum6qo7P5SXKj97ZS+pE3GrZwRdmNJNlDFZ1NzlQptpmM2LZ8aEfcLyp7hamSd4+OZSVn1zCnbBkmC7aPmTr070Fnk254Wz1ly+DHIWWLEnxo/gHC6L8Q5jesfcSUsj+c9zaMqr68rdV8gs3PqZX4fD78QE3V+65JG3LAXFvZDT1ZteNCKRfvecalI7l5R7TBmylNm7KKcpcOhXFTllHu3trtHQBCU7nP+3AH2q7IZxcfcV1upsdrGyrf9fRnbN0V+nGqH2HpgtmvH3+iu33q0w93t59/6Kg5rroU28eue6xdz63veGN3u/Ty7zZlU2rcrblBqK76nv8O6/egxxUAmNRd0dv4DBDSgys/hBBCCEkVnPwQQgghJFVsfZ0vyS0wKVO7R7sh+6jAZonSZ4LV2WXtknlVLatp+cq7s+vVNx/dVy8TL27YpTi9JF92y4x5tezuJYZaRkWf9UusypVT8v2Xl0dCgmzoYmcnRgs2EUR9tNb65SOIAta1Ei1bD9md++M5VD0Hlb0bsC6sUii6MiWlOUljvBDPKe49lJVkmqnae9ZynDjX+lYxLtdL0UafjZLKcEVMQf9l9Worlvjo6+tK6vIRfLWbqV9y1v1qwklPOkpwxknNk8o9WoeJKDvpTCtdfplcR4EtuQfK6kzgvl3rMcR3MSXT9kS/NeErNp89erPousz3hMuPrNftezm+FPucj9S8nlAnOjrzkely3zLt2g7YcA+ai2v2HT2/HO9r0bnSn1YyWzZjx/i6DndQsP19TI89Xn6vK4k9oS57GGHU9cuiQy+4Il1HIrZUhyLxphy5php3XbiRzEaUBxvnTpiyxcePdrdPf/X57vZD3zpvjju+Ht/f61z09t33x3MU71yw1+5EtgeA2bKVMHXXz4o1ewjqzcyUnBlJQt+g7EUIIYQQ4uDkhxBCCCGpgpMfQgghhKSKTQnWAVF/T9TbPFozdzYdSMr2rTT64OxgQj66SIai1RBXlHuocf90Oni10d8NflzZMVS9a+hytG3Rmb8Bq8f6N6TtK4rOnT2zpDLpKpsRIGbuHZVzdJL1SZJrdI9LdZLOqmwjJOMtidQp6ta+yoRJKE93t8+JrfO6emd7Zg7acyjd39+zzsjeE16/plIeOHfa1kIMBx8a9p4zs3tjmbdN6toAjTC9hfS3n/FX1bY7vvbGCv1d/bOqIYSEBlR0diHGjkrZIITg0hBkovZfc/Z+uj/2tDhtR+HamQ/NYK6n7QZrzl5N2U6EwtW1C/HvXddXw6WU0O7mmYwfz2L/OzRl7Sr2jsdn31G0v8usnOtuZ5fP2JtrxOu1JqLb89zsAXPYhqqv44vWTkSn1pisOtfzoLedPZ7K1u5TX8iaCiPi7YFU/+9ptiNIb2Fs8RLaX86lDWqp+vOZzk1GJX9O1VZ73PzVOJgZnzZlEwd3dbfnbpvrbt/7zII5bl8pXu+O19p63vGqV8Tb2nmDKdNpYXwG9hllT+htaPV3uRBcCAqVcT4UbNqiQeDKDyGEEEJSBSc/hBBCCEkVm5K9BFbW6ctmXAb1sT7iptpvOSmooSJ+LrgoqGdXG2o7ShnFnF0G3zsRl+IOTpki4/p+bHGjb9lZFxE1n7BcqV3tCxUr4020+kcl3cqS3pXQy7F++TchQXMiRmZwUqRuEdkxJxOtRsmvVXeuqWo5uKmWuh964qI57LmFuJz+V26xkUdvm1CZy5WLOgDIRv/ozEYmcVIIlNTV8lml9TP4ZekRR5EFLhOxWy3nexlTt9eGa687K/FeC+vuvblwBfb68SKZDRvJvKkiceuwBrmZXea4xnRcUg+lHaZMu+A7Rdr8a67oMrCbKM4JEeo3xVXO6q7xkuKu8TimVJx8MKFlr0kbGqR88pHu9vrDf2nKzj7yeHd79YR1ex5XUsme739rd7t1235znJa6njxn+8radKwTHzKhqKJpWyMB2H7l67Kojs5c3SjOl6Xf91CHeXHSXUFLvT6kiJJ7ktqfDwHTnFL14rKuV/Yc6W4fmpyOv6nb8eu2sdinDv3Q99oLPvj27uY5900eV7JlKTjzAiXV+S+dzhTgx24d3iQ47b3lQoxcDq78EEIIISRVcPJDCCGEkFTByQ8hhBBCUsXWBdDNZHVPQmmwXo/VNiQNl9V3UWU0PrFs7UTOr0V7DJ2Rff+EdZmdzSu3W5dlvbUzaqJaLweAY0rDvuhSX/gUGuacqqyUtZrkeIL799UmM6AXtnepFl2XPvu1/p3L+K5pLl3sW7ZxS3x/H/jsc6bs4oV4ztccmjFlmUzUijPzJ+1J9XvP23vW4RSQt2q0zj6f2XDP4zLTm3O6/4+ExP5n60zbhowXbFlx4Xg85TNfM2WN+ejq31qzdlP15RjKvrqwYsoaq9bV+RLTd95k9ssv/Y7udmH/rClbb8QG2nL9ra4evZD3NoQqlYlLe2BSoPhxSNkOJbXrUeDH0ozqc/5frvvU+DbmbH70WJd7+rOm7OLHP9rdfuqPv2LKzn0z2vkUXVb3u47s6W6HGx/obv+5c4/+8MOxz6249BavvDnW7W1zdozXzzDtsxstRRd8qdr+p+tS24X4fT9+jYxL1/G2O9rt3t+LsQeq9S3rsZNVaUAuOLubVZO+xo5nO3bEepg6EtMNzdz6hDlu+p6YnV1e+QOm7HMnYz1MuVQo5alYgToNEgAbCseHLajF8aJnXqFsY4NPYwLa/BBCCCGEGDj5IYQQQkiq2LSre9c92kseCUvtJnKu+12rNIF+NFWW3zWXcfrCelzS85mCdQTTm9R6aeHol8xxtWce627Xly6Ystz+uAx/013fYcowFZdnHztnl/VPqMzKPjL0hnoGnS0ZAPaUVMRqF236qmcbTsDcmZcHdN369iGqDTj37+Z8XMJefPKYKSvNLnS3V18Z38OTj9pos6K0Oi11AkDmYlx2bzmJKjMd3XVbZRvvoFVS+255WUcjl7qVcnRG+5C3TrpbDSOwKfwSunrfWVdnOgt6dsm+0+Yjn+huzz/8dVO2ciLW2foF2wfWzsfI2Bvzdhm7ofpqfizW00Enj+3ffSje18whUzat6qnass+qo6j7d50x2dn7Swl+TGqNzV72OEAv01+NirXP5KPh6jAkk601U5Y780x3e+ObXzRlFx57Npa5+irPxLH0xu++1ZTtfu/7utsfVmryr338KXPcykI856699t0enIn9Y58zS5jLRVkje8HJ1UtRds0UXR8rRMnDy5Qh31+SHtk4e+m8A0bABy7TPvXp1DPUxD7f6eX4zh47a8e6x9W3asaNkW+6MbbxydUoY5d3TpvjCjff293+5rJtf189GU0W7t9nx9Ksvk8vbenvgc/yoCO2u0jvpsyNs8zqTgghhBDi4OSHEEIIIamCkx9CCCGEpIrhxfrWdgZOt9OanrGjALAOqz1qaspvdblq3fYurEdtc/e4dWe8Wdn5ZL7yx93tcw992hy38ER05V27YPXR2Tuibr3XuS7vu/uvdLefX7Kv8MJa1GqPXbS2IDWVRf7MrHXFu2s2aq5StfYP28lmco83pX9zEm3H5FJYVE89390+/6h1YZ/YH1MbrKo2sHjiWXPckfvv7m4fmLBtqvGtaEeUmbBu8Not1rfNlkrR4dNCiPKQz/o0CiobvLdvGjSMwAvC2/woewJtjwQ4e7xTT5qyha8/GrefPG7Klk/GNrp8yrq6r5yNz19zhjcF9QLKOk1Fzb6noGyzcgv22lKP5y9O7DZlUClssi3bzjIJofJ16gtv83N2Pd6ntikCgMPjl97tkCu2j82CttHaUeqftT6zYp+vpUJIBNf/SrPRPfrAg9auYubWg93tyR98ryn74/nYX37lL2LbuXjGjl/TO+NY98CNNlXJS/bEd71/3Pbb3Nmn4/2fs23APIPr09D2Hwl9ISnL+kjwqTW0XU+PnZ5yg3f2hsutuH/KpVf6wvML3e0/e8za8J06H9v/y2+dM2Xfr/dzsR6Ke/aY48JsbA/fPG3r+eySGkv2mSKT3qjXPkf1KVcnOg2ND1tg3ucW7LW48kMIIYSQVMHJDyGEEEJSxZZlLy8D6GTv3u1dR26uu0i560rKqLqTbqgl5jOrdnmvpZbT9407991vf7K7feIj/727feFb1l3ywpNxabi+aiM150px6W/XhdOmrCjxnqdK/V9h0y35Tyv39lLWzjtrKutt8SpEkW0GYKUjK/posEnoJ2q5ubOuvh55R7slOnfzVeU2veIkFC17fVst25ZnrNxx9y1x2bZ4zkYlrZ6JS+b5kov8qTMfu+VlXX2+LrP6ARMixWo5pb3fXta9GurXIOjI5g3XzoMK1ZBx0ZJz5Si5aHdoACiMxfZbmbNL3CV17Pj+WGdz991m76sSpZiwYd22pRjLMi66b0lF7PYZ5TOrMZxFWLFl0O3CyRPrKkTFF0+43+1vSz9+7HrB9GtTOvKv/4mPBGxOp97LxLQp23H74e52dtKWFe97fXf76bzVMv7i8Sg961ATt99uM4a/SkVxvmmHiyysxs+sj/Ch30HOha/QpgiZhPHLhybQYSh8+IpR0XkOp5hagw//zSzE97Rct2XHFmM9/+VRGxH/E9+MIQCOPm3DtzRqcSxaOTRty3TQaJXhPTtlo6uvTsTM8E8/ZqXIsyrMy6LLfLCgok1XczaSdzkfK77gGoEkSJN6XuHnHEmhAi7BlR9CCCGEpApOfgghhBCSKra87ueXKPWuX+qHisy4UrPLU4vVuH9xve7K4pLXkovifGQ6LqdP1qxnw/LnP9nd1lKXlrkA4JyKRDues/PA0kxcdszO2GXcmvJqarXsPefVEuwNc5W+ZXn3ArWK4hO8JskqWyUjg8ldfjFfqz++LCirfUmIUh18+9D3lbceLLtecVd3++Hno+SwY5/1GnntzVFCqX/9I6Zs+WhsAzN7j9jr6XtpOY8B7dHldLxE2SopwevVSKaYkHTYSwQ6mnhmfNqUTd12Q3e7vMuWNTf6LyuXdse6yB+0UYFzc3u72yYSb9HJkUrCafmovPp5vDfd+kLcdpG3tceTr2vJqTrLWomlWY/tOuPa9VSx3Uh8+3jBdJ6r4Twoc1o+9r/RMo6XgpRslHXeUaLaaHbWevc0x9XY5y74ahUV+M23x0jpeyesV860krZ869fvzSfinN0V207GRV/X7SPjEiUnySFBtxcve404kr6POK4jFldbtv0sKYlqwb2XzxyL37E//bqVqk8+F8tWF20C0ZKKqN5wN/OU8kye2hslaG3iAQDfvhCP+9IzVnJbUt/vJ867ZM8K3z5my/G+/DepqLw3ywmSmJe/kzJHdH9zxSMIIYQQQl5EcPJDCCGEkFTByQ8hhBBCUsXwfP20Xuq0U62/ra/bsmUV2fXUitUoz670tyu4a1d0l8uee8aULT51Ip5fRaJdX7DRbXW02dmbrQ4+d2/M6p45dJcpm1cabDNY7XSP0jO9XZS2F5guWbsCc+zVSRDdl1aSy7rCF+XUwT02Ycq2RpwdTHlXfPeH32jfdeF1P9Tdfu4vluJvxu05XnUw2gQsfvxxU1ZbjnpwqNk2oKOsStPab2Xzyk7JZ6nXO95WSLd/b0Ygl9r7CCvZ3au+UiNr7Wdy09F1NVOaNGV51e6LVRuCQNtVtJw9Rn08hiE4vmptcnRk9hUV5mImY/vDZDFq/d79dUZFNi44d3aNiV4NIKPd2QvW7kBnbl/OWFu9Zy7GZ8+7DlHvdJYwour0/UibTHq7jVxGhdIoWHdiHTpASvb5sBr7VWvJukfnL0Z35sO7bdiCQ7dfPiq91F2EetUfWuPWflJHz553ti1Lyh50qmgjEs9M6JAGS6ZM2wB5+0k9Dl0N20qNN7FsqPAm/juiWW/Y97Ki7F/rTTvAZFUIleld1o5ufEplg2/Y3x1diLY8M8oGZ8VFXv+jb5zqbh87tmDKxiZjn1pYs2NpVV0v6+zmtK2ofw/m8Zz9kW5XdRedP9sz8PbClR9CCCGEpApOfgghhBCSKrYse/WoIcrtzCR2BIBKlDXqzs15US3hrVTtEts5JYPtdMlLK2oNsbWy0Pc+i2opzjlLoqAS6e1/zU2mbOKBB7vbGzuO2HtWETb9Et5uFd227NY59Sr1eMHJKCN2s0zCJ2us5LTfvb2vjFoa7mkDRkJyyW0bUW4SF6117LYor0y46LPnyjGq7LmlGAl6YtLKNzeqVf4zC1aiEbUUHKouseeA7z2xvbd8Il+1tO4iI8ewDyOM8eyjmyaEmqg19X1Y2auej3J1qWyj++pfLbtzHj0eZYgvP29lqUfVfkGFl7j7gO2d+1T96mV4ALhpJsov+yfs77QY2nTyqk5i65MkNlTk+Ysrdsn+0TOxPc1U7L1ckuR8ZIcXTEeSqbnI0SdXYlvbcNKFHhP3jtv3Ui7GOsmM2XpunDoaz3nyhClb+da3uts6ASoASD6+X58sVZPfF0MmZO96rSlrZqMMdnTehibQ4UAOTtn+rt/KVNHJtXqMclK2xkcB1u1jFPSYCajEu3kn01TK8V6qDTuG3KZMPpZdCJhTs7EdH3DRtI+oMt+nJgpRRjy7Gr+7n3fS1mceOYV+HFZJal+yz9bJPbtj2VTRhZVR40DR6bxm131TtHu7ZF1dFq3sezm48kMIIYSQVMHJDyGEEEJSBSc/hBBCCEkVm7L5CYhaa09GYZ1t2Gey1pl0g9UaNT78dy7Bz1oX9YTlvzm67zZUGH4fkn/iUAzJPvcdrzdlrVte3d0+t2a1Ru2N57O6l5S9jL9/vetthQxXIwWCouc9J4QtMGlM6s5+RoeZ97Y0OuN00brMZm96SXe7OWnD6z92LtqPabfO195mXWazK9EeqLLThi1o1pV9TtHZDih7mJBUJ96up9HfRd6c37vTjsi2y/RN6W9rtuoyRJ9bjfd+0WViXqv3T0MyVYzPVXW2J8cWY7t41Nn8HFX7ZRUWYrrSf1woutQz2g7GP09Wud3XCtbuQB+7XrODTXU91ufj563N4gVle/iKA9OmrOkHrSGg69K3lqCsXR49a23bdml7Q/fOCpMx/EDehS0QlSF98Wlr8zP/1Jl4bRfSIafCdWSUcUZh0vbvA9+t6qFg3a8XVlWm8Zptb3pY2uuyFayruvShEHLK3iOzaJ/H2H1dhXE26ZupbfGS7vPAuB3PKvn4Pm+bte9Tt8YJZ1eq933IBt03np6P/Xfd1cmksiO6Zbe1q3nnyw50t+/aae2NxmsLcafhMrWrXW8/2TN+anT9+VQlA8CVH0IIIYSkCk5+CCGEEJIqhhbh2SwhOhdCUdmVy87FVMtGF9ZsRm9Ny0V+1Mt0rR0HTFnlltu724WJuPymXTMBoHBjdLEON9xvyi4i/q7pZDztpu5d8/QSrF8Q10f6ZcfEiL8jkEpE3U9vcveEObGWN1025czavD86/kzVe9NFeQ3FuKZ9oWXbx/x6lEnuUVGc73WulCEbl0vHbjhkL67qz2etDnl1PZ+BXeGXY70M1vecbtm22pFshl2juj79NXMq9EQha9/v+bXaZbcB4NyqkoydvLPLhZ7QPHs+RvhddJFeQ59ItuMu4rmOlL6j1F8SW67avrmmxgXvJq5llUXnInxWPevReSt77VJu94enR+sODXTqstPP8i47u454++0zNpLyhnJl9r/DjihF7Znab4pye2J/Kc3a6OiF0zFzd2PdvrOmlkSUq3S+4qKI74zXW8xaqeTEUpTgFqu2rcyoeveRjNfVGDXuOpMJq9Fw7S/JnT2hT28VQYgu9Vk7vujmmXFlOmJ2tmrrWY+eO92YJbX4rZVzNlp3c/5sLCtbuax88J7udnUyjrOvPGwlt/v2x7I7d9m6vG0svr/cyYdNWdhQbukle20o2dV/64xMWSj3L3Pvb5BvJld+CCGEEJIqOPkhhBBCSKrg5IcQQgghqWLLNj9Vp6cXlc2PDx2vXdLKeWvrMq60Yu2qCQAbKpvtmnNp1Rnf9++1dhzjd7+uu52/MdqheL23ORVD9i/AlmlXSu9qO6aMZJLc2b0XrHV1t2XaLsXbbIyCgKg5J2bA9e6gCW7wSS76IR/fr86gDQDrIbaBNZfiZL+yt3jjrVHtvmXWulKGQvxd4eZ7bZl2S5+19mEtlf26x/VVPV9PWSFe37tjmq7h2sBVSWLi60Xtl3O24ekUAvmscylXLuwLzh5oTdl7+AzR2j123IWCmDo43d2+XdltvUzZEgDA4emo7/t71lfb8HY9KizFsstIXVcdsuU6p+6b+10qhfFCfAZvs1TOZ3t+PxQ67a3pnk9HY1h3z3dB2S3Vmwk2hDusrcYeZe8480qbIb0wGdv5+tn+Nn2l2Vh/Y7feZgtvfUV388xqf1srTyUfx4WCa5t63K04o0VZi3ZEIePsSJPc20fi+i7R1sen08ioMBvum6lTRGU2bLiI5rmTcfvCaVPWOB/3V5W9FgC0VMiPySP2m1lWdjh7j7y8u53bZ2MM6HY+W7LvNnv++XitJWtvFJpxTMi49EbGFtePXWps9d/vnnmGOemV65IrP4QQQghJFZz8EEIIISRVbFlfqbllYy0NhYKTJLJxmcvLRDqbLCb6uxqfWLbRhI1r6oJdRts1Fpf0KuNR2mq0vOtrXGLz7uz6eSZdpEztzp7ksJ7zIod2pfTLckmS0WXOPQy6VdHqv9TYg75PtwzZUrKlX27WbULLXICVGMtuCfvIdFzaPDQVt32oAJ09Pew8bG+5HiP0endJc5zLGgzlLtwje2mZ151H73vps9C572GrJCaSbIIcmdSGtMwAAHPjsT4LTvrVslfTtZ8dyg3+RucOe6uKSHv7XGwTe8ZtH87X47K/lkwB4GI1PsUxJ8edUtGYfYRqHVVdy+2eopNYblJRbafdUn+5MxYM81+RSRGeS2rs0S74ntNLG33L6q6+6juiZHXgpX/VlE3OxCj4lTPH+p4ztzu6y7f232nKzmenu9vHLtrwGFp2na1YGUNL3kUnfU4VYz0UGzY0gcnW7lz+zbjk3aNHgOmXblw1pg9J3wMXEVm7jWv3dQBYOXG+u71xwcplGSXfagkMAForC/G+Lj7X3d5dnrb3pQeQFSdZqu+bd2cXVQ9ScvMD/R1x76ilxuuQd2O3D+ewSbjyQwghhJBUwckPIYQQQlIFJz+EEEIISRWbsvnRIfQnfE4E7RbsXdBUxlU/29L2M95lcaeyIVl0LtAnlKa9fNKWHVKuqnMVlTk3IWu3z4Cr3dl9CgsdPt0jWk/34dKN215/vXm9j5vqMG1+6i3gdMftdO9Ygqt2D/G91IrWPbmp3IB9FoNaPajjrM2BbgOTYnVknTJDhzBvla27fENdT0r2viRXxVbYatZn3VycqcLQ7bYuIVApOJxuHlRbdt62mClHW5tK3taLtquaKVn7Ge027lMP6D53ZMbq9EemYlnp4tFY8PRJc5wO/RDmbLqSwsRB9EPb+WRdf8+piplyLvjjKku9d4PX9KalGT66LmtN2wb1GHnvXpviZUml7PCpSpY2YpqHowv2esvKfmt9ztpqHLklhg0p7jllynT/aFRiGoSFhn23Z1bitf37OzgV28dU0f5uUtn1zJWtrVWpGdM4ZFatW7UhKdv3CNIGeXSqEm/X02Obp9H3XbQ2MqK+HRmXpqI4E23sxIcHKMXfZSv2nK3VGOIgezFmmM9UrN1QUKFB/PPo775M7TJl+thW3s4P9LjeM3fQ30xn+2dsuzYRdqV76iseQQghhBDyIoKTH0IIIYSkiq2HEt7EMpN2NZaii7KsZKKMcwDWq3Z+SXRFLVtfXLeZe799ProCVvJRotrvXEMPGFfK/hFEe2QuHTG44SQV7VacddmotWtegmujVxTrCcvww6DhTt8TfVqhM2Vf3LBSiL7Nftm7gV539olMXK7Pn7ZZpRuno3ttVmWHDkW73LsR4tJpM+eWgotqKdjXpXZvzwy+LJ1UI1o29NnQC0kvd1g4uVVHoG4FW2elXJQTJgpOClL71Ub/dzHp3L9ny/F30y2bkTr33CPd7frRb3W3m8rVFgCy49Pd7cKYlXcmVITw3ePe3Tsu51/csOOCHkNKOS8n6WjCto52KMmlvOGi5rpo5cPGh+fQUt7ecSsR6LAhk07WOzofZaINV5fzavx8+JSN8HxiKfarvRP2WY05gPJgbwbb/rQ7++FpW19aBvPmBTocQbFu25GWw3sygWt5xLuza3ODpOj1oyDp/F5C0nKPe4bMVKyHbM2OZyUVPbm4w4UAUNnT9XbPbapziovGjGLsiy03BoeCy9auy3SduGdVEWd6vn16DG64tRp9Zz7K/iCjLFd+CCGEEJIqOPkhhBBCSKrg5IcQQgghqWJTNj86VHei255Pj1CLgnAhWBfMyWLUM70WXVDufj4txrhygzy/ZrV97Xq7Q7nyTpesfqlD1e9wdgu5WtSYpWZDsktdaaLOtiVoNz73jloDuk579+hsdvgpEfIZYM/Y5avfXMf5Rtdb8Tc15xOvzRO8R7CuP1+XmVq0Rwh1ez3Rrpw5db8uFUUDsR3VnOu1iLLn8O1W7/s0H7pN99hoqVDuCWHyvb3WqGx+Anp170vovjnmRHVt++JtLhLTdKh+nNlYNmXZ+XPd7ebpo6asdi660dbPn7ns/QLWnbdx3rpYZ4sx0/TeHdYNvqjqYkfFvo/larR38nZnU0WVzqboxgL1WnyI/VGkntF1ucOZZpxRWetbbuzJK5u1ijPVmFDPtLhhf6ftuXxKEP3O/DirbY70eDzt7I10m/PhTDTeTNC0saptY9KM99JjW6lDYpRsVnITksXfQKePD70u+4z79pvpjtE2Py5VUEbZw+WcTU52NqZ2CrX+IVngbXn0fWm7qLJ9f/p9enu3ZZWmaMMZkga1L2LbmP4cTLi0M7pNi6sYE/rElQ0SkYIrP4QQQghJFZz8EEIIISRVbN3V3aNdvH30Yr105eSKvNrPOxddLSntcHcaptSy3Yy9nnENVEuL0nCSSiPKWbJk3QK1rNIjballyJDLubL+r1SSllx12RYjCw8LfW/i7mVcuaVn3Nq6lrOKLbvkqqOwZubtEnbr3PHudt1lKQ71uLwtqixfsyEGZidVRFHX/kIragc+gmhIigCrnz0ptIMry6qycu7a+vdFqbHat0yqVbevpN+mlTxEZV1vLVn37+ZqrN/WhpOMlYxZPHxL/LvL9KyPy07MmLKWet/ZlXOmbE5liJ4tW4mqrtzi/aq4VvxOr9lxaKmqxhA3ZN48+sTghhklLy1XbbtT3voYFysfrKgI+XUnV+ss75mWfTMbjShR1Nfs9bREpsMkHJm27z2j5Ef/3rUkVmq6tqLCiPgxMaiI0l4WMm7VCe7l/l5GG1Dkcjegvpn+u6GlPBWqAwCahdhXpGyj2ZvvlgvDIur76t3nTbRkfZw3E0iIqlxUsWlaPWNi3PaZFrx8a66nzpNzJhgt1ddtCxisLq+tkZkQQgghZMRw8kMIIYSQVMHJDyGEEEJSxfBsfjQ+q7Talpazu9G2BE1fptNIWJuDRDdkXZZwnNGK/TmULUjL67F63z+r1jN7yq7DuabTbjPVqMuPez1dacX6OMDWZVhdsJdoqkzcs3vt7/q4ZErehXyvR3f5lrf50fXsXUp1nSSkHNlUOhd1bOZaqPMEezxjg+f7h7ItQM2Fym+pzNKTO2yZs9Hph60X11e0G2tP9uj+brpmDGk4t1kfykCh7SP2OruwMWVDcnrF2gNdbXSohFum7HtZqMf9prOjGN8dbahqTWtfVVVuyD5MhD5P1tlq5JWNhw6T4O3cKvlYVnCZxgtQ9iW+3+r210p470l97Frof/1I+jbl+6efMHasPd9MZSPp35myvwsrJ0xRc3lBnT6Ox5mSS1mh+p5/syX1u/FdB0xZqxRtk3pstFS9e/f5Wojjgsv3bp8vwda2H9dwyyCEEEIIGT6c/BBCCCEkVQxN9jLSli80GWpdBl4lL0lwbsitPvKVJ2lpc9BMvUnncFLJwPJV0lKcX+Yc7IzbjomYmiAF9WS71ku1swnZ0hOiLOsl3aaL6hqSsjUnSVv6HG5ft+Mel8+E311rS+2mb2a9JKj7psuMrMMv+BABYTru+OX1pD7Xr2wzdWZchL2TqyKbIDt7ma1gpSCNjnE7PnOVfduTcM8wrauo5z1L3zJjUpAQbqSHJNMDfVhTXbuVUJc+7ERS1oCksgHZ7jFXt8cet3v9Pp0EpkO2hHz/dguriEHUaTKu34iSt1qrS7HAmZtItn9/09nme/qlludyTkpT9eeluqxqEz4cQFJYmUG4tkZpQgghhJARw8kPIYQQQlIFJz+EEEIISRUjcXVPsp1AQgbs4HVqLRtu1Y6in9v7ldiq++SA19huvXlQesKgJx2bcFzos30lkjLF9zvOo3+3mZD210sdbYaevqnbsrcHSrCrMOdJCjWReDNbtONLYghuz9dLvSeOs0l10hM6QNvLWfuSnjG53zkHZRP3NXAYigReFHXp+55KCdKTSqqo7Gm8nWJSuIA+9rU+rU0r2z/MRGJrSLDdSWoDWXXWF2rj4+HKDyGEEEJSBSc/hBBCCEkVEpJcGf3BIucAPDe62yFX4HAIYecwTsS63HaGVpcA6/MagH3zxQPr8sXFZetzU5MfQgghhJDrHcpehBBCCEkVnPwQQgghJFVcN5MfEdkjIh8SkadF5Csi8lERuVVEjojIoyO6ZlFE/quIPCUiXxCRI6O4TtrYprp8vYh8VUQaIvLXR3GNtLJN9fkTIvJNEfm6iPyFiBwexXXSxjbV5Y+LyDdE5Gsi8hkRuXMU10kj21Gf6to/JCJBRB4Y5XW2ynUx+RERAfBHAD4ZQrgphPAyAD8FYPeIL/1eAPMhhJsB/HsA/3rE13vRs411eQzAewD83oivkyq2sT4fBvBACOFeAH8A4N+M+HoveraxLn8vhHBPCOE+tOvxF0d8vVSwjfUJEZkA8L8B+MKor7VVrovJD4DvBFAPIfzapT+EEB4JIXxaH9SZzX668y/8r4rIazp/3ysin+r8y+JREXmdiGRF5AOd/W+IyPsuc923AfjtzvYfAPiuToMiW2db6jKEcDSE8HVcIRYX2TTbVZ+fCCGsdXY/D+DACJ8xLWxXXapMmhjD9ROb8Fpnu76bAPAv0V4s2BjVw71QRhLheQTcDeArAxx3FsCbQwgbInILgA8CeADAOwH8aQjh50QkC6AC4D4A+0MIdwOAiExf5nz7ARwHgBBCQ0QWAcwCOP/CHifVbFddktFwLdTnewH8j63dPlFsW12KyN8D8BMACgDe+AKfg7TZlvoUkfsBHAwh/ImIvH8oTzICrpfJz6DkAfyyiNwHoAng1s7fvwTgN0UkD+DDIYSvicgzAG4Ukf8I4E8AfGw7bpj0hXX54mIk9SkiP4L2QP0do7x5Yhh6XYYQfgXAr4jIOwH8DIC/NeJnIJGh1ae08+X8ItomBtc014vs9RiAlw1w3PsAnAHwErQHxAIAhBA+BeD1AE4A+ICIvDuEMN857pMAfhzAb1zmfCcAHAQAEckBmAJw4YU8CNm2uiSjYdvqU0TeBOCfAXhrCKH6wh6D4Nromx8C8ANbuHfSy3bU5wTaK06fFJGjAF4F4CNyDRo9Xy+Tn48DKIrI37n0BxG5V0Re546bAnAqtLPxvQud1KjS9gQ5E0L4dbQr634RmQOQCSH8Idr/0rj/Mtf9COK/QP46gI8HRoV8oWxXXZLRsC31KSIvBfCf0J74nB3Bc6WR7arLW9Tu9wJ4cojPlGauen2GEBZDCHMhhCMhhCNo2+O9NYTw5dE84ta5LmSvEEIQkR8E8Esi8k/RNqI6CuAfuUN/FcAfisi7AfxPAKudv78BwPtFpA5gBcC70bbn+S2Jaa1/6jKX/s8AfkdEngJwEcDbh/VMaWW76lJEXo6258MMgO8XkX8eQrhriI+WSraxb/5bAOMAfl/aPgjHQghvHdJjpZJtrMu/31nFqwOYByWvobCN9XldwPQWhBBCCEkV14vsRQghhBAyFDj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIakit5mD5+bmwqFDh0Z1L0NDtvCbMPS7GD7Hjh3D+fPnt/J4PVwvdVltxprJSnz0PBp9fxMym2rW28Iw6xK4fuqz3wNfD/0viTT2TY6zV+b6qctg9gbheqhLAHj44YfPhxB2+r9v6itx6NAhPPTQQ8O7qxEhrT4fxoSP4vVQkQ8++ODQznW91OWT87Xu9kwp293e3Vro+5vm2Owob2koDLMugeunPk3flLjwHOT6XoROY9/kOHtlrpu6bNbUjuqL13ldAkClUnnucn+/Kv9EHngKHVr99/3gqMr6dkK4VQBdwY6eexx0MPbHDfi766XheFrqxv0z6LJq09blUjXul3L2betDF6tNU6aP1du1nJ3g1NQKUdnd14YqqzbsXdfVTRey9r7G8v3rMju0NZvtZcuPkdQ3NQl9U/+u5z78WNDnd5vheu1zgzKUcTbp/EnjrK4TN85KUlvhOHtZhtIvkxiwX/acT/0u8Zu5mT66TXV5ff9zixBCCCFkk3DyQwghhJBUwckPIYQQQlLF0Gx+EjXKQXXIBC1aGs5eR2uPrWbfsozWNpsJmnWuYP+gbYWc0VdQXkderwz5Ut8y/TyZBM21lSv1LbvaeJ3/wkZ89vGCfT5td7NSs3Wi7W4urNuyE0tVtb1hytbr8djljXgvF1eq5rimst0ZL+X6lq1s2OfZORnf9aEd1lrolh1j3e2Zct6UTSvja/8etFdaOXdtGQcl2tYkae9JdiJJ/TuhTBqqrlsJ58i4PpZ0nwllMujvBrUh3AaGMs665zM2OX6M1GNws973PEnjrBmf/Xipx9Kse7faID5rx+eQVf3Rj8/qPJmEd7Ld4+zAdTmkfjlwPQdlXeP7wqD2W0n1nFBfSeeUhGf17WMQuPJDCCGEkFTByQ8hhBBCUsVVWcNdb8Ulr0pj1RYmLDeb5beGlUO0HCN1K4H0c4PvWbLW+05WC8rNOWSytqxQiT9z88eVWry2loEAYEc5Xs8H6QsJbr9X211TX3+5aZ+vmNBiVuvx2f1is5aCPMVcvMaRaSs9ZTLxd8cX17vb6zX7/k4txPZxYcXW5ekLa93tc88vmrIxJXvdeMOM/d2Bqe72bTvHTdn+yWJ3e7xgX8q+ibgkX6va2psqXv1/byQur+s+lyTp+CV0/TsvWZn+5yTpfufchGt7knxl+mrScQkSy3WDlyQSXJRlUFnKj7P1uJ+pr5sy49Jei2Nwa8OO8UG3DydhSiH2P3HjLAqxj4WC7X+t0kQs881WmR70SKQJIVK2VdLcjHQ8qLTlvmmmnn04Av09bdT7H6elpiTZy0tbuTgmeolK1LH+WwtljpJ0va18M6/DHk8IIYQQsnU4+SGEEEJIquDkhxBCCCGpYssiZ48dQYJ7aEXpht5dUmt8vkyqK33LzHFOp0Y1atNB6ZdaXwaAUJnubrfKO0xZozjZ3V6pO81V3Upwmuv5dWVj5ITHBZW6oZSz2ubuSqwK77SX6byHHle/UaHqMuc0WG27E9wD6txbno1GvPfJon3CnZWoB3u38bKyB6o1o86/fuucOW5d1dGZVdtWnrwQbRAurtuyrLIp2jteNGXavX3C2fVUCvFZp4v2udUpUXEpMs6std9t4ypV5RVJtLlTN+nterSNgO+3ur87ezzdV6URy0I9IfVMvr+bc6ILdNa71MZjpcfdNuF36th+9lOjCmiwqbQfxm7D2Wqod72Z9BY6dEezaO1uQk7Z5CjbjGbG1kkjIYVMdm0+3qMa73vwNlqKHrsy1f7qWTvma3NAbztWyLbvc1Q2lltuI75faputnjL1vfP2W9U4Dnr7rdbqUtxeXojba0v2uHVns6vIlGNokExl0pTJWBy7s64MxWhD28pbm09tX4ucHZ+T7IESw1pcuscrHkEIIYQQ8iKCkx9CCCGEpIoty15+adAs6Xn3O+8iqcjU4tJczxJ2KS6P+esZuczrS2ZpPa5z9ixnardLF+F5XekSCxt2WXVeSVunXKThxY247OijHOfVku9sxV7vOw5Ht2q/CF2+tPR3ldxxz1Xjfe7K2+drZOIyclYGXyDW8lnGrf/q3fz6vC2sxnddVG6343knYZbjsuqusYopu2lGLd27tqJlvKKLxmwCnbrfNRMePaceMOm4q0ZSlGUdCsJFvNWSrsDJUiHBrbqm+vvSWVPWuHC6u92cj2V6qR0AQlNFQ6/Y+syU1PL6xLQtG4tjhlQmbJlabg95e07jiuvLimNqx73LEffJnnE2wVXbSIzeRVn1nUFlLgAIxfgO54MtO6Yisx+dj5LV8y5K+/MXY6gJHW0dAA7PxXf78v3W9ODgVJQ5SlkfckNUme23us/5cCMbql2NO0m6LO3xelQ1mlSXPeEc9DfMh3JRZZma/bZKLdZDWLZjaVDf2oaTmVuLF7rbzaWL3e2NcxfNcdWFeP6NCzZsiKjvaXHG9r3Knli3hVlrspCdmu1u+/6cnYxlrfFZU9aqqNAkPkwBZS9CCCGEEAsnP4QQQghJFZz8EEIIISRVbN3VPUH7zmws27IEN3gTat3rdMoGIeStm5t2YdxwfsOZfNT2tf2F86JOdK3U+mzV6cZnVqMGe3bV6rHza9HmZ2HNugCvKHugmrtnbXty164xU7Z/PN9zT8PEu2BOKdftEKzdUjYkuNMqN0tfXwUdirzhbRWibp3UdrQLdWjY9x5UWaVh3/tYPdocYKO/qyZK9r23lL1DGNDNF0Bimoi5ToqT3Aj/2eH75op63WP5/n3Mh3SwTdQ+YzNEG5lK2b63stLiM0Vbpm2/tJ1PY8O2pfpSrLPsmrUhKUwoO0Hv5qxty5QdDwBIWdVhQnh/79av31goOFfcEWcGTwxv0fB2WMoeqO7cnNV76knXo2wrddoIAFhUXenEsr3escV4jfmN/qFIdDiJKWfrqMv8OK5TBVUzbvRT3d+7z2804rG+uevrZVzKnaa0++bIxtmkEAPefkvZ8mTWrO2OrCj7HGVDBwAtZefTqrk2oPuGso0DgNyhW+Nx++7sbi+3rP3b/LoO12Jf7sHJ2N+KC8dNGc48He9L2RcBQGt1WW1b1/qM2s/ttafU4Q9aPot8oYIrwZUfQgghhKQKTn4IIYQQkio2JXsJEiKc6mVkt6ScXY3LXN6d3R5ob6eplphrYn+3rKIlL1XtcmJJuUHOqkzqsuGW1PRyopMuJrUb3bgty4iKSOnWSLXstbhm38M3nolug4sq0zgAnFVuo//mbXeasktu9/5ao0LLg+JDEeuldSc9mXrPOsmh1T/Kt5Efk+QItbQZXLRPnYm4dfQbpqz69GPd7Y0Ltg3kJ2NdlvYfMmW5fTd0tzNTO01ZqxDlHC0bADZacMstv+Y6gQxGFREYAJru3zQZFZJg1UlbWgZYdP1oTR274GSN5Vr/bO3aLXnn2Iwp27M/vsfJPbd1tycWT9qTaIkzIaO3l8p1u/CCUdAhN5wEoV2Ge9q1cuENrl0jkyDpbxEzziZIJT0hPvSxLnu6bofBSZFrEt/ZmmsDi2qc9TLRrbPxPMVdsY6mXMTzJLd0PabpCPgAUFeFDWd6oO/Lh684q8I45931bpqJ3xTfF1pBeu7phZL0zdR4l/XMunIjX7DSVuPcibh94ZQpa63F74q48AC5PXF8y97yMlN2tHiwu/0bn32+u/3hP/28Oe78M9/qbk8fvNWU/Y233tHd/rFXHDBl+/fF73B+woW/OB+fwct4QUWU9tGmpazCWrgxIiSZ2nTgyg8hhBBCUgUnP4QQQghJFZz8EEIIISRVbNnVvUeLVu6hPuy6di8W5xmqbYC826i286k27fW0O+Na3WrFpZzKkC4qTH7NujmbfReGP6PslKaL1v1zYibqo4Ws1c+1/lx19jJfUtnGT33zUfu7RrTzqX3fHaasq7WP0lBkUHRoAm+fo9qEz3a/1TQAOtWADmfu3XVzF492t+snnjZlZ77yeHd7+XnrNjp9Q7RB2Vlytl0T8XrZgm2biVmDlX2FeDf4EaVDEMT3v+a633q9vxGDiO5Hts7Orcb6PbpgbRK0DdC6s//RWbxnx+zzzylX59lKtJ+pNqwf61l1bW9vVMjG6x2csjY4h9T+XMUObzvGol1Arm5t7vSTZ51rcdBtvq8NztUxyDN9zPUBPTy0SlOmrKFsIpZr9hlWavH5Fr3djbK1qRTs9SrKlmfveHzv5ZOP2HMceyLuuPADORUmYecN95uy89np7na1Yd/vxfXYJk73pBiKz1PJ23vW7W/cVVm+8z6HXpOX6syngdGhCnwGdtU+vcu6Oc6NL/ld093tzJRNByE33tfdPpnfY8q+fDzaGNUasQ088PL95rjsK6Mtzx37rK3jqw7F8bInpIaOJOG+p9k90eZM8vZ5tOu7uPauCbL5jyNXfgghhBCSKjj5IYQQQkiq2Lrs5UmKRFpQ0XBd9FvttmqytAJYWIvLb3Xnf7igljZPLdtlT+2Sqd0ux300ViXHNU4dNUVNF4VSk52NS4Y3qqVEAJg5HN0JD0zY651T2Y6f+aKVjM4/81R3+9jiK03ZwYn2UvEwVa9EF8ykukzIKq1ly56ox1ru8efXbrjONbw1HmWpZSXL+IzMQS2Prhy37pJa6mps2Htu1vu7REpR1V/ButabKM49ErCKSp3wjoZJC0C11a7Rnki59f5u6bpbrTfscVrC9fKBv4amlIvHeldjLVGfORf77WPP2wzR31bL8CuLdtm/pOSyl99uQxC84kjMHv2SPXZ5XUcCnnLtrMe93RT2/zdiFe17aY1Kk/Ztq9m/veqo6o2sbWcXVGTeJSdTatfwxao9f1FF8fUu5WNKosipbOKN08fMcevPRNk5P2YjZOf239TdzjgZfV1d78SSrZ/jqk0cvWglTB1J/zbXBvTzeJf26Xz7XeeulnlBQlZ3TWZ6ru9+3tVJsxLbf0OZZwDAk/Px/T59ykbSn1EhYf7Bg0e623vG7Pc6o8LF+JAlrVLsl8G52bdEfdvdd16q8Tucce8hW4rhYbzEp5/ch33w84zLwZUfQgghhKQKTn4IIYQQkio4+SGEEEJIqhieq7vGh5wvRpc4n+1b2/lc3LBatHa71G63gHW99ZrvrEpHcc+uqPnet2faHDc5Hl0BW+uPmbIzn/lKd3v+6TOmrKDcd2fv+owpm3npvd3tl99mXTdveFN0Z89mfsCUffTPo3v2Wee6KWJdCodOYrZh555pskP3t9/y6Ug0BZ8OQtnFNMSe89xqLNN2X5PB2oE0no82U7UlG9JgbHe8r5xzZ588El2s84dsuHbZHdNbNMetbYlpxwnZ7Xvo2jUM3zVaOrZu3mV9Rdl4aHscT9mVlcbjvrf52TPhbKAUs2WV8d3bZilt/pSyNdk9Zs93eC66QH/1qHU9b6p2UHCZpfMqXYe36agpt+11ly6hYlKS2PAVJu2OG9su2RFdtSgUCekt1nPxvmvOJkuPpRfWbfvUdlg+PMegZYW5aEO19/Bd5riSsq3MVKwNTuZATHGyUN5tyo6fi2P8iWXb3/UY2XQVrTPH512aj7JqL2Vn3LMe2mUJX7atcanNDJBy4RLaHbw14cYeFR5mvWWf4fRKvMbzx61dzzPz8Ts5W7Hj4Cv2x+tNPPeF7nbDhQ2RShy7M66eN8rxe3pi0baxjYZKceL67I5ynANM7bT2QJlKtL3NVO24HlzYBFt45Vrkyg8hhBBCUgUnP4QQQghJFZuXvfotJ+mM204OCcW43yrbyKNrIS5drTm3Y+16e9bJXl89FpfCH1HZ0gEgp5boz94Wlwz3TNilvgklXbRWFkyZlrpOfcXKXg213Hvum+dN2Z4nj3e35+75timbec3ru9v/+nu+x5TdoqSZ2+dshtpLEXOvUlL3gaMQBycPaKlr3S2R6xX6mjt9VkXuXFi3beDMSlw+1RF7swvPm+OqC+e62xOH7PL5zK3R5TMzMW3K8gdujjv7bzNlS2P7uts9kW/XlQTnMgrPZFWWeieJVfPteh62a3QrxHeey9hz66X/uouUXjQyQH+JquiymWu38fGC/d1kMz5/dtX2TR2J9dCkem9T9hwrN8VIsl85OG3KHj+vXWPts+5VclzevQftdd/TwpW0FfIuHEFCmAbpyJgy7N7Zb5xVY6uPoq73kqJ6l7LenTieM5PpH5rg7KqV4z/2rYXu9o0qq/tfu/OQOe7AA7viPbp3e7Ee7+WiD02uKDvZdf9UPE9+xrU/9Tw++r+W/HZWbJsuJ6goL4h+cpeO1p2zsu9GJX63zrn3or+TPrq1DgGwXLO/myjE9/LK/db0QEtdtae+3t3OzdrI63LLA93t80U7zn7u2YXu9pePL5gyG37AXvvVqn+HcVsJM+ob00r4LvVEO2dWd0IIIYQQCyc/hBBCCEkVnPwQQgghJFVs3dXd62/KrsDrl1BhqZsZa3dTV2654vR77WV90blnajuf55+wdjd5FWb71G6Vydnp2VKNemltYcmUFSfjM+y6x7oaasozVsPOT0SXz/qadc+sPxdtgLy8/KP3vrG7fa7msidn2rp1fyflF0iSjU+CG7y3jdAu1t7mR+ProanOeXLZ2si0lN3JrM7Sfc7Zee2OdgbFWZuxWCajC6bOEg8AzbEYDn4hY213njofXW2PuxQL+hGOzNiQ/dM74nvJuAzhrbx19R0ml1x+fUZlXRcrVVsv2s7B28iY1AbOnXhcZfiuXHjKlNUe/nh3e+HJZ2yZCkPQWI3vNDdm29Lut3xvd/vO295synTIA5c9AzPqgSaL9j3o91L0P1RpHfz4Zdq5s2esdd2jR+Ts7u0n9Y7rt3k1fpbz9n7K+XieQsaOPnrc9WES5lX4kY89bcfZz3/tVHf7C6rtZF07un1n7Ff5zLop01nWJ5zt2AFlo7l33GX7Vi+i5FzW9fji3fNrLW3H5m3CRlGHYSA72VbRjj3PL8Zx0Kf20ClIlly6Hm3n413KdbqX3a0FU9ZajvvF26NdT33/3ea4Z1fiO/qys7X96DdiWqHjp62bfUmFh9H2YYCtL2eOZuyHpZHwnfLhdbL9Q61cgis/hBBCCEkVnPwQQgghJFUMTfYyy1PurDqTtZ9t6ezcftldR2RtuWimDRW1NlewF5xUMsTrb4kZcC9lR++e4+tf7G5vXLCyl3aPPvidNqtubjpKJZmSy0K+oaJNt6ybpY5umhmz7n6yGiNZ7hybhWXEc1S/LJskg6kyH4251ozP62USHVB3o2HL9FLt+TUre90wE99vpamWzCdsncjeO+L5Szacgm5VrolhqRaf3buUaqnr6Qs2uqiOLOzdg2/ZMd3dHncul5XQXsLODNk1OifAjlJbivJnXlfLxX5pX7usu1VyIymNFa3oWlyOkkf9GzbK+bGPfKK7/ewnjpqyE/PxnS4pSeK2aSt7PajCFUzfa8NC7FZL6D7TuH2e/nKIdz8Ouf7L5HoJ3bvQFjqnHHoP7dcHE9zu87U4hhXc2CPVGH4gs7FoylqLUc6qFGw9zByIEet9tOSzR090t5u1WK8fcTLwl2ZjH37LPVaSvm9PdGWeqlmJWFSWdy9Xe9dm+0NlglFy0nwztgEve11qHsMVvwS41H68+7UKr7Di5MbTSv7XEdoBoKEG07x7Bl1Hu1zU9B3lhM/9jS/tbq6Oxb53dNGOx4+cjm1Hy1wA8NSzsf6K7loP3BC/ma92oSv2jMdjJzL2HWU2lHzm35+WhAeI6Ozhyg8hhBBCUgUnP4QQQghJFZuSvVqIidTK3lsiAWnEJVGfnCxXizJRIWtvpzwVo0u+dK+Vic4+ECPAnluynjj3H47J0X7w9iiP5L79SXPc4mOP9r3n8f3xd+V7X2XKZNcRtWPnj1KNS89hbQV9mZzrWyRuCc9HzB42wT+D3nF1grr11jCHqtM03Qql9tBZdB4K88qTr+7kskru8u1sY+5ms39GJUBdd0u1c2oJ1ifa1PLciovi7CUVjZa9fNJP7TEz5r11uvujS4XppZlighSkn9F76WhvkXzVysKZizHC9srjT5iyM9+IEdGfOG+TDj+lkpnqIMRe9irujf17xXns6Hv23kmVfCzzEqeWXptu6NOKjh/a9CV6xLEBo6EPDXU9cdHDM9UoEYgbZ1snnuxub7hklesnz3a3sz7x73fHcbeSt16v+VL02lm/cLK7/fyTF8xxE0qmvG3ORoWfXo4R8Rtf+4Qp02YD2ZldpiinPTpL9pxaItPJlgGgojMM+O7dkfGH3jMvjed+HFf91Cfh1WOKl7Y2VH/w42U5H9/ZjAtZrce6xdKMK4vbNeXh5/vQpMrWsO7kuLySxt98/35T9o6XxG/5oTEnN67F9iJ1+y3XSaKDizIfirFd9cjWA/RLrvwQQgghJFVw8kMIIYSQVMHJDyGEEEJSxfCMSXTkX6dtajuf7Mo5U9aaP622bVlOZeB+5d5bTNnLXhttAry+l52POnL9L369u33y8w+b42pL0R5hxx2HTVnxBpXh+8AdpqwxqTLdOvuKzHp0BZSytXcw76hgXTfNM7Sc297IQjsPgDPe0fYkOdj71G7G3m5iTRlceHf2C2tR19WRgwEgp060kY0utOdX7X3pKMYzJXuOqeJgc/wNl/Fcu4XrLNIAMKWiiO8dty6l+tF7sgt328BwXd2DOqO4/lfKxWt5OyZRd+tDTWhzq0zN2pA0F5VO78KyztwY7QnudTY5R05FO7iCyqr9kr/9anNc9lVv6277EATaRsy7X/tn0DQSIkPrKMf+DAVRzzDk5O2bRtsDevde3VdXbPTdphpb64vWfquxEftjcdpG322prNreti1fiu2+MBFdmSd2WFf3N90dXadvdhHxwyMxg/jaszYauKa0YdtfUPZOub032IO1W7x7R8ZOyttWuijLI0fdy1jRvpedY7Fv+D6rW/xFZz+pTezqbjzTx9acrdCUstfZVY49YHfBu5DHd7tr0o57h+di2Q+7kAZHENtj9vQZU4aGeoa8s91RY1lPeANVfz12sgO4vnPlhxBCCCGpgpMfQgghhKSKTcleGQDlTGe5zLsB62Uml1RML1d5SSxUo2vb6uOPmbKVE1YG0+TUkmtoWpe7ukqYuHo6Lrc1a3YpbOqG6D5ZOWxlr9yRu+L5tMwF4KJyBSy4Jf9xHZ1Zu1XiMhKILnOuqaPGyCQ9hWo5sWkTyuqlWp+0c0I9+3rdto+6enYtcwHA0+fjsx9yS+Y6yveqklC8vLFnKra5wtIJUwYlmzRnDpqiarO/jjFZim11l2vTu1SixUkn1WnXVJ9gzy/PDpNLj+LlyIq6vUuJOONv4vMXvBbUtPKkRtQyfeXIIVN2QEVcn73DtpFMXiUevSX+rvC6HzLHPdOIEsS8S2o8VYznmKvY8WRaSZ7jLqyBGaN8X0yoFlFyUlIk6JGQkFhYfBRntR9qzmVYvXcdRgAAKrfHfis33GvKnsru627/3he/aco2lha62wfuiGYJ73yzDUPxXTfG84/BJunUMQa8m32rHt+7uPdu9p1U0lLjrnePTnKdHhmXTBq8NKMS5vqIHnPl2I6TRgwfnqKuxjOdABUAzqowE62KG89UP8ouxejtPSYlEkO0+ASld6j9G52CmHnu2XjtDWcOotpAZnKHLTPSlh2rzZ43FfHh6i8DV34IIYQQkio4+SGEEEJIquDkhxBCCCGpYlM2PwExFYK3XdD73s2sUYgCYNhp9eDMeAyZXlmyYdGXno3a48UnTpmy1TNRN5SEVBsFFVp96vC0Kdtxe7TzKdxste7GTLRHWKja51lXWcld5H00g7ZL8T7qcd/fchnK5iezzXPSBNuIjE5v4VNdqGPLJRsKv6pe1NkVq/t/+Zlol7W4ZsXiW3ZEV9sblJvstHNn13Y+OtwAADSmo43DqTVbYc8vxXtZq1sbiikVyn3GZSneUeqfMkO7VHv7t1ES21R/WxdvBpMNysXb2fiYsA3eJk1l/87NWpu4srLHqNxkQ1Tkdsd+1bzx5d3tx1fsjZ1fjfcyV7G2GTqUwWzRdqTssnKjXelvY+fTtmhbxJC3dmfazsfbLI4iSUmSPZ4Zd119mfpzY09W1VFmh3VD3th9Z3f7z59dMGUf+vLj3e1Hv3rSlOUrMfXFWx6MY+n33mr7/v6JWH+yZsd4nbaidMi6rId6tM/JTs2asuzOmD6hOW6vp21+etIP6faRt67aW8kMfiUSbSs1ri5LyrbS211qG9pyxY6XFWVHtO7TwqjdSRf+Y7KhUqPU4rjenLRtZXEh9qm7d9vUIffujveSu/isKfM2aBrRbbXh+qxOYSHuDeq+6L+ZdHUnhBBCCLFw8kMIIYSQVLEp2Uuglu6Ssqa6Jad1pSYs9khIcYn5llf+oCnbq5bWC5MPmbKL3zrW3W45d+XSTIw0WZqNS7Pj++3yaOFwXJJvTbmlYBVNuO5c5LV7cLPHVTq+l3XnpKjds71bcdlEsnTVMoLlWFOXSfhrK2krLFs35qxaup2o2KzB0yoi8prLBry8FJdEv+7cmg+rLNA6S/EOJ3u1yvF6K+P7TNnzS/GcT11cMWXaU3TOuX/qKKs6AioAlFT99XhUqybRKo65wtG4ugfE7OP+fkwmcHf9rKrPzMayKdOyl4/w3GqoenJLzhkVmd1n427tjyEknlmN7/TEkpVQ907Evn9wwspe5epCPP8ZK4dDRwL2snNOL5M72SsXr9dTQ0oi8909m+iIvDUEqp78OKtcen0GbC2dZMYmTVFLSXn1Xbeasi+djO/sTx49bcqePb7Q3R6ftlGIX/tAlJ5+7OVRWt7dtNGljaLvJOmg2pGWRNt/UHJZ2UWeLkbJpTVmJbGGGoNzPlzDJr5bwyAghusoZvtf20SeBiDKpCDjJWd9n0u2vsyb8M+qx4Fl624eVlXU78nozr6csZkIKoU4Xtw7YetkD9Q5ls+7S6u+WCj1L8u5fqmfoUeq7h9CZxC48kMIIYSQVMHJDyGEEEJSBSc/hBBCCEkVW8/q7lygW0ozz9SsnphV9jNVl2n26Yvx2HrTupgeuD/aAO285QFTNnf8W93tsGZtFYIO+662xWmNOhtws2Q1cp31tur92RXucbCeoBs31VTT2/wE73apSdKpt4hxwUzSuhN0Y63XA0C4GN1Iczn7rg/PRTuDlx2waT8+9Vj83dK8tf146ImoHe+bjOfMZqybZV293GOL1q5Ap0fwmakPqmzt2sYHsC7VXq83mb5dmgHjfe3fbff9DddJOgOg0Dn1ssukPqGNgBJCVHhXd52OpVW19dJaV3YILrR8pqzsnKaszc9SLtbbogozUMzZetk9pkIJrFq7Bjn/XHe7ccGWmeNyLrVBMY4vmYptPxhX7bUnPY8K/e9tfEbQN815E9Jb9NZlHLNCzo4n2v1bp+cBgBWVBmH/DmvjseflMR3Mnc61+TtvmO5uzz7/xe5209VJdtbaU9obU+PzpLXd0ffccuNJULZ0ay0fUiS+h/xmbH5GQAZAsTPWS6O/u3dPXervq++Xa3F8ay7a0AEtFS6mtbxgy1SflawLhaBCB+Rmos2kTyN0eDL2qfKGte3KnnsmXsulsDD9reC+dQ2dPsbVs2rHIW/bpk4PshW48kMIIYSQVMHJDyGEEEJSxeYjPHe2ZRMug2WJS5vjTnaYUi7QRxfs0vrjKtv3nok5U3bjLW/ubs+W7TnzK1FGya6qZUEn1TVVtvb1gpW9zqvosMs1+7uyWqL33ovai7vmNLEJ5S5dcMuJIdM/EvAld7/++cdfIH2lGfRKAGrfy4jN+bOx7NQzpmxMvfs3HL7DlM2/PsqPf/gVm5F9UUX6/R/KDfeLR+2S67hqR7Pj9p4PKLnsJresv3e8vzt7rqmWqesu8qh+R87V20QL7hPhedh1qfvmREI2c/HZzBv9M7cnoZfNg88urpa4TbRd2Ojo4+p9785biWo6xGXzzMo5U6alLr+0r7OX97jUquzf4jOBG7dZJ6PounbjXnD/HwYDRwX2v9PP0CMTxTrx3f3GHVEOPDxtTQ9yapw6MmXrqHDm293t2tG4HbzbtpIfMxM2BIaWuppT+01ZU7WdmosxsKH2m052HVPt30dHTpS9tuAufSV0XfZEe1d90dezdvHuKVPRklsuK0LjbBw/V05Yd/PacvyeFqethDk5Pu1vHYANKwEAmTUV3mTByptN5S7vZTUUbbsy6L7ovzfF6E7fKtl71uNsz7g2gLzJlR9CCCGEpApOfgghhBCSKjj5IYQQQkiq2Hx6i0uCsXfNU9s6C3K7MM6xdpXs74pzUQt8btHOxZ65GG2Aji9ae6C6sqc5NGX17Z2V6F5bURl/g9MBtcvnhSWrDeus0sWc/Z1K6I1qw2rROjN43rmza7dBcRlqezLWKsT9fxiYuvSh/vV21ur82vUw4zIKi9J8fRbfxvEnuttja0um7O2339/dvse50/6lsu05pWzCtI0PANyxK97L7Tvtfc0pmzCfFiOjbMJkub/9S4+bZUKoet03fDqEXMdVetjZwBNTIiTp3zp0gdfbld2IlOzzazLeRqas7EucTl9oxSffWYntbqpg7zEzH20LgnOb1YjP5qzL/DikbE+8Kzj0sUnvz497nbKh981LNgz+XnSIf19mMtM7V/dCrL+sSzG0Qw1ovlnXVQNe8SEUdt4cf/eq3epizv5CtauGswHT5/Tnbyq7y0bLdqRiLr7xrBs7TRgRfy9Jdj2jSiPU77yZweqyx1ZI9zfX97StTa5UcGXxGoUJ2591OhSpxvE5u2zt7XwYA3P+ohovfCgJ3Ta93Zx2Zy+6NCaqvTTE1p3Z20LdceWHEEIIIamCkx9CCCGEpIrh+fb55UVFU82x8us2E/iM+t3UnM26rpdjjy1WTZmWoqpOWzi3plwI1Qpow93j/HrcX6zaspJaImwFe34tuRXcOvGUuueyk8u026jzdO9x5dQUs8MWSRx9lvIBIPhMutr10P0uN6fcT33kUbVfV26xACDHoiR2z5SN8nrfgegGH27QS7U+23yUx2TjpC1TcpZ3wdTZrv3ycqugypIi7foVV5WZuOWrdcRVCSA5dIFH1W/Ly0T6FD4CeSm+U/+IQUVL99mWJ1WfMNGlqyv2JErOyjg32aCW6FvezV49g16GB4BMRf0u786p7zNBSrvaEYKTlvNDwUmxjThGemlBv2sfXV6PkW6owyk1Rn7mOTt2P3U2RtZvqoY+VbHtaP90vPZ4wY6z2hTAmxfsHottbqzgIhKre550oU4yOpKyl/S1xDIC1/Yt48ZZKBd9bxKRVRHUs2681Iy5iNlaIhYdhR1AdiaaioS12BdbOts7gJYyWcjO7jVlmekYjiYUnHylwzD4dqv6Yt39bl1lV8iKbZzaSKHi398AcOWHEEIIIamCkx9CCCGEpApOfgghhBCSKjaf3uKSlpygKfe4simprlWx4c0z64uX3QaA/SoUesWF7Ndh8oO3ydGJj1XZunOl1Bm+J4v2VYwr11tvc5MUyl7r6d68I6O023qPMUh/RhZCv4/9grnvBLdElKdNWVPpruJsKnLK9dG7SzYunOpu15/8hi37xle629mCcsktWRsUKUUNOzNm3SwzSvvW2jYAQNV7cFnABybBRsR5cCNgNKlKEuszoW9W0f+Zc5X+9gTZoGw3nO2EtsHzzbxg9uO9ZJ0dgLEn81njVRvsccFvRFsJn37FhNHfjO1Oq/+zjqxv9rNH0XZS/l5UWIpMddWUaXu2Mefmv5aNfbUnjUQj2lRpGx8A+MQXn+9uz5+OY3dlytbJjt3RjuOmA9bV/Y590Q7rRpd6RocN8TY/2j6onHPjc1D1nOtvMyUuDcelcABXbZxNsJM1ISi8+7f6pslua7uT2x1tJDdj+yc6c/xGQvZ3NX7KzkOmrDERy1rK7g+wIT82Gva+9Hix5ELO6M/+RLG/3ZdnkDrkyg8hhBBCUgUnP4QQQghJFVfF1y+nl8jdUpyRUZx8oJfiCs6VLatcJHvcxNWyWl1FlPXLZuNqTW3GRf41S5IuimbSkppe5q+5NX/9dL5sUpTrsM/InHC9kdMje6klZZ/NXMl6kvMZtePSeqZil75Le+LyaWvdLtcHtQQbmnEZvCdrsM4cXbJLwTqjcE+kZuXC3ZOB3WRn95GuC5c9zrOtdddBh5oIGfuM51djO682bd/UkXN9OAnt2lxv2aXqtbp2T7X3oqVmTU8UdSXFTI3bZX8dQqKwZkMqJIXcMPKfrzM9LjX7Z4je9voc0D3byBhw2bhdn66M7ehu54tWrtg/EdvL219qs66/7FA0Szi/Fq+3smHfX1lJVhPOvOCeXVGirhQu3zYAYIdzZ59SY7lxbb8CJnSH7+9XGz2++O+izmjv1KuMkqO9hGTNQezvtKnFmDMj0RGzyxklq9VsdPWGckW/sG7DTDw3H+thccOasOh+v+HGmVJCtHz9u/0TdgzObES3+9z5Y6astvfuvufs/v6KRxBCCCGEvIjg5IcQQgghqYKTH0IIIYSkiuHZ/Cgd2dvgFLULtNLpAEBUGO8e93m1P+FtLvLRpiT0aMVxX5sc+BD65tor/XX+pCzPPvR4S2nmPuO71lyXa1YvnZi4dkKt67uWpGf3P9Q6srdN0Pq2d2vWdhrW5MDYXkmrv9uqtj9qeTsl6X9fxpan5551PW8iU/o1hnZLP1e17VWnF1izpjtYVzZW3zxn+86T56It1sUVm3pmTbVtb/9h7svneFHsnY79+6UHp03ZrrFoq3HzDms/NqZiC4y7OAPGI9rZBmm3Z98+k+yIrjqq3bWK1rYto+7T9wFRaUB01m4AyJx5prtdqFtboV0qRchul/7kgYl4/dZctN1pucztOn1Bb7oONQ76kCK6PwaXxkS78vv+rusvM/xM7cNC29EV3TOsq7Kzq7b9raj+pW2tAODUcuyLF9dth15QHXy6ku9bppkd93ZRMdzBt0/adpR0/ht3RVuhgrPv2z8Z29h9u61930SCbddqLra5KWZ1J4QQQghJhpMfQgghhKSKoWktWgLpyUKeFCFVLfdJwy6fQ7lrZmou6/NGgiSh0MvZ4jJAm6zdm5AxQlKGWrXto1LrVfid5f5undvuTjsoiXKgW4rW0ucmlijNu9AuyP7a2u3ey2NJ8pUmSdraTPsY+Mirg5YPdsO6oGpdeHzCao7ajXa5ZiN2a1d376Z+Qclgvkz/7rnz0Y12vGTHhQsrsd8+dspGFr6oJLE1F7VdS2I3TNtIxlpmyDq5emcl9mMvxl1L2b+TJOlWKcoA4qI4m7GvYOtSxqLrdKbuxmAtO3t3bB1tuh7rMuvc7M3768mynlfbLjyGHk+c2QOUBOdDVJhvSkK/3e5+qrMBnHIBuevqvTedz7ruUzfM2LrUEtJiguTsMwwsbtQvW1Z1rvQ3zcR+osMUAMCU6sO+3+v+Ftyb31WJv5toOdf6fJTBNlq2TVfU+6vtuxebhSs/hBBCCEkVnPwQQgghJFVw8kMIIYSQVDESMdtrqTr7cMgnaLB5l4V5UNsQr+tqLbrZP7XGwHYcPWkc+v8uB3WNHk/eayhM/oD01KXZ2cTcOcEeaGCyCe0h6ZxJ7WgILuvXS10CQNO5IZuwFO5BtCv6jdO2bx6cjPp7yWXVXq33f986sv2Z1Wgb4s0ExwtxzMg5l/ilauzTM2U7hE2osBc+hH8hGx/QZ8TRaQF8lvBrFd+PzDhbcLY1ug84m58kxKf66HszW+xjg4aT2IStnq6966Vv7hmz9WVaYMJ3K2l8DuifvqPH7lJi2AKd0qLlbFptJnprh2XDRdjxQvc33/d09w4te886TEfGpbh6oXXLlR9CCCGEpApOfgghhBCSKiT41K9JB4ucA/Dc6G6HXIHDIYSdwzgR63LbGVpdAqzPawD2zRcPrMsXF5etz01NfgghhBBCrncoexFCCCEkVXDyQwghhJBUcd1MfkRkj4h8SESeFpGviMhHReRWETkiIo+O6JrvEZFzIvK1zn8/OorrpI3tqMvOdf8XEfmmiDwmIr83quukjW3qm/9e9csnRGRhFNdJG9tUl4dE5BMi8rCIfF1E3jKK66SRbarPwyLyF526/KSIHBjFdV4o107SmgRERAD8EYDfDiG8vfO3lwDYDeD4iC//X0MIf3/E10gN21WXInILgJ8C8GAIYV5Edo3qWmliu+ozhPA+dQ//AMBLR3WttLCN4+zPAPhvIYT/R0TuBPBRAEdGeL1UsI31+X8D+C8hhN8WkTcC+HkA7xrh9bbE9bLy850A6iGEX7v0hxDCIyGET+uDOrPZT4vIVzv/vabz970i8qnOvxIfFZHXiUhWRD7Q2f+GiLwP5GqwXXX5YwB+JYQw37nm2RE+Y5q4FvrmOwB8cOhPlj62qy4DgEtZdacAnBzR86WN7arPOwF8vLP9CQBvG9HzvSCui5UfAHcD+MoAx50F8OYQwkbnX/ofBPAAgHcC+NMQws+JSBZABcB9APaHEO4GABGZ7nPOHxKR1wN4AsD7QgijXml6sbNddXlrp+whAFkAPxtC+J8v8FnI9vZNiMhhADcgDrZk62xXXf4sgI91VvDGALzpBT4HabNd9fkIgL8G4D8A+EEAEyIyG0K48AKfZ6hcL5OfQckD+GURuQ9AE50PHoAvAfhNEckD+HAI4Wsi8gyAG0XkPwL4EwAfu8z5/hjAB0MIVRH5uwB+G8AbR/0QBMDw6zIH4BYAbwBwAMCnROSeEMLCSJ+CXGLY9XmJtwP4gxBCc3S3ThzDrst3APhACOHficirAfyOiNwdwqD5jcgLZNj1+Y8753sPgE8BONE57zXF9SJ7PQbgZQMc9z4AZwC8BO2ZawEAQgifAvB6tCvhAyLy7o788RIAnwTw4wB+w58shHAhhFDt7P7GgPdAktmWugTwPICPhBDqIYRn0V7Ju+WFPQrB9tXnJd4OSl7DYrvq8r0A/lvnHJ8DUAIw90IehADYvu/myRDCXwshvBTAP+v8beGFPsywuV4mPx8HUBSRv3PpDyJyr4i8zh03BeBU518M70Jb3ri0NH4mhPDraFfW/SIyByATQvhDtA3u7vcXFZG9avetAL41xGdKK9tSlwA+jPaqDzrH3wrgmSE+V1rZrvqEiNwOYAbA54b8TGllu+ryGIDv6pzjDrQnP+eG+mTpZLu+m3Mi3eyrPwXgN4f8XEPhupC9QghBRH4QwC+JyD8FsAHgKIB/5A79VQB/KCLvBvA/Aax2/v4GAO8XkTqAFQDvBrAfwG+5SvL8QxF5K4AGgIsA3jOkR0ot21iXfwrgu0Xkm2gvwb7/WtOgr0e2sT6B9qrPhwLD1A+FbazLnwTw6x3j2QDgPazTF8421ucbAPy8iAS0Za+/N6RHGipMb0EIIYSQVHG9yF6EEEIIIUOBkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq4OSHEEIIIamCkx9CCCGEpApOfgghhBCSKjj5IYQQQkiq+P8B3dMo17o0yKsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 720x360 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "coef = clf.coef_.copy()\n",
    "plt.figure(figsize=(10, 5))\n",
    "scale = np.abs(coef).max()\n",
    "for i in range(10):\n",
    "    l1_plot = plt.subplot(2, 5, i + 1)\n",
    "    l1_plot.imshow(coef[i].reshape(28, 28), interpolation='nearest',\n",
    "                   cmap=plt.cm.RdBu, vmin=-scale, vmax=scale)\n",
    "    l1_plot.set_xticks(())\n",
    "    l1_plot.set_yticks(())\n",
    "    l1_plot.set_xlabel('Class %i' % i)\n",
    "plt.suptitle('Classification coefficient vectors for...')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kK8v34AK6xXJ"
   },
   "source": [
    "### 1.2 Feed-Forward Neural Network\n",
    "\n",
    "The first step is to create the functions that will allow us to implement a feed-forward neural network and manage the training and validation process.\n",
    "\n",
    "The MLP class will define the architecture of a feed-forward neural network, with a set of hidden layers (fully connected layers [nn.Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)), with a activation function in between them ([relu](https://pytorch.org/docs/stable/generated/torch.nn.functional.relu.html#torch.nn.functional.relu)), and a [softmax](https://pytorch.org/docs/stable/generated/torch.nn.functional.log_softmax.html#torch.nn.functional.log_softmax) in the last layer. Since the dataset poses a multiclass classification problem, the last layer should have a number of neurons equal to the number of classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "In9r_o8vvNaz"
   },
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "    def __init__(self, dim_layers):\n",
    "        super(MLP, self).__init__()\n",
    "        self.dim_layers = dim_layers\n",
    "        layer_list = [nn.Linear(dim_layers[l], dim_layers[l+1]) for l in range(len(dim_layers) - 1)]\n",
    "        self.lin_layers = nn.ModuleList(layer_list)\n",
    "\n",
    "    def forward(self, X):\n",
    "        X = X.view(-1, self.dim_layers[0])\n",
    "        # apply relu\n",
    "        for layer in self.lin_layers[:-1]:\n",
    "            X = F.relu(layer(X))\n",
    "        # use softmax for output layer\n",
    "        return F.log_softmax(self.lin_layers[-1](X), dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h6OVD_1xUwWH"
   },
   "source": [
    "##### training validation function for the MLP and CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "B1eUu01N8wIR"
   },
   "outputs": [],
   "source": [
    "def train_val_model(model, criterion, optimizer, dataloaders, num_epochs=25,\n",
    "        scheduler=None, log_interval=None):\n",
    "    since = time.time()\n",
    "\n",
    "    best_model_wts = copy.deepcopy(model.state_dict())\n",
    "    best_acc = 0.0\n",
    "\n",
    "    # init dictionaries to save losses and accuracies of training and validation\n",
    "    losses, accuracies = dict(train=[], val=[]), dict(train=[], val=[])\n",
    "\n",
    "    for epoch in range(num_epochs):\n",
    "        if log_interval is not None and epoch % log_interval == 0:\n",
    "            print('Epoch {}/{}'.format(epoch, num_epochs - 1))\n",
    "            print('-' * 10)\n",
    "\n",
    "        # execute a training and validation phase for each epoch\n",
    "        for phase in ['train', 'val']:\n",
    "            if phase == 'train':\n",
    "                model.train()  # set model to train mode\n",
    "            else:\n",
    "                model.eval()   # Set model to eval mode\n",
    "\n",
    "            running_loss = 0.0\n",
    "            running_corrects = 0\n",
    "\n",
    "            # iterate over the data\n",
    "            nsamples = 0\n",
    "            for inputs, labels in dataloaders[phase]:\n",
    "                inputs = inputs.to(device)\n",
    "                labels = labels.to(device)\n",
    "                nsamples += inputs.shape[0]\n",
    "\n",
    "                # set the parameter gradients to zero\n",
    "                optimizer.zero_grad()\n",
    "\n",
    "                with torch.set_grad_enabled(phase == 'train'):\n",
    "                    outputs = model(inputs)\n",
    "                    _, preds = torch.max(outputs, 1)\n",
    "                    loss = criterion(outputs, labels)\n",
    "\n",
    "                    # if in training phase, perform backward prop and optimize\n",
    "                    if phase == 'train':\n",
    "                        loss.backward()\n",
    "                        optimizer.step()\n",
    "\n",
    "                # increment loss and correct counts\n",
    "                running_loss += loss.item() * inputs.size(0)\n",
    "                running_corrects += torch.sum(preds == labels.data)\n",
    "\n",
    "            if scheduler is not None and phase == 'train':\n",
    "                scheduler.step()\n",
    "\n",
    "            epoch_loss = running_loss / nsamples\n",
    "            epoch_acc = running_corrects.double() / nsamples\n",
    "\n",
    "            losses[phase].append(epoch_loss)\n",
    "            accuracies[phase].append(epoch_acc)\n",
    "            if log_interval is not None and epoch % log_interval == 0:\n",
    "                print('{} Loss: {:.4f} Acc: {:.2f}%'.format(\n",
    "                    phase, epoch_loss, 100 * epoch_acc))\n",
    "\n",
    "            # deep copy the best model\n",
    "            if phase == 'val' and epoch_acc > best_acc:\n",
    "                best_acc = epoch_acc\n",
    "                best_model_wts = copy.deepcopy(model.state_dict())\n",
    "        if log_interval is not None and epoch % log_interval == 0:\n",
    "            print()\n",
    "\n",
    "    time_elapsed = time.time() - since\n",
    "    print('Training complete in {:.0f}m {:.0f}s'.format(\n",
    "        time_elapsed // 60, time_elapsed % 60))\n",
    "    print('Best val Acc: {:.2f}%'.format(100 * best_acc))\n",
    "\n",
    "    # load best model weights to return\n",
    "    model.load_state_dict(best_model_wts)\n",
    "\n",
    "    return model, losses, accuracies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0CBE5tRMZEfr"
   },
   "source": [
    "We will start by creating a simple network with some hidden layers. Thus, in addition to the input, it will have 3 fully connected layer which, in this implemetation, is assigned to the input of the MLP Class. We will use the Stochastic Gradient Descend optimizer ([optim.SGD](https://pytorch.org/docs/stable/generated/torch.optim.SGD.html)) with 0.01 learning rate and 0.5 momentum. The loss function to be optimized will be negative log likelihood ([nn.NLLLoss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html)). Training and validation will be managed by the function \"train_val_model\" previously define."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 981
    },
    "id": "200WI3xND6_M",
    "outputId": "79913a00-abf0-4e48-8177-64b49bbd6fac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/14\n",
      "----------\n",
      "train Loss: 0.8353 Acc: 75.38%\n",
      "val Loss: 0.3007 Acc: 91.05%\n",
      "\n",
      "Epoch 2/14\n",
      "----------\n",
      "train Loss: 0.1805 Acc: 94.77%\n",
      "val Loss: 0.1527 Acc: 95.51%\n",
      "\n",
      "Epoch 4/14\n",
      "----------\n",
      "train Loss: 0.1114 Acc: 96.74%\n",
      "val Loss: 0.1116 Acc: 96.67%\n",
      "\n",
      "Epoch 6/14\n",
      "----------\n",
      "train Loss: 0.0776 Acc: 97.72%\n",
      "val Loss: 0.0882 Acc: 97.21%\n",
      "\n",
      "Epoch 8/14\n",
      "----------\n",
      "train Loss: 0.0565 Acc: 98.35%\n",
      "val Loss: 0.0810 Acc: 97.32%\n",
      "\n",
      "Epoch 10/14\n",
      "----------\n",
      "train Loss: 0.0415 Acc: 98.81%\n",
      "val Loss: 0.0764 Acc: 97.46%\n",
      "\n",
      "Epoch 12/14\n",
      "----------\n",
      "train Loss: 0.0307 Acc: 99.20%\n",
      "val Loss: 0.0722 Acc: 97.79%\n",
      "\n",
      "Epoch 14/14\n",
      "----------\n",
      "train Loss: 0.0226 Acc: 99.44%\n",
      "val Loss: 0.0748 Acc: 97.62%\n",
      "\n",
      "Training complete in 2m 35s\n",
      "Best val Acc: 97.79%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjbUlEQVR4nO3deZhU1Z3/8fe3G5q2kU1odgQU6BaXxNhBjU4SNRo0EcxiBpQsEydoMkQTzQJmhichiaMx+0gWkhh9xEiIcUHFoKP+komJhnZFQBYRZdHQKiKK2Czf3x/fKru66aWgq7u6bn1ez1NPVd17q+sLNJ86de6555i7IyIiha8k3wWIiEhuKNBFRBJCgS4ikhAKdBGRhFCgi4gkRLd8vfGAAQN81KhR+Xp7EZGC9Mgjj7zk7pXN7ctboI8aNYra2tp8vb2ISEEys+da2qcuFxGRhFCgi4gkhAJdRCQhFOgiIgmhQBcRSQgFuohIQijQRUQSouAC/cEHYdYs0Ky/IiKNFVygP/IIXHklbNmS70pERLqWggv06uq4f/rp/NYhItLVFGygr1qV3zpERLqaggv04cOhokItdBGRpgou0EtKYNw4BbqISFMFF+gQ3S7qchERaSyrQDeziWa2yszWmtnMZvYfamYPmNljZvakmZ2V+1IbVFfDs8/Czp0d+S4iIoWlzUA3s1JgLnAmMB6Yambjmxz2n8BCdz8WmAL8LNeFZqqqinHoa9Z05LuIiBSWbFroE4C17r7O3euBBcDkJsc40Dv1uA+wOXcl7ktDF0VE9pVNoA8DNmQ835jalumbwDQz2wgsBr7Y3A8ys+lmVmtmtXV1dQdQbhg3Lu7Vjy4i0iBXJ0WnAte5+3DgLOAGM9vnZ7v7PHevcfeayspml8TLSkUFjBypFrqISKZsAn0TMCLj+fDUtkwXAAsB3P3vQDkwIBcFtqSqSoEuIpIpm0BfCow1s9FmVkac9FzU5JjngdMAzOwIItAPvE8lC+mhi5qkS0QktBno7r4bmAEsAVYSo1mWm9kcM5uUOuwy4HNm9gRwE/AZ946N2upqeP112Nyhp19FRApHt2wOcvfFxMnOzG2zMx6vAE7KbWmtyxzpMqzpKVoRkSJUkFeKQvShg/rRRUTSCjbQhwyBXr00dFFEJK1gA90sul3UQhcRCQUb6KChiyIimQo60KurYcMGeOONfFciIpJ/BR/oAKtX57cOEZGuIBGBrm4XEZECD/QxY2IFIwW6iEiBB3qPHjB6tAJdRAQKPNBBy9GJiKQVfKBXVUWg792b70pERPKr4AO9ujrWFn3++XxXIiKSX4kIdFC3i4hIYgJdJ0ZFpNgVfKAPGAD9+inQRUSyCnQzm2hmq8xsrZnNbGb/j8zs8dRttZm9mvNKW6xNk3SJiEAWC1yYWSkwFzgd2AgsNbNFqUUtAHD3L2cc/0Xg2A6otUXV1fCnP3XmO4qIdD3ZtNAnAGvdfZ271wMLgMmtHD+VWIau01RXwwsvwLZtnfmuIiJdSzaBPgzYkPF8Y2rbPsxsJDAauL+F/dPNrNbMauvqcreGdHr1Io10EZFiluuTolOAm919T3M73X2eu9e4e01lZWXO3lQjXUREsgv0TcCIjOfDU9uaM4VO7m4BOOww6NZNLXQRKW7ZBPpSYKyZjTazMiK0FzU9yMyqgX7A33NbYtu6d4fDD1cLXUSKW5uB7u67gRnAEmAlsNDdl5vZHDOblHHoFGCBu3vHlNo6DV0UkWLX5rBFAHdfDCxusm12k+ffzF1Z+6+6Gu6+G3bvju4XEZFiU/BXiqZVV0N9Paxfn+9KRETyIzGBnh66qG4XESlWCnQRkYRITKAfcggMHKihiyJSvBIT6KCRLiJS3BIV6FVVCnQRKV6JCvTqanjpJXj55XxXIiLS+RIX6KB+dBEpTokKdI10EZFilqhAHzUKysoU6CJSnBIV6KWlMG6culxEpDglKtBBQxdFpHglLtCrquCZZ2JeFxGRYpK4QK+uhj17ItRFRIpJIgMd1I8uIsUnq0A3s4lmtsrM1prZzBaO+YSZrTCz5Wb2u9yWmb1x4+Je/egiUmzaXArCzEqBucDpwEZgqZktcvcVGceMBWYBJ7n7VjMb2FEFt6V3bxg6VIEuIsUnmxb6BGCtu69z93pgATC5yTGfA+a6+1YAd9+S2zL3j0a6iEgxyibQhwEbMp5vTG3LNA4YZ2YPmtlDZjaxuR9kZtPNrNbMauvq6g6s4ixUV0cfen5WNxURyY9cnRTtBowF3g9MBX5lZn2bHuTu89y9xt1rKisrc/TW+6qqgldfhS15/Z4gItK5sgn0TcCIjOfDU9sybQQWufsud38WWE0EfF6kR7qo20VEikk2gb4UGGtmo82sDJgCLGpyzG1E6xwzG0B0wazLXZn7R0MXRaQYtRno7r4bmAEsAVYCC919uZnNMbNJqcOWAC+b2QrgAeCr7p63WcmHD4eKCrXQRaS4tDlsEcDdFwOLm2ybnfHYgUtTt7wrKYnx6Ap0ESkmibtSNE1DF0Wk2CQ60Nevh507812JiEjnSGygV1XFOPQ1a/JdiYhI50hsoGvooogUm8QGuibpEpFik9hAr6iAkSM1Fl1EikdiAx2iH10tdBEpFokOdE3SJSLFJPGB/vrrsHlzvisREel4iQ70qqq4V7eLiBSDRAe6hi6KSDFJdKAPGQK9einQRaQ4JDrQzRpOjIqIJF2iAx00dFFEikfiA726GjZsiNEuIiJJVhSBDrB6dX7rEBHpaFkFuplNNLNVZrbWzGY2s/8zZlZnZo+nbv+e+1IPjJajE5Fi0eaKRWZWCswFTicWg15qZovcfUWTQ3/v7jM6oMZ2GTMmVjBSP7qIJF02LfQJwFp3X+fu9cACYHLHlpU7PXrA6NEKdBFJvmwCfRiwIeP5xtS2pj5mZk+a2c1mNqK5H2Rm082s1sxq6+rqDqDcA6OhiyJSDHJ1UvQOYJS7HwPcC1zf3EHuPs/da9y9prKyMkdv3baqqgj0vXs77S1FRDpdNoG+CchscQ9PbXubu7/s7m+lnv4aOC435eVGdXWsLfr88/muRESk42QT6EuBsWY22szKgCnAoswDzGxIxtNJwMrcldh+mtNFRIpBm4Hu7ruBGcASIqgXuvtyM5tjZpNSh11sZsvN7AngYuAzHVXwgdDQRREpBm0OWwRw98XA4ibbZmc8ngXMym1puTNgAPTrpxa6iCRb4q8UhYZJuhToIpJkRRHooEAXkeQrmkCvqoIXX4Rt2/JdiYhIxyiaQNeJURFJuqILdHW7iEhSFU2gH3YYdOumFrqIJFfRBHr37nD44Wqhi0hyFU2gg0a6iEiyFV2gr1kDu3fnuxIRkdwrukDftQvWr893JSIiuVdUgV5VFffqdhGRJFKgi4gkRFEF+iGHwMCBCnQRSaaiCnRoWL1IRCRpii7QNXRRRJIqq0A3s4lmtsrM1prZzFaO+5iZuZnV5K7E3KquhpdegpdfznclIiK51Wagm1kpMBc4ExgPTDWz8c0c1wu4BHg410XmkibpEpGkyqaFPgFY6+7r3L0eWABMbua4bwNXATtzWF/OaaSLiCRVNoE+DNiQ8XxjatvbzOxdwAh3v6u1H2Rm082s1sxq6+rq9rvYXBg1CsrKFOgikjztPilqZiXAD4HL2jrW3ee5e42711RWVrb3rQ9IaSmMG6dAF5HkySbQNwEjMp4PT21L6wUcBfw/M1sPnAAs6sonRjV0UUSSKJtAXwqMNbPRZlYGTAEWpXe6+zZ3H+Duo9x9FPAQMMndazuk4hyoroZnnoH6+nxXIiKSO20GurvvBmYAS4CVwEJ3X25mc8xsUkcX2BGqq2HPngh1EZGk6JbNQe6+GFjcZNvsFo59f/vL6liZy9EdcUR+axERyZWiu1IU4qQoqB9dRJKlKAO9d28YOlQjXUQkWYoy0EFzuohI8hR1oK9aBe75rkREJDeKNtCrquDVV2HLlnxXIiKSG0Ub6JkjXUREkkCBrkAXkYQo2kAfPhwOOkhDF0UkOYo20EtKoh9dLXQRSYqiDXTQ0EURSZaiD/T16+HNN/NdiYhI+xV1oFdVxTj0tWvzXYmISPsVdaBrpIuIJElRB3p6ki4FuogkQVEHekUFHHqohi6KSDJkFehmNtHMVpnZWjOb2cz+i8xsmZk9bmZ/NbPxuS+1Y2iki4gkRZuBbmalwFzgTGA8MLWZwP6dux/t7u8EvkcsGl0Q0oGuSbpEpNBl00KfAKx193XuXg8sACZnHuDur2U87Ql0bDxu25azBK6uhjfegE2b2j5WRKQryybQhwEbMp5vTG1rxMz+w8yeIVroFzf3g8xsupnVmlltXV3dgdQLW7fC8cfDrFkH9vomqqriXv3oIlLocnZS1N3nuvvhwNeB/2zhmHnuXuPuNZWVlQf2Rn37wqmnwlVXxa2dNHRRRJIim0WiNwEjMp4PT21ryQLg5+0pqlVmcM01MZn5zJnQrx9Mn37AP27IEOjVS4EuIoUvm0BfCow1s9FEkE8Bzss8wMzGuvua1NMPAWvoSCUlcP318NprcNFFMGgQTJ7c9uuaYaaRLiKSDG0GurvvNrMZwBKgFLjW3Zeb2Ryg1t0XATPM7APALmAr8OmOLBqA7t1h4UK47DI48cR2/aiqKvjzn3NUl4hInmTTQsfdFwOLm2ybnfH4khzXlZ2KCvh5qndn1y5YswbG7/8Q+OpqmD8fXn8dDj44xzWKiHSS5Fwpetll8J73wOOP7/dL0ydGV6/ObUkiIp0pOYH+la9A795wxhn7ncwauigiSZCcQD/0ULj33nh8+umwYUPrx2cYMybOs+rEqIgUsuQEOkRTe8mSGNL48Y9nfTVpeTmMHg0rV3ZseSIiHSmrk6IF5dhj4a67YgVos6xfNmEC3HxznBydNq0D6xMR6SDJaqGnnXwyHHdcPP7977NaY+5nP4uXffKT8N//rcm6RKTwJDPQ05Ytg6lT4dxzY1hjK/r2hbvvhvPOg8svhy98AXbv7pwyRURyIdmBfvTR0fS+6y74zGdg795WD+/RA264IWYU+MUv4KMfjZkYRUQKQfL60Ju66KKYofHyy6MZfs01rfatl5REl8uIEfDFL8Ipp8Cdd8LAgZ1XsojIgUh2Cz1t5kz46lejtf7ww1m95AtfgFtugaeeipkF1nTs7DQiIu1WHIFuFlPt/u1vcMIJWb9s8mR44IGYA+zEE+GhhzqwRhGRdiqOQIcI9fQkXvfdB9ddl9XLjj8e/v736K055RS47baOKlBEpH2KJ9Az/c//wAUXwB//mNXhY8ZEqB9zTJwonTu3g+sTETkAxRnoN94YrfXzzmuYLqANlZXR/XL22TBjBnzta20OmhER6VTFGeg9e8bQlSOOgHPOieZ3Fioq4kTp5z8PV18N558Pb73VsaWKiGSrOAMdolN8yRIYOhQWLMj6ZaWl0eVy5ZXxsg9+MKaOERHJt6wC3cwmmtkqM1trZjOb2X+pma0wsyfN7D4zG5n7UjvAoEEx8uVHP4rnWV7vbwZf/3r03Pztb3DSSfD88x1Yp4hIFtoMdDMrBeYCZwLjgalm1nRZoMeAGnc/BrgZ+F6uC+0wlZVxNdG6dTFD1zXXZH3N/3nnRSN/48YYDXkAa2uIiORMNi30CcBad1/n7vXAAqDRiszu/oC770g9fQgYntsyO8GuXdCnT1weeuyxcP/9Wb3slFPgwQejK+a97836HKuISM5lE+jDgMzVIjamtrXkAuDu5naY2XQzqzWz2rq6uuyr7AxVVZHGt9wSi4uedhp87GNQX9/mS486Ki46Gj0azjoLrr++E+oVEWkipydFzWwaUANc3dx+d5/n7jXuXlNZWZnLt84NM/jIR2Kli+98J5a0KyuLfXv2tPrSYcPgL3+B970v5gH79rc1Ba+IdK5sAn0TMCLj+fDUtkbM7APAN4BJ7l7Yg/nKy+Eb34Df/jaer1wJY8fGsJZWUrpPH1i8OOZUnz0bpk/XFLwi0nmyCfSlwFgzG21mZcAUYFHmAWZ2LPBLIsy35L7MPHvrrRjmOHVqNMEfe6zFQ8vKosvlG9+AX/8aJk2KuWBERDpam4Hu7ruBGcASYCWw0N2Xm9kcM5uUOuxq4GDgD2b2uJktauHHFaZ3vhOWLoV586K1ftxxcPHFLbbWzaLH5pe/jFEwo0fDnDnwyiudW7aIFBfzPHX01tTUeG1tbV7eu122boVvfSuuNv3ud2Pbnj0xzKUZ//hHhPsdd8DBB8f07JdeCkOGdGLNIpIYZvaIu9c0t694rxQ9UP36wY9/3BDm99wTs3b97/82e/iECbBoETzxRMwD88MfRov985+Poe8iIrmiQG+v0lLYuRNOPz1GyLSQ0sccA7/7HaxaBZ/+NFx7LYwbB9OmxSIaIiLtpUBvr9NOg+XL4YorYhz7+PExc1cLxoyJvvV16+CSS+DWW2Pp03POyXoxJRGRZinQc6G8HGbNiub3uefG+HWI+XVbOEcxbBj84AcxB8zs2TGG/YQT4vPhvvs0hl1E9p8CPZeGDYMbboALL4znv/oV/Mu/xDJH27c3+5L+/eMc63PPRcN+xQr4wAci3G+7TXOui0j2FOgdqXfvWF36Ix+J5D7tNPjpT5s9tFcv+MpX4Nln4Re/gLq6eNnRR8P8+bpASUTapkDvSFOnxlSMDzwAX/4ybNkCt9/esP/qq2M84xtvvL2pvDwa+KtXR5CbxZWn48ZF0O/cmYc/h4gUBI1D72w7d0Zqv/lmDEbfti0uL33f++DMM6NZPmrU24fv3RuZf8UVMaZ98OAYx37RRdGqF5HionHoXUl5edwfdBD8858xfn3GjGjJX3ppQwt+61ZYvJiSnTuYPDlmc7zvPjjyyFjPdMQI+NSnYOHC+EwQEVELvStZvz6a3f37x3JI06bFB8D73x+t97POgjFj+Mc/Yh2Ou+6K6QS6dYtzr2efDR/+cMwjJiLJ1FoLXYHeVe3cGWMZ7747pnBcvTq2r18PI0fC5s3sKenOQ89UcuedseZ1+gKlceMi2D/8YTj5ZOjePW9/ChHJMQV6EjzzDPzf/8Vk6wBTpsDvfw8DB8YKG0cdRd2hx7Gw/FPccUech62vjyl9J06McD/zzGj8i0jhUqAn0V//GjNAPvUULFsWV6tWVcGjjwKw69ypvLz+dR7ZdTR3rT+Kv247itVWTc17yt5uvR95ZIyiEZHC0Vqgd+vsYiRHTj45bml79zaan7f7Ib0ZvHIZH1r1Jz6UGsS+auzZnL9zEbNmweZZP2Vn5aEMm3g0J0wdzftOKXn7fG1W9uyJpfq2b2+4HXYYDBgQJ3hvv73xvqFD45zA8MJbblakUKiFnnT19dH//tRTcMghcMYZbF67g8HjelHicRnqG1TwdMl4HnznDMov/DQnj/0n1TfNpuT17Y1DeebMmNrgkUegppkGwvz5cP758Oc/x4lciMnLDj44huI88EBsf+aZuHJqwgQo0UArkf3R7ha6mU0EfgKUAr929yub7H8v8GPgGGCKu9/croold8rK3u5jTxs6pgJe2wYrVlD/yDK23PMUPWuXsX6986MLYRi7eJTbqC/vhfXuRVn/Xhw8ZAjlB1VgEK3sb34zRuRk3o49Nt7ghBPgxRfjStny8ujX2bgxBtED/OxnMY/w4MGxpNM558Cpp0KPHp38lyOSLG220M2sFFgNnA5sJJakm+ruKzKOGQX0Br4CLMom0NVC73rcYe3auIDpH/+I2R8feywa+RD5O2ECHH983L/73XHSdb+9+mqM3rntthjB8/rrMbD+ueci/HfvjrGYIrKP9rbQJwBr3X1d6octACYDbwe6u69P7dNUUgXMLMawjx0bPScQYf7kkxHu6ZBflLHAYHV145A/5pj4UtCq9PqsU6fG8Mz774cXXogC3KOlP3RotNwnTYpJz0SkTdm00D8OTHT3f089/yRwvLvPaObY64A7W2qhm9l0YDrAoYceetxzzz3XvuolL159NQbYZLbk//nP2NejR+RxOuAnTIDDD9+P0TT19fBf/xUTxa9ZE9smTIDLL4fJkzvijyNSULrMKBd3nwfMg+hy6cz3ltzp2zcWaDr99HjuDhs2NG7F/+pX8JOfxP5DDoF3vSvW/jjyyLgfPz6276OsDK66Cq68Ep5+OoL9ttsaZiVbty5WCDnnnPjU0ElVkbdlE+ibgBEZz4entokA0fo+9NC4nXtubNu9O+Z2f/jhuD35JPzmN40mlmTw4H1D/sgjUxc/mcERR8Tt8ssbVvyorY0Tqt/7HgwaFG9aWhqhP2hQTJkwf370wXfrFvu6dYs379kTbr45VpZK708fc+WV8eGweHEsANunT6wfm74df3y8v7sG70uXlU2gLwXGmtloIsinAOd1aFVS8Lp1i/70Y46Bz30utu3dGy35FSviOqj0/W9/G+dF0wYO3Dfkx483KiuBT3wCzjgjgnfx4hh7v2dPhDJES/6VV+ITJfOW/kBYsSLGyO/Z03j/VVfF/ttvh3nzGv9hevZsKHDatDiJkBn2o0fHHwLgD3+I/qfM/QMHRr9Tsdi1C15+Of4dXn45LngbODCGz954Y8wyOnJkw61nz3xXnBhZjUM3s7OIYYmlwLXu/l0zmwPUuvsiM3s3cCvQD9gJvOjuR7b2MzXKRdLcY1RjZsivWBG3115rOG7AgH2DfsyYOH+azvN227sX3norxs1v3Rq3HTtiGSmAm26KEwjpfVu3xvDMO+6I/aeeGuPtMx19dHxFATjxxHhcUREzblZUxLb0B8KMGfDSS7Evvf/oo2NlcYiVxvfsafz6oUNjAh+Iq4ZLShp/O+ndu6F/a+vWxvu6dWu528o9rj8oKYlrCV57Lf6c6aBO3/7t36L/7dFH4zqDpqtz3XRTTFXxwAPx99PUPffE65cujT9fZtiPHBm161vR23TpvxQkd9i0qSHcMwM/c8rgsrL4f3/YYdFYbnrfr18nFl1fH2eNMwO/e/eGD4S5c2NZqh07Yk78HTtiWNF3vhP7zz47Tgan9+3YEd9Ibr019g8aFAulZDrvvGj5QrR2d+xovP/CC2N1lL17m//ku+wy+P73I7CHDm0I+e3b49vLFVfEmrnPPddorn769m1YQ/H882Ok0lVXxbb+/SOI+/eHd7wjWujuUcMLL8TPSt8+9akYyTR/ftTatP5ly+I6isWL4wOlaeAPGdL4Q2nPnvim9uab8eGcHiW1Zk38Qr35ZsOtpCRGW0F8mCxb1rBv585oRfzgB7H/e9+LX0CzeF26r3H27Nh/1VUxeV56n1l8M/vSlxr2b9kSowamTWv21ycbCnRJFPfIhBUr4qLTdesiI599Nh5nzIAARHd4S2E/alSBXc+0eXND0KdDf8CAaMVDdBnV1zfuTho3Dk46KcL0pz9t6G5K359wQnxo7NgRI4zS23v3jkB+73vjooPdu+MvvH//CPOOuFbAPVr9mYE/fXp8UP3kJzBnzr7/wK++Gv/Il14a80rv2tWwL72YDMTSX/PnN35t//7xjQjgox+NaUvT347Ky+MX5f77Y/9558GDD0aN6Q+nI4+MbxgAH/xgXLiR3ucef7eLF8f+446LheQ//nG47roD/itSoEtR2batccBn3j/7bDTa0syiUdo06IcNa7hpZaguZvt2eP75CPuNGyPwIU6MP/xwQyCnu6Q++9nY/9RTjbuzystjf3p+oQI54a1AF0nZuzdmJWgu7Neti2/kTf9L9OoVoZ8Z8sOGNd42eLAubpXO0WXGoYvkW0lJBPHQodEL0dRbb0Xjb9OmhtvmzQ2P//zneJ6awPJtZtG93VroDx0aPRUF0AiUAqVAF8nQo0fD9Act2bs3JotsGvbp27p1MV19067e9M8fPDjO47V2P2iQVpqS/adAF9lPJSURuIMGxRWwLXnzzcaBv3lzdPe8+GKc1F2zJlYZbC74Ic51thX8Q4ZEl5Ba/QIKdJEOc9BBMWqtrWuK3norrkVKB33mffrxqlXxOD3zZaayssajBA85pPHjlu4POqhj/tySPwp0kTzr0aNh6oTWuMew9szQf+GFGLiRvtbnlVdiCuT088wRPU0ddFDr4d+3b8OtT5/GjwtqqGcRUaCLFAizhgA+stXrsIN7dPtkXoXf2v3KlQ3Pm570baq8vOXAb+4DoOnjigp1E3UEBbpIQplFcFZUxPoh2Upf8b9tW1yzk75v+jjz+SuvxNDP9PbmuoYylZY2hHufPg231p433bdfa+AWCQW6iDRiFheJ9u69fx8EmXbubP1DIH3LfJ7+QNi2LWYhaOsSmR49Ggf8wQc3vvXsue+21m4VFYU/G7MCXURyrrw8RuGkl5HdX3v3xgSXmYHf2ofBtm0xNfPmzfG611+P59u3x8/KVkVF8x8KPXvu+3h/9rW5ileOKNBFpMspKWn4ltAe7nFiOB3yLd3eeGPfbdu3N2x/8cV4nH6+Y0fb3yAyde/eOOi/9S34139t35+tOQp0EUkss/i2UF4e4/pzJX3COf1hkA76bB/375+7WjIp0EVE9lPmCeeuJKtTAGY20cxWmdlaM5vZzP4eZvb71P6HzWxUzisVEZFWtRnoZlYKzAXOBMYDU81sfJPDLgC2uvsY4EfAVbkuVEREWpdNC30CsNbd17l7PbAAmNzkmMnA9anHNwOnmemyARGRzpRNoA8DNmQ835ja1uwx7r4b2Abs0+1vZtPNrNbMauvq6g6sYhERaVanDqN393nuXuPuNZWVlZ351iIiiZdNoG8CMq8XG57a1uwxZtYN6AO8nIsCRUQkO9kE+lJgrJmNNrMyYAqwqMkxi4BPpx5/HLjf87W2nYhIkWpzHLq77zazGcASoBS41t2Xm9kcoNbdFwG/AW4ws7XAK0Toi4hIJ8rbItFmVgc8d4AvHwC8lMNyOloh1VtItUJh1VtItUJh1VtItUL76h3p7s2ehMxboLeHmdW2tOp1V1RI9RZSrVBY9RZSrVBY9RZSrdBx9Rb4ZJEiIpKmQBcRSYhCDfR5+S5gPxVSvYVUKxRWvYVUKxRWvYVUK3RQvQXZhy4iIvsq1Ba6iIg0oUAXEUmIggv0tuZm7yrMbISZPWBmK8xsuZldku+asmFmpWb2mJndme9aWmNmfc3sZjN72sxWmtmJ+a6pNWb25dTvwVNmdpOZdak1683sWjPbYmZPZWw7xMzuNbM1qft++awxrYVar079LjxpZreaWd88lvi25mrN2HeZmbmZ5WwtpYIK9CznZu8qdgOXuft44ATgP7pwrZkuAVbmu4gs/AT4k7tXA++gC9dsZsOAi4Eadz+KuOK6q11NfR0wscm2mcB97j4WuC/1vCu4jn1rvRc4yt2PAVYDszq7qBZcx761YmYjgDOA53P5ZgUV6GQ3N3uX4O4vuPujqcfbicBpOu1wl2Jmw4EPAb/Ody2tMbM+wHuJKSdw93p3fzWvRbWtG3BQavK6CmBznutpxN3/QkzbkSlznYPrgXM6s6aWNFeru9+Tmrob4CFiEsG8a+HvFWIhoK8BOR2VUmiBns3c7F1Oakm+Y4GH81xKW35M/JLtzXMdbRkN1AG/TXUP/drMeua7qJa4+ybg+0Rr7AVgm7vfk9+qsjLI3V9IPX4RGJTPYvbDZ4G7811ES8xsMrDJ3Z/I9c8utEAvOGZ2MPBH4Evu/lq+62mJmX0Y2OLuj+S7lix0A94F/NzdjwXeoOt0B+wj1fc8mfggGgr0NLNp+a1q/6RmT+3yY5zN7BtEd+eN+a6lOWZWAVwOzO6In19ogZ7N3Oxdhpl1J8L8Rne/Jd/1tOEkYJKZrSe6sk41s/n5LalFG4GN7p7+xnMzEfBd1QeAZ929zt13AbcA78lzTdn4p5kNAUjdb8lzPa0ys88AHwbO78LTdx9OfLA/kfq/Nhx41MwG5+KHF1qgZzM3e5eQWlP1N8BKd/9hvutpi7vPcvfh7j6K+Hu93927ZCvS3V8ENphZVWrTacCKPJbUlueBE8ysIvV7cRpd+CRuhsx1Dj4N3J7HWlplZhOJ7sJJ7r4j3/W0xN2XuftAdx+V+r+2EXhX6ne63Qoq0FMnPdJzs68EFrr78vxW1aKTgE8SLd3HU7ez8l1UgnwRuNHMngTeCVyR33JalvomcTPwKLCM+H/XpS5VN7ObgL8DVWa20cwuAK4ETjezNcS3jCvzWWNaC7VeA/QC7k39X/tFXotMaaHWjnu/rvvNRERE9kdBtdBFRKRlCnQRkYRQoIuIJIQCXUQkIRToIiIJoUAXEUkIBbqISEL8f6sJlYJbUH/NAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_mlp = MLP([D_in, 256, 128, 64, D_out]).to(device) # [D_in, 512, 256, 128, 64, D_out]\n",
    "\n",
    "optimizer = optim.SGD(model_mlp.parameters(), lr=0.01, momentum=0.5)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "model_mlp, losses, accuracies = train_val_model(model_mlp, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=15, log_interval=2)\n",
    "\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HXhHtX1TkUba"
   },
   "source": [
    "### 1.3 Convolutional Neural Network\n",
    "\n",
    "Convolutional layers capture patterns corresponding to relevant features independently of where they occur in the input. To do so, they slide a window over the input and apply the convolution operation with a set of kernels or filters that represent the features. Although it is not their only field of application, convolutional neural networks are mainly praised for their performance on image processing tasks.\n",
    "\n",
    "The training and validation management for the CNN implementation will be performed as the feed-forward network, however we will have to define the network's architecture.\n",
    "\n",
    "For that we will implement a CNN class to define how many layers it comprises and how the layers will be connected.\n",
    "\n",
    "The initialization (`__init__`) function will define the architecture and the `foward` function will implement how the different layers are connected. This architecture will be a sequece of 2 convolutional layers ([nn.Conv2d](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html)) (1st: output channels 10, kernel size 5; 2nd: output channels 20, kernel size 5), then 2 fully connected layers ([nn.Linear](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)) (1st: output features 50; 2nd: output features 10 (the number of classes)). Once again, the final layer will be a [softmax](https://pytorch.org/docs/stable/generated/torch.nn.functional.log_softmax.html#torch.nn.functional.log_softmax) function  that will choose the most probable class of the 10 in the input.\n",
    "\n",
    "Between the second convolution layer and the first fully connected, we will set a dropout layer ([nn.Dropout2d](https://pytorch.org/docs/stable/generated/torch.nn.Dropout2d.html)). The idea behind dropout is to disable a percentage of randomly selected neurons during each step of the training phase, in order to avoid overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "id": "PZ0mCl24EoaM"
   },
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    \"\"\"Basic Pytorch CNN for MNIST-like data.\"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)\n",
    "        self.conv2_drop = nn.Dropout2d()\n",
    "        self.fc1 = nn.Linear(320, 50)\n",
    "        self.fc2 = nn.Linear(50, 10)\n",
    "\n",
    "    def forward(self, x, T=1.0):\n",
    "        # Batch size = 64, images 28x28 =>\n",
    "        #     x.shape = [64, 1, 28, 28]\n",
    "        x = F.relu(F.max_pool2d(self.conv1(x), 2))\n",
    "        # Convolution with 5x5 filter without padding and 10 channels =>\n",
    "        #     x.shape = [64, 10, 24, 24] since 24 = 28 - 5 + 1\n",
    "        # Max pooling with stride of 2 =>\n",
    "        #     x.shape = [64, 10, 12, 12]\n",
    "        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n",
    "        # Convolution with 5x5 filter without padding and 20 channels =>\n",
    "        #     x.shape = [64, 20, 8, 8] since 8 = 12 - 5 + 1\n",
    "        # Max pooling with stride of 2 =>\n",
    "        #     x.shape = [64, 20, 4, 4]\n",
    "        x = x.view(-1, 320)\n",
    "        # Reshape =>\n",
    "        #     x.shape = [64, 320]\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.dropout(x, training=self.training)\n",
    "        x = self.fc2(x)\n",
    "        x = F.log_softmax(x, dim=1)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mv9vdZZ7OlSh"
   },
   "source": [
    "As previously, lets describe the model to be trained. We will use the ADAM optimizes ([optim.Adam](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam)), with learning rate 0.001, and the same negative log likelihood ([nn.NLLLoss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "ImTlr5JeEsb6",
    "outputId": "a8d8e0a6-e3cc-4b37-d022-adc0241e8b88"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/24\n",
      "----------\n",
      "train Loss: 0.5470 Acc: 83.14%\n",
      "val Loss: 0.1097 Acc: 96.72%\n",
      "\n",
      "Epoch 2/24\n",
      "----------\n",
      "train Loss: 0.2096 Acc: 93.83%\n",
      "val Loss: 0.0579 Acc: 98.28%\n",
      "\n",
      "Epoch 4/24\n",
      "----------\n",
      "train Loss: 0.1699 Acc: 94.98%\n",
      "val Loss: 0.0485 Acc: 98.46%\n",
      "\n",
      "Epoch 6/24\n",
      "----------\n",
      "train Loss: 0.1550 Acc: 95.46%\n",
      "val Loss: 0.0444 Acc: 98.61%\n",
      "\n",
      "Epoch 8/24\n",
      "----------\n",
      "train Loss: 0.1385 Acc: 95.94%\n",
      "val Loss: 0.0395 Acc: 98.74%\n",
      "\n",
      "Epoch 10/24\n",
      "----------\n",
      "train Loss: 0.1330 Acc: 96.06%\n",
      "val Loss: 0.0360 Acc: 98.79%\n",
      "\n",
      "Epoch 12/24\n",
      "----------\n",
      "train Loss: 0.1220 Acc: 96.41%\n",
      "val Loss: 0.0352 Acc: 98.85%\n",
      "\n",
      "Epoch 14/24\n",
      "----------\n",
      "train Loss: 0.1208 Acc: 96.44%\n",
      "val Loss: 0.0326 Acc: 98.94%\n",
      "\n",
      "Epoch 16/24\n",
      "----------\n",
      "train Loss: 0.1156 Acc: 96.51%\n",
      "val Loss: 0.0318 Acc: 98.97%\n",
      "\n",
      "Epoch 18/24\n",
      "----------\n",
      "train Loss: 0.1127 Acc: 96.69%\n",
      "val Loss: 0.0312 Acc: 99.02%\n",
      "\n",
      "Epoch 20/24\n",
      "----------\n",
      "train Loss: 0.1092 Acc: 96.65%\n",
      "val Loss: 0.0338 Acc: 98.86%\n",
      "\n",
      "Epoch 22/24\n",
      "----------\n",
      "train Loss: 0.1078 Acc: 96.85%\n",
      "val Loss: 0.0307 Acc: 99.00%\n",
      "\n",
      "Epoch 24/24\n",
      "----------\n",
      "train Loss: 0.1077 Acc: 96.74%\n",
      "val Loss: 0.0287 Acc: 99.04%\n",
      "\n",
      "Training complete in 7m 6s\n",
      "Best val Acc: 99.04%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAdHklEQVR4nO3deXQc5Znv8e8jyYtsGS9YCC8SEsQwONgYULxMEkzuAMGBMTAEMOcwwDkhzk3CQBIwy4VJAslMWMOS8dyJLyGDE4hZEsCZmBiHIQMmLJYJ4wUCeBzbyLstvAhja/F7/3jU6ZaQrLbcUqmrfp9z6lR3Van7KTf8quqtt6oshICIiMRLQdQFiIhI7incRURiSOEuIhJDCncRkRhSuIuIxFBRVF88fPjwUFlZGdXXi4jkpaVLl24LIZR2tlxk4V5ZWUlNTU1UXy8ikpfMbG02y6lZRkQkhhTuIiIxpHAXEYkhhbuISAwp3EVEYkjhLiISQwp3EZEYyrtwX7wYbroJdKdiEZGO5V24L10Kt98O27dHXYmISO+Vd+FeXu7j99+Ptg4Rkd5M4S4iEkMKdxGRGMq7cD/iCOjTR+EuInIgeRfuBQW+965wFxHpWN6FOyjcRUQ6k7fhvm5d1FWIiPReeRvu69dDc3PUlYiI9E55G+5NTbB5c9SViIj0Tnkb7qB2dxGRjijcRURiSOEuIhJDeRnuQ4fCwIEKdxGRjuRluJupO6SIyIFkFe5mdpaZvWNmq8zsxnbmX2FmW83szZbhytyX2pouZBIR6VhRZwuYWSEwGzgDqAWWmNn8EMJbbRZ9LIRwVTfU2K7yclixoqe+TUQkv2Sz5z4RWBVCWB1CaADmAed2b1mdKy+HTZugoSHqSkREep9swn0UkNkAUtsyra0LzGyZmT1pZuXtfZCZzTSzGjOr2bp1axfKTSsv90ftbdhwSB8jIhJLuTqh+mugMoQwHlgEPNzeQiGEOSGE6hBCdWlp6SF9obpDioh0LJtwXw9k7omPbpn2FyGE7SGEfS1vHwROyU15Hauo8LF6zIiIfFw24b4EGGNmVWbWF5gBzM9cwMxGZLydDryduxLbpz13EZGOddpbJoTQZGZXAQuBQuChEMJKM7sNqAkhzAeuNrPpQBNQB1zRjTUDfhHT0KEKdxGR9nQa7gAhhAXAgjbTvp3x+ibgptyW1jn1dRcRaV9eXqGaonAXEWmfwl1EJIbyPty3b4c9e6KuRESkd8nrcE91h9Teu4hIa3kd7uoOKSLSPoW7iEgM5XW4j2q5w43CXUSktbwO9379oKxM4S4i0lZehzuoO6SISHsU7iIiMZT34V5R4XeGDCHqSkREeo+8D/fycqivh507o65ERKT3iEW4g5pmREQyKdxFRGJI4S4iEkN5H+4jRkBhocJdRCRT3od7YSGMHKlwFxHJlPfhDunukCIi4mIR7rqQSUSktdiEe22tLmQSEUmJTbjv2wdbt0ZdiYhI7xCbcAc1zYiIpCjcRURiSOEuIhJDsQj30lJ/cIe6Q4qIuFiEu5m6Q4qIZIpFuIPCXUQkk8JdRCSGYhXuGzZAc3PUlYiIRC+rcDezs8zsHTNbZWY3HmC5C8wsmFl17krMTnm5B/vGjT39zSIivU+n4W5mhcBsYBowFrjEzMa2s9wg4BrgtVwXmQ11hxQRSctmz30isCqEsDqE0ADMA85tZ7nvAXcAe3NYX9ZS4a7ukCIi2YX7KCBzf7i2ZdpfmNnJQHkI4TcH+iAzm2lmNWZWszXHN4KpqPCx9txFRHJwQtXMCoAfAtd2tmwIYU4IoTqEUF1aWnqoX93K4MEwaJDCXUQEsgv39UB5xvvRLdNSBgEnAL83szXAZGB+VCdVFe4iItmF+xJgjJlVmVlfYAYwPzUzhLAzhDA8hFAZQqgEXgWmhxBquqXiA1C4i4i4TsM9hNAEXAUsBN4GHg8hrDSz28xsencXeDAU7iIiriibhUIIC4AFbaZ9u4NlTzv0srqmvBw2b/YHd/TrF1UVIiLRi80VqpDuDllbG20dIiJRi1W4qzukiIiLVbjrKlURERercB892scKdxFJuliF+4ABcPjhCncRkViFO6g7pIgIxDTcdfMwEUm6WIa79txFJOliF+4VFbBjB9TXR12JiEh0Yhfu6g4pIqJwFxGJJYW7iEgMxS7cR40CM4W7iCRb7MK9Tx848kh1hxSRZItduIO6Q4qIxDLcKyoU7iKSbLEM99SeewhRVyIiEo3YhvuePfDBB1FXIiISjdiGO6hpRkSSS+EuIhJDsQ53dYcUkaSKZbiXlUFRkfbcRSS5YhnuhYV+parCXUSSKpbhDurrLiLJFttw11WqIpJksQ732lrYvz/qSkREel6sw72xEbZsiboSEZGeF+twB3WHFJFkin24q91dRJIoq3A3s7PM7B0zW2VmN7Yz/3+b2XIze9PMFpvZ2NyXenAU7iKSZJ2Gu5kVArOBacBY4JJ2wvvREMK4EMIE4E7gh7ku9GAdfjgUFyvcRSSZstlznwisCiGsDiE0APOAczMXCCHsyng7EIj8Zrtm6g4pIslVlMUyo4DMiKwFJrVdyMy+DnwL6Av8r/Y+yMxmAjMBKioqDrbWg6ZwF5GkytkJ1RDC7BDCMcANwC0dLDMnhFAdQqguLS3N1Vd3SOEuIkmVTbivB8oz3o9umdaRecB5h1BTzpSXw4YN3t9dRCRJsgn3JcAYM6sys77ADGB+5gJmNibj7dnAe7krsevKy/1Rexs2RF2JiEjP6rTNPYTQZGZXAQuBQuChEMJKM7sNqAkhzAeuMrPTgUbgA+Dy7iw6W5ndIY86KtpaRER6UjYnVAkhLAAWtJn27YzX1+S4rpxInbNVu7uIJE1sr1AFXcgkIskV63AfNAgGD1a4i0jyxDrcQd0hRSSZEhHuujOkiCRNIsJde+4ikjSJCPdt2+Cjj6KuRESk58Q+3FPdIWtro61DRKQnxT7c1R1SRJJI4S4iEkOxD/fRo32sHjMikiSxD/f+/aG0VHvuIpIssQ93gMpKePVV3fpXRJIjEeE+axYsXw7f/nbny4qIxEEiwv3CC2HmTLj9dnjuuairERHpfokId4B774VPfhL+/u9h06aoqxER6V6JCfcBA+Cxx2D3brjsMti/P+qKRES6T2LCHXzP/f77YdEiuOuuqKsREek+iQp3gCuvhIsugptvhldeiboaEZHukbhwN4M5c/zK1UsugR07oq5IRCT3Ehfu4E9nmjcP1q/3PfkQoq5IRCS3EhnuAJMmwT//M/zyl74nLyISJ4kNd4Brr4XPfx6+8Q2/yElEJC4SHe4FBfDww95Mc/HF8OGHUVckIpIbiQ53gLIy+PnP4U9/8j14EZE4SHy4A5x+Otx4Izz4oJ9oFRHJdwr3FrfeClOm+D1oVq+OuhoRkUOjcG/Rpw88+igUFsKMGdDQEHVFIiJdp3DPUFnpTTNLlsAtt0RdjYhI1ync27jgAvjqV/3eM488EnU1IiJdk1W4m9lZZvaOma0ysxvbmf8tM3vLzJaZ2fNmdlTuS+0599wDU6fCpZf6hU66glVE8k2n4W5mhcBsYBowFrjEzMa2WeyPQHUIYTzwJHBnrgvtScXFsHChh/vNN/stCvSIPhHJJ9nsuU8EVoUQVocQGoB5wLmZC4QQXggh7Gl5+yowOrdl9rx+/WDuXH8030MPwbRpusmYiOSPbMJ9FPB+xvvalmkd+RLw7KEU1VuYeRfJf/93ePFF+PSnYe3aqKsSEelcTk+omtmlQDXQ7qMwzGymmdWYWc3WrVtz+dXd6vLLvZlm/Xq/4VhNTdQViYgcWDbhvh4oz3g/umVaK2Z2OnAzMD2EsK+9DwohzAkhVIcQqktLS7tSb2Q+9zl/uEdxMZx6KjzzTNQViYh0LJtwXwKMMbMqM+sLzADmZy5gZicBP8aDfUvuy+wdjj8eXn0Vxo2D88+H++5TTxoR6Z06DfcQQhNwFbAQeBt4PISw0sxuM7PpLYvdBZQAT5jZm2Y2v4OPy3tlZfDCC3DeefDNb8LVV0Nzc9RViYi0ZiGiXc/q6upQk8eN183NcMMN3if+nHPgF7+AkpKoqxKRuDOzpSGE6s6W0xWqXVRYCHffDbNnw4IF3g6/YUPUVYmIOIX7Ifra12D+fHj3XTj5ZLj+er83jdriRSRKCvccOPtsWLwYTjoJ7r0XJk6Eqiq47jp47TUFvYj0PIV7jkyYAM8+C1u2wE9/Cp/8JDzwAEye7HebvPZa70q5f3/UlYpIEuiEajf64ANvsnnySXjuOb9H/OjR8MUvwoUXevAXaPMqIgch2xOqCvcesnOnB/0TT/jVrg0NMGqU97T57Gd9qKiIukoR6e0U7r3Yzp3wH//hQf+f/wm7d/v0igr4zGfSYX/88dqzF5HWFO55orkZli2Dl17yk7IvvQSbNvm8YcP8ZmWpwD/lFOjbN9p6RSRaCvc8FYI/oPull9KB/+67Pq9/f79x2ZQp3iNn4kRv2hGR5FC4x8jmzfDyy+mwf/NNaGryeSNHpoN+4kSorobBgyMtV0S6kcI9xvbu9YB//fX08N576fl/9VetA3/8eH/4iIjkv2zDvagnipHc6t/fu1FOnpyeVlfn95lPhf3Chf4kqdTyU6f606SmTYMxY/xBJCISX9pzj6kQ4P33PegXL4bf/hbeecfnVVWlg/5zn4OBA6OtVUSyp2YZ+Zg//9lD/tln4fnnYc8e731z6qlw1lke9scfr716kd5M4S4HtG+f79E/+6wH/sqVPr2iwoP+9NO9Z055ucJepDdRuMtBWbfOQ/63v4Xf/S59YdWRR/pJ2UmTfPjUp+Cww6KtVSTJFO7SZY2N8N//7Xe0TA2pvvZm3hsnFfYTJ/pjB/v0ibZmkaRQuEtOffCB36c+M/C3bfN5xcV+L/sJEzzox43zu2Kqv71I7qkrpOTU0KFw5pk+gPfGWbPGQ/711308d266OQe8/f6EE9KBf8IJvtevPvci3U/hLl1i5l0qq6pgxgyfFoK33S9fDitW+Hj5cli0yJt6wB9PeOyxHvbHHec3Rtu/3++xkzm0nZZ6X1zsJ3krKtLDyJFQpP+SRVrR/xKSM2Zw1FE+nHNOenpjo7fZZwZ+TQ08/nj67woLPegLC9ND2/eFhVBf701EmQoK/B47qbBvG/7HHecXcokkidrcJTLNzR7MB9vVsr7eL9Bat6794f3300cK4Hv1J56YPgk8aZJfpavbKUs+0glVSaz9+/1ma+vWwdq18MYbfl5gyRLfMAAMGdK6i+ekSTB8eOefvWePP0pxyxb/jtTrpiY/Yqms9EFNRdJdFO4ibTQ3w9tvt+7xs2JF+rm2Rx/tIT9+vG8E2gvx1MahM0VF3jyUCvu2g8JfukrhLpKF+npYurR14K9f7002paVQVgZHHOFDe6/Lynw5M28OWrOm9bB2rY83bPATzimFhX6kMHSoD0OGpF9nDpnThw3z79L5g2RTuIt00a5dUFKS2zb5ffs+Hv5bt/rJ4dSwY0d6nDqaaM/gwX7lcFmZDx29LitTt9M4Uj93kS7qjtsr9OsHn/iED53Zv9+vF0iFfWqoq/OmoU2bvKlo82Z/ROOiRb5se0pK/Lv79vVx5tB2Wup9SYkfMQwefOBxcbHuO9SbKdxFepmCAg/QwYP9JG029u5Nnx/IDP+6Oj9qSA0NDR9/v3t362n19b6xSD3tqyNFRV7jYYd5yLd3jUJHr/v2hdGj011XU91XU6/Ly30jI12Xn+G+bBkcc4xuRC7Son//dL/+XAgBPvoIdu70oO9ovGOHN2OZtb4u4UDXLBQU+MaottabqhYuhI0bW5+TAD/PkBn8xcXtb6DaGxoavDvsoEF+riLbobjY//ajjzof9uzxsRkMGOB/29m4uLjnuuDmX7jX1fkNyCdPhvnzfRdARHIqFVgDBsCIEd3/fY2NftI5dZ1C6jqG1OtXXvHQ7qhJKTUcdlj6dVGRH5XU1Xkvqbo6HxoaDrze3X0asl8/+NGP4Mtf7t7vySrczews4H6gEHgwhHB7m/mnAvcB44EZIYQnc1xn2rBhcM89cOWVcNll8MgjvjsgInmrT5/01c3dKQTf406dw2g71Nf7UVBqL7ujIXNPPHWUk9qT37On9ev2xuPGde96QhbhbmaFwGzgDKAWWGJm80MIb2Ustg64AriuO4r8mC99yX+J66/3Y7d//Ved2RGRTpl5a+7Agd7mH2fZ7LlPBFaFEFYDmNk84FzgL+EeQljTMu8AHbhybNYs2L4d7rgDPv1puPTSHvtqEZHeLptwHwW8n/G+FpjUlS8zs5nATICKXJz5+cEP/B6yF1986J8lIhIjPXrrpBDCnBBCdQihurS09NA/0AyuuMIb7DZvhmeeOfTPFBGJgWzCfT1QnvF+dMu03uXmm+GCC7wHjYhIwmUT7kuAMWZWZWZ9gRlA70vQe+/1Z71ddBH8139FXY2ISKQ6DfcQQhNwFbAQeBt4PISw0sxuM7PpAGb2KTOrBS4EfmxmK7uz6HYNGgQLFvit/f72b+GPf+zxEkREeov43TisttZ7z1RVwQsvqIukiMRKcm8cNno0PP+839lIwS4iCRXPB4194hN+s+yGBr/Qafv2qCsSEelR8Qz3lBUr4IEH4AtfyP4ROiIiMRDvcD/5ZHj8cX/UzqRJMG+e329URCTm4h3uANOnw9NP+919LrkEvv71qCsSEel28Q93gHPO8SaaX/0KvvpVn7ZqlfeNV3ONiMRQMsId/A75558PJ57o7596Cr71Lb/H6G23+V0mRURiIjnh3tasWfCHP3if+O98x0P+H/8x6qpERHIiueEOMGWK34tm2TJvm898yvD63nf7HBGRbMXvIqauGDfOn+iUulr35Zf9UX4nnADjx6eHKVP8OV4iIr2cwj1T6orWo4/2JpolS+D3v4ef/9ynv/wy/PVf+43Jnn46Hfpjx/rztkREegmFe3tGjIDvfjf9vq4Oli+HCRP8/cqV8OMf+8MQwU/WHnssLF4Mhx/uG4E1a2DkSB9GjYKSkh5eCRFJMoV7NoYNg6lT0++/9jX4yldg9Wpvr1+2DN55x5cD+OlP4Sc/af0Zw4fDli1+dPDgg/Deex76I0b4MHo0VFb22CqJSLwp3LuqsBDGjPHhggtaz7vvPu+Ns2GDn5jdsMEfe55q9nnxRXjsMb/3Tcoxx3jfe4Arr4T/+Z908I8c6Y8TPPtsn79pk39Wv37Qt6+PCwu7fZVFJH8o3LtDSQkcd5wP7Zk7Fx5+GLZtg40bfci8LcKQIdDYCK+95vM++shP8KbC/dRTfc8/0znnwK9/7a9PPNGbkjLD/+yz4fvf9/nnn++f37evD336wN/8jT+yMAS47jqvobIyPYwcqQ2ISB5RuEfFDEpLfRg/vvW8u+9Ovw4Bdu2CDz9MT/ve9/xOlw0NsG+fj485Jj3/jDM83FPz9u3zsE6pq/MrcxsbfX5jo/fzB9/IzJnz8St3Z82CO+/0Oq6+unXwV1X5EYbCX6TXiN/DOiQ39u6Fdev8xPCaNb4BmjwZ/vxnv/Br48bWyz/wAPzDP3jT0tVXwxFHQFlZejx1KpSXQ1OTL1+U4/2KEHyDuWmT1/jBB37dwo4dMHAgXHghDBjgG6eiIj+aEclDyX1Yh+RG//7eA+jYY1tPr6rycwh798Latenw/+xnfX59vZ84Xr7cx6nzCk8/7eH+u9/BtGneq2j4cA/ZoiI/WjjlFH/Qyg9+4NMyhzvv9C6qv/kN/Mu/eGhnBvh77/nnP/hg+1can3++j2+9Fe66y49kMjc+8+b5kccf/gCbN6ePqoYPh6FDvUeUSB5RuEvX9O/f/nmFCRMgdUQWAuzc6SF/5JE+rarKb/ewZYs3LTU2+tC/v89vbvYNR1OTD42NPk5tJPbu9b8bOhQqKjykhwxJ//3FF0N1tc8fMsQvOtuzx5+xC35v/0GDPMC3bPHx6tXpJqXZs+HRR1uvU2mpLwveRXbFCg/91AagogLOO8/nL17sG50Q0hfFDR3q50nAN147d/q8vXt9Y3jkkXDuuT7/+ut941lfnx5OOw1uv93nT5rk42HD0sPUqfDFL/r0hQt9vVPzSkp83YqK/Dt37/bx/v3pcXGxH900Nfm/R2Fh6/M1uT7K6kh9ve8obNzoR2Cp8U03+b/zU095F+QBA7ze1PiWW3xda2rgjTdazxs0yP976NPH/xsqLExM86GaZUQybdniz+Hdtg22bvXx/v3wzW/6/GuugUWLfPr27T5v3DjvDgt+kdsrr7T+zMmT09PGjfONQ6bTT/fPBJg40T+3pMSHQYM83G+80edfdpnXWFfnw/btcOml8KMfpU+St3XDDb5xqKvzI6a2vv99uPlmPxJrrzvufff5ev/pT76Rygz+vn39xnvTp8Nbb3mTXJ8+6SOuPn38Bn2TJ/v8++/36Y2NHtybNsG//Zs/e2HuXLj88tbfPWCAH02deKIfXd17r2+sU8OHH3o35NS1Kbfe+vH6d+yAwYN9w3nXXR76hx2WHl55xQP/Zz/z1wUF3sRn5t+f2rD+7Gf+O6fmmfmGNPXbPP+8/xunPnfQIN+wjxr18ZoOgZplRLriiCN86Mj996dfNzf7XvqePelpDz2UPvmdGRApTzzhJ7jN/GijpKT1LS1ef/3A9c2d+/FpqR20ggIPp1Twp06cT5ni8wcM8JP1qfAqKPBh8mSff/jhvmfc3Nz6ZHzq70tKvNtv5ryGhvQFek1N3rNr9+70kVdTk78HPyqYP9+nFRT4EcuIEen6p071LsIjRqTnZV78N2OGDx2ZNQu+/GX/99+zJ33kkzpqO/NM/zfYvds7Keza5UdPqT35N97w3yfzyOaww9Lh/txzftvw1LwQPLhT4X7HHemNdMrYsX7RI8DnP+/NlX/3d9602M205y4ikgsbN/qR1K5d6Q1Iv35+VANwzz1+9HPSSX4hZBdpz11EpCelLjrsyLXX9lwtJP2WvyIiMaVwFxGJIYW7iEgMKdxFRGJI4S4iEkMKdxGRGFK4i4jEkMJdRCSGIrtC1cy2Amu7+OfDgW05LCffJHn9k7zukOz117q7o0IIpZ39QWThfijMrCaby2/jKsnrn+R1h2Svv9b94NZdzTIiIjGkcBcRiaF8Dfc5URcQsSSvf5LXHZK9/lr3g5CXbe4iInJg+brnLiIiB6BwFxGJobwLdzM7y8zeMbNVZnZj1PX0JDNbY2bLzexNM4v9Y6zM7CEz22JmKzKmDTOzRWb2Xst4aJQ1dpcO1v27Zra+5fd/08y+EGWN3cXMys3sBTN7y8xWmtk1LdOT8tt3tP4H9fvnVZu7mRUC7wJnALXAEuCSEMJbkRbWQ8xsDVAdQkjEhRxmdipQD8wNIZzQMu1OoC6EcHvLxn1oCOGGKOvsDh2s+3eB+hDC3VHW1t3MbAQwIoTwhpkNApYC5wFXkIzfvqP1v4iD+P3zbc99IrAqhLA6hNAAzAPOjbgm6SYhhBeBujaTzwUebnn9MP4ffex0sO6JEELYGEJ4o+X1buBtYBTJ+e07Wv+Dkm/hPgp4P+N9LV1Y6TwWgOfMbKmZzYy6mIiUhRA2trzeBJRFWUwErjKzZS3NNrFslshkZpXAScBrJPC3b7P+cBC/f76Fe9J9JoRwMjAN+HrLoXtiBW9TzJ92xUP3f4FjgAnARuCeSKvpZmZWAvwS+EYIYVfmvCT89u2s/0H9/vkW7uuB8oz3o1umJUIIYX3LeAvwFN5MlTSbW9okU22TWyKup8eEEDaHEJpDCPuB/0eMf38z64MH2yMhhF+1TE7Mb9/e+h/s759v4b4EGGNmVWbWF5gBzI+4ph5hZgNbTq5gZgOBM4EVB/6rWJoPXN7y+nLgmQhr6VGpYGtxPjH9/c3MgJ8Ab4cQfpgxKxG/fUfrf7C/f171lgFo6f5zH1AIPBRC+KdoK+oZZnY0vrcOUAQ8Gvd1N7NfAKfhtzvdDHwHeBp4HKjAbxl9UQghdiceO1j30/BD8gCsAb6S0QYdG2b2GeAlYDmwv2Xy/8HbnZPw23e0/pdwEL9/3oW7iIh0Lt+aZUREJAsKdxGRGFK4i4jEkMJdRCSGFO4iIjGkcBcRiSGFu4hIDP1/p/b1iDOF/oAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = CNN().to(device)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "model, losses, accuracies = train_val_model(model, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=25, log_interval=2)\n",
    "\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ULZ91b0cPhy5"
   },
   "source": [
    "We have now completed training and validation with 3 different models: Logistic Regression, Feed-Forward Network, and Convolutional Neural Network. \n",
    "\n",
    "We have seen that with the CNN, the performance of the model in the validation set, outperforms the other models (~99% accuracy against ~90% and ~98%). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PHyGUuZbTvhr"
   },
   "source": [
    "The difference in performance between CNNs and MLP is small but how many learnable parameters are we using in the MLP and in CNN models?\n",
    "\n",
    "We can find it out using the following lines of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "acy0l3-YQjT2",
    "outputId": "ceb6251b-4ea1-4168-f23a-0c42feea37ce"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of parameters in the MLP model: 242762\n",
      "Number of parameters in the CNN model: 21840\n"
     ]
    }
   ],
   "source": [
    "#model_mlp = MLP([D_in, 256, 128, 64, D_out]).to(device)\n",
    "model_parameters_mlp = filter(lambda p: p.requires_grad, model_mlp.parameters())\n",
    "params_mlp = sum([np.prod(p.size()) for p in model_parameters_mlp])\n",
    "print('Number of parameters in the MLP model: {}'.format(params_mlp))\n",
    "\n",
    "model_parameters_cnn = filter(lambda p: p.requires_grad, model.parameters())\n",
    "params_cnn = sum([np.prod(p.size()) for p in model_parameters_cnn])\n",
    "print('Number of parameters in the CNN model: {}'.format(params_cnn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Sj28CWvrMbOw"
   },
   "source": [
    "You can see that we have ~11x more learnable parameters to achieve almost the same performance.\n",
    "\n",
    "We can experiment and try to find out the number of layers and corresponding sizes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2RmgJhIPMECw",
    "outputId": "3c677343-5c57-4a1d-a047-9bca2d8fe52f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of parameters in the MLP model: 25450\n"
     ]
    }
   ],
   "source": [
    "model_mlp_test = MLP([D_in, 32, D_out]).to(device)\n",
    "model_parameters_mlp_test = filter(lambda p: p.requires_grad, model_mlp_test.parameters())\n",
    "params_mlp_test = sum([np.prod(p.size()) for p in model_parameters_mlp_test])\n",
    "print('Number of parameters in the MLP model: {}'.format(params_mlp_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B_oq9682QWCF"
   },
   "source": [
    "And how does that model perform? We are about to find out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 979
    },
    "id": "w6oa0TeBQU9E",
    "outputId": "e8c478bf-27c6-4ade-f0bd-59302eaf0e49"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/14\n",
      "----------\n",
      "train Loss: 0.4684 Acc: 87.31%\n",
      "val Loss: 0.2689 Acc: 92.38%\n",
      "\n",
      "Epoch 5/14\n",
      "----------\n",
      "train Loss: 0.1535 Acc: 95.53%\n",
      "val Loss: 0.1545 Acc: 95.36%\n",
      "\n",
      "Epoch 10/14\n",
      "----------\n",
      "train Loss: 0.1099 Acc: 96.78%\n",
      "val Loss: 0.1270 Acc: 96.04%\n",
      "\n",
      "Training complete in 2m 14s\n",
      "Best val Acc: 96.54%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnSUlEQVR4nO3deZhU5Zn38e9NQ7MqILQsTUMjm7JJly3BBQ1qFKOCicYXjRlNZoLJ6IyJGqMZX5NBJ3GJjiZjYtBx1EThVUwiOq5RYzQRpREU2aRBlEbQVokIsvf9/vFU2dVtLwVU96k+9ftcV11dZ6u+2+V3Tj3Pc55j7o6IiMRXu6gLEBGRlqWgFxGJOQW9iEjMKehFRGJOQS8iEnPtoy6gvt69e3tpaWnUZYiItCkLFiz4wN2LGtqWc0FfWlpKRUVF1GWIiLQpZvZ2Y9vUdCMiEnMKehGRmFPQi4jEnIJeRCTmFPQiIjGnoBcRiTkFvYhIzMUm6DduhGuugQULoq5ERCS35NwNU3uroACuvhratYPDDou6GhGR3BGbK/r994ehQ+HVV6OuREQkt8Qm6AESCQW9iEh9sQv6NWvgo4+irkREJHfELugBFi2KtAwRkZwSq6AvKws/1XwjIlIrVkHfuzeUlCjoRUTSZRT0ZjbZzFaYWaWZXdHEfmeYmZtZeXK51My2mtmi5Ov2bBXeGHXIiojU1ew4ejMrAG4DvgRUAfPNbK67L623337AxcDL9T5ilbuPy065zUskYO5c2LwZunVrrd8qIpK7MrmiHw9Uuvtqd98BzAamNrDfNcD1wLYs1rfHEglwh9dei7IKEZHckUnQFwNr05arkus+Y2YJoMTd/7eB4web2UIze97MJjb0C8xsuplVmFlFdXV1prU3KDXyRs03IiLBPnfGmlk74Gbg0gY2rwcGunsZcAlwv5ntX38nd5/p7uXuXl5U1OCzbTPWrx8ceKCCXkQkJZOgXweUpC0PSK5L2Q8YDfzZzNYAE4C5Zlbu7tvd/UMAd18ArAKGZ6PwxpipQ1ZEJF0mQT8fGGZmg82sEJgGzE1tdPeP3b23u5e6eykwD5ji7hVmVpTszMXMDgKGAauz/lfUk0jAkiWwLdLeAhGR3NBs0Lv7LuAi4ElgGfCAuy8xsxlmNqWZw48BXjezRcAc4Dvu3uITFCQSsHs3vPFGS/8mEZHcl9E0xe7+GPBYvXVXN7LvF9PePwQ8tA/17ZX0Dtny8tb+7SIiuSVWd8amlJZCjx5qpxcRgZgGvVmY90ZBLyIS06CH0Hzz+uuwc2fUlYiIRCvWQb99OyxfHnUlIiLRinXQg5pvRERiG/TDhkGXLgp6EZHYBn1BAYwbp6AXEYlt0ENovlm0CGpqoq5ERCQ6sQ/6zZuhsjLqSkREohProNczZEVEYh70I0dCYaGCXkTyW6yDvrAQxoxR0ItIfot10EPt3PTuUVciIhKNvAj6jRvhnXeirkREJBqxD3p1yIpIvot90I8dG26eUtCLSL6KfdB37gyHHKKgF5H8lVHQm9lkM1thZpVmdkUT+51hZm5m5Wnrrkwet8LMTspG0XtKDwsXkXzWbNAnH+59G3AyMBI428xGNrDffsDFwMtp60YSHiY+CpgM/Cr1sPDWlEjAhg2wfn1r/2YRkehlckU/Hqh099XuvgOYDUxtYL9rgOuBbWnrpgKz3X27u78FVCY/r1WlOmQXLmzt3ywiEr1Mgr4YWJu2XJVc9xkzSwAl7v6/e3ps8vjpZlZhZhXV1dUZFb4nxo0LP9V8IyL5aJ87Y82sHXAzcOnefoa7z3T3cncvLyoq2teSPmf//cP89Ap6EclH7TPYZx1QkrY8ILkuZT9gNPBnMwPoC8w1sykZHNtqEgmYNy+K3ywiEq1MrujnA8PMbLCZFRI6V+emNrr7x+7e291L3b0UmAdMcfeK5H7TzKyjmQ0GhgGvZP2vyEAiAW+/DR99FMVvFxGJTrNB7+67gIuAJ4FlwAPuvsTMZiSv2ps6dgnwALAUeAK40N1373vZe04dsiKSrzJpusHdHwMeq7fu6kb2/WK95f8A/mMv68ua9KkQjj8+2lpERFpT7O+MTendGwYOVIesiOSfvAl60B2yIpKf8i7oV66ETz6JuhIRkdaTV0FfVhYeQPLaa1FXIiLSevIq6BOJ8FPNNyKST/Iq6Pv1gz59FPQikl/yKujN1CErIvknr4IeQtAvXQrbtjW/r4hIHORd0JeVwe7dsHhx1JWIiLSOvAt6dciKSL7Ju6AvLYUePRT0IpI/8i7o1SErIvkm74IeQtAvXgw7d0ZdiYhIy8vLoC8rg+3bYdmyqCsREWl5eRn06pAVkXySl0E/bBh07aqgF5H8kJdBX1AA48bpaVMikh8yCnozm2xmK8ys0syuaGD7d8xssZktMrMXzWxkcn2pmW1Nrl9kZrdn+w/YW4lECPqamqgrERFpWc0+StDMCoDbgC8BVcB8M5vr7kvTdrvf3W9P7j8FuBmYnNy2yt3HZbXqLCgrgy1bwvz0I0ZEXY2ISMvJ5Ip+PFDp7qvdfQcwG5iavoO7b0pb7Ap49kpsGeqQFZF8kUnQFwNr05arkuvqMLMLzWwVcAPwr2mbBpvZQjN73swmNvQLzGy6mVWYWUV1dfUelL/3Ro6EwkIFvYjEX9Y6Y939NncfAvwQuCq5ej0w0N3LgEuA+81s/waOnenu5e5eXlRUlK2SmtShA4wdqw5ZEYm/TIJ+HVCStjwgua4xs4HTAdx9u7t/mHy/AFgFDN+rSltAaioEz/mGJhGRvZdJ0M8HhpnZYDMrBKYBc9N3MLNhaYunACuT64uSnbmY2UHAMGB1NgrPhrIy2LgR3n476kpERFpOs6Nu3H2XmV0EPAkUAHe5+xIzmwFUuPtc4CIzOwHYCWwEzksefgwww8x2AjXAd9z9o5b4Q/ZGeodsaWmkpYiItJhmgx7A3R8DHqu37uq09xc3ctxDwEP7UmBLGjMm3Dz16qvw1a9GXY2ISMvIyztjUzp3DqNv1CErInGW10EPmpteROIv74O+rAw2bID166OuRESkZeR90OsOWRGJu7wP+nHjwk8FvYjEVd4H/X77wfDh6pAVkfjK+6CH0E6vK3oRiSsFPaGd/u234cMPo65ERCT7FPTUdsiq+UZE4khBT2i6ATXfiEg8KeiBXr1g0CBd0YtIPCnok9QhKyJxpaBPSiTgzTdh06bm9xURaUsU9EmpDtnXXou2DhGRbFPQJ2kqBBGJKwV9Ur9+0LevOmRFJH4U9GnUISsicaSgT5NIwNKlsHVr1JWIiGRPRkFvZpPNbIWZVZrZFQ1s/46ZLTazRWb2opmNTNt2ZfK4FWZ2UjaLz7ZEAnbvhsWLo65ERCR7mg16MysAbgNOBkYCZ6cHedL97j7G3ccBNwA3J48dCUwDRgGTgV8lPy8nqUNWROIokyv68UClu6929x3AbGBq+g7unj76vCvgyfdTgdnuvt3d3wIqk5+XkwYNgp491SErIvHSPoN9ioG1actVwBfq72RmFwKXAIXAcWnHzqt3bHEDx04HpgMMHDgwk7pbhJk6ZEUkfrLWGevut7n7EOCHwFV7eOxMdy939/KioqJslbRXEgl4/XXYuTPSMkREsiaToF8HlKQtD0iua8xs4PS9PDZyiQTs2BFG34iIxEEmQT8fGGZmg82skNC5Ojd9BzMblrZ4CrAy+X4uMM3MOprZYGAY8Mq+l91y1CErInHTbNC7+y7gIuBJYBnwgLsvMbMZZjYludtFZrbEzBYR2unPSx67BHgAWAo8AVzo7ruz/2dkz7Bh0K2bOmRFJD7M3ZvfqxWVl5d7RUVFpDUcfXT4+eKLkZYhIpIxM1vg7uUNbdOdsQ1IJGDRonDzlIhIW6egb0AiAVu2wMqVze8rIpLrFPQNUIesiMSJgr4BhxwCHTuqQ1ZE4kFB34AOHWDMGF3Ri0g8KOgbkUiEoM+xQUkiIntMQd+IRAL+/ndYsybqSkRE9o2CvhHqkBWRuFDQN2LMGCgoUIesiLR98Qr6Tz6B+fOz8lGdOsHIkbqiF5G2L15Bf/HFMGkS/PWvWfm4RAIWLFCHrIi0bfEK+p/+FIqL4eST4eWX9/njEgl4/31Yvz4LtYmIRCReQd+3Lzz7LBx4IJx0Urgc3wfqkBWROIhX0EO4on/22fDw1/PPh5qavf6oQw8NjxdUh6yItGWZPDO27Rk4EJ57LjwPsN3en8v22y/MT68rehFpy+J3RZ9SWhpS2h2uvnqvnw2YukNWRKStim/Qp2zYAHfcAccfD2++uceHJxLwzjuwenUL1CYi0goyCnozm2xmK8ys0syuaGD7JWa21MxeN7NnzGxQ2rbdZrYo+Zpb/9gW168fPPNMeIrIccfBqlV7dPjUqaEJ54QTNB2CiLRNzQa9mRUAtwEnAyOBs81sZL3dFgLl7j4WmAPckLZtq7uPS76mEIWRI0PYb9sWwn4PEnv4cPjTn2DjRjj2WF3Zi0jbk8kV/Xig0t1Xu/sOYDYwNX0Hd3/O3T9NLs4DBmS3zCwYMwaefhq2b9/j9vrx48N5YvNmOOaYvWoBEhGJTCZBXwysTVuuSq5rzD8Cj6ctdzKzCjObZ2anN3SAmU1P7lNRXV2dQUl7qawsNN18+ctheefOjA9NJMJAnu3bw5X9smUtVKOISJZltTPWzM4FyoEb01YPSj6Z/BzgFjMbUv84d5/p7uXuXl5UVJTNkj6va9fwc/bsEPzvvZfxoWPHwp//HAbyHHssLF7cMiWKiGRTJkG/DihJWx6QXFeHmZ0A/Bswxd23p9a7+7rkz9XAn4Gyfag3e4qL4a23wmicPfgWMWoUPP98eArVpEm6mUpEcl8mQT8fGGZmg82sEJgG1Bk9Y2ZlwG8IIf9+2vqeZtYx+b43cBSwdwPas23iRHj00dCUc8IJ8OGHGR86YkQI+y5dQt9ulibMFBFpEc0GvbvvAi4CngSWAQ+4+xIzm2FmqVE0NwLdgAfrDaM8BKgws9eA54Dr3D03gh7CJfncubBiBZx4Inz6afPHJA0dCn/5C/ToEc4TL73UcmWKiOwL8xybg7e8vNwrKipa95c+/niY7fLHPw6T2+yBtWvD+eK99+Cxx8IXBRGR1mZmC5L9oZ8T/ztjM3HyyfCTn4SQX7EiPMAkQyUl4cq+uBgmTw4jc0REcomCPt2WLeHy/NRTw/sM9e8f2uwHDw4jN596qgVrFBHZQwr6dF27wk03wYsvhrkPtm7N+NA+fcLV/PDhMGVKaMYREckFCvr6zj4b7r47zGn/la+EaRMyVFQUDhs1Ck4/HR5+uMWqFBHJmIK+Id/4Btx5Jzz5JFx33R4d2qtXmC4hkYAzz4Q5c1qoRhGRDCnoG/Otb8Ejj8APf7jHh/boEdrpx4+HadPg/vuzX56ISKYU9E059VTo3DnMZjZpUpgULUP77x++EBx9dPiCcM89LViniEgTFPSZWLcO3n033FT1rW+FOYsz0K1b6JQ97jj45jdDa5CISGtT0GdixAh47TW48kq491445BB46KGMDu3SJdx8e9JJ8O1vw69+1cK1iojUo6DPVKdO8NOfholt+veHX/4yTGOZgc6d4Y9/hNNOgwsvhFtuadFKRUTqUNDvqbKyMF3Cgw+GO2nffTcMx2wm9Dt2DCNwvvpV+P734cYbm9xdRCRrFPR7o0OHMGge4PbbQwP8iSeGaY+bUFgYpsGfNg0uvxyuvjo8ylZEpCUp6PfVT34SGt5ffhlGjw7tMk2kd4cO8NvfwnnnwTXXwJFHwhtvtFq1IpKHFPT7ql07+O53YcmSMATz+9+Ha69t8pD27eF//ieMr1+9Otxc9e//Djt2tFLNIpJXFPTZUlISbrCaNQsuuiisW7eu0fQ2C7MtLF0KX/ta+GJw2GF6iImIZJ+CPpvMQgN8r15QUxPmyjnsMHjllUYPKSqC++4LQzA3boQJE+Cyy/boGSgiIk1S0LeUdu1Cb+vGjXDEEXDppU1OfXzaaaH159vfDhNoph5ELiKyrzIKejObbGYrzKzSzK5oYPslZrbUzF43s2fMbFDatvPMbGXydV42i895p54a0nv6dLj55pDeb77Z6O7du4dBPM8+G5YnTYILLoCPP26lekUklpoNejMrAG4DTgZGAmeb2ch6uy0Eyt19LDAHuCF57AHAj4EvAOOBH5tZz+yV3wZ07w6//nW4PB81CgYlz4FNjLufNAlefz004dx5Zzjs0Udbp1wRiZ9MrujHA5XuvtrddwCzganpO7j7c+6ealWeBwxIvj8JeNrdP3L3jcDTwOTslN7GHHtsaIjv2DE8qvDII8Owm+3bG9y9S5dwU9VLL0HPnqFp55xzoLq6lesWkTYvk6AvBtamLVcl1zXmH4HH9+RYM5tuZhVmVlGdD0lWXR2eXvX1r0PfvqF95sUXQwduPePHw4IFYfjlnDkwcmQY2JNjz3QXkRyW1c5YMzsXKAf26AZ/d5/p7uXuXl6UuuM0zg46CCoq4IknQjv+734HEyfC4sVhe70hmYWFoV/31VfDoeecEx5XWFUVQe0i0uZkEvTrgJK05QHJdXWY2QnAvwFT3H37nhybl9q3D1Na/va38N57YTbMsWPDtgsugPJyuPXWsC1p9Gj429/CqJxnnglt9zNnNvhFQETkM5kE/XxgmJkNNrNCYBowN30HMysDfkMI+ffTNj0JnGhmPZOdsCcm10m6bt3CbGdmYXnChJDe3/seFBfDl78c2veBggK45JJw8X/YYeGccPzxUFkZXfkiktuaDXp33wVcRAjoZcAD7r7EzGaY2ZTkbjcC3YAHzWyRmc1NHvsRcA3hZDEfmJFcJ0254ILQTvPGG2H2syVLagfV19TAM88wpHQ3zzwTruhffTV8GbjpJk2SJiKfZ55jvXrl5eVeUVERdRm5paYmdN527QrPPRceWdWvX5hD4dxzWVc0ju/+s/HII3D44WE052GHRV20iLQmM1vg7uUNbdOdsW1Bu3Yh5CHcZfvgg2E4zi9/CYkExZPH8PAtbzFrVpgpubwcTjkltOeLiCjo25pOneDMM8Mjq9avD5fvpaXYwBKmTYO3f3wXD59xL4vmbeOoo8LF/7PPajimSD5T0LdlvXrBd74Tbptt3x6ALn+8nykPnUeVlfDypCvY/MYajj8+3J/16KMKfJF8pKCPm6efhj/9CTtmIuOfv5GXPziIl6b8jPXrw921iUS48UpDMkXyh4I+bszCeMvf/x7WrMF+9CMmXHY0K1fCnOtX8bV1/8m3v7aRUaPCEP5du6IuWERamoI+zkpKwtOuJk6kQwc4o/ARflR9CdWFxVzz3nR+/g+vMWJEGKLZyJQ7IhIDCvp88r3vwcKFtP+Hr3PGtt/xGuOY9cEJfOeCGoYMCTfi6oEnIvGjoM8348bBHXdg69bBzTdz+L8cwZNPtWPoUFj+vV9zZMlarrsONm2KulARyRbdMCXB6tX40KHU0I4/+lTu6XohZZdM4uLvGQccEHVxItIc3TAlzTvoIGzVKgp+cClTuj/P3C3H83+uGcXkAW9w+eWwYUPUBYrI3lLQS63Bg+H66+mwfi3cfTcDx/dlzKmDuOkmOHfQC/zkxL/x+P/WaKSOSBujphtpVmUldDiynEHVC1hPX57qNJVPTzyd8suPo/zIws8m3RSR6KjpRvbJ0KEwaOUz7Lr3fmqOnMhZu+7ju3NP5p2jz2b48PD0q1VvbI26TBFphIJeMtO9O+2/cTbFf32Azp9Us+X/PUqnyy9m4ED4zU/W029ML17seSrPfP2/+WBZHjwOUqQNUdON7LN356/j3Ut+Tv+X/0D/nW+zm3YsPeBoqn7wC47910Pp0iXqCkXiT0030qL6H15M+Qv/Sf/tb7HygYU8d+RVFHzyd759ZS/69IGbJz1C5T/MYNfCxZpVTSQCuqKXFlFTAy+8EJ57Pvrey/mXHT+nHU519yHsnvIV+lzwFeyoI6MuUyQ29vmK3swmm9kKM6s0sysa2H6Mmb1qZrvM7Mx623YnHy/42SMGJf7atYNjj4U77oALPr6Bx+98l9vH3c7CTUM54Le3suz4C7n2Wli9Gnj4YVi4EHbsiLpskVhq9orezAqAN4EvAVWEZ7+e7e5L0/YpBfYHLgPmuvuctG2b3b1bpgXpij7eNm6Eh+/9mOd+W8W9C0ZRwC4+sf3p7Fup6VCIjTsUO+wwOOMMOOGEqMsVaTOauqJvn8Hx44FKd1+d/LDZwFTgs6B39zXJbZrlXJrUsyecf3F3zr+4OzPehjkPFvBPD76Oz6+gbOcCjli0gMRrs1jz9/4MnHAC3XZ8BCedFB6Cm3qNHg2FhVH/KSJtRiZNN8XA2rTlquS6THUyswozm2dmpze0g5lNT+5TUV2toXn5YtAguPQy476Xh/JfH0yj+L4bue2MZynu9BGJ2T+gVy/41ukfUbVpP2pmzYbp00PQ77cfPPRQ+JAPP1Szj0gzMrmi31eD3H2dmR0EPGtmi919VfoO7j4TmAmh6aYVapIcc8ABcM454bVzZztefLETjzwCjzwylJLKZzFqOOXg1Zx7yAKO6bqAPiNHh6uUxx+Hb3wjXOGPHVt71f+1r0GPHhH/VSK5IZM2+iOAn7j7ScnlKwHc/WcN7Hs38Gh6G/2ebAe10Utd7rBiBcnQh7/+NYzo6dMHTjkFzpz4HpPaPU+nNypgwYLw+vhjWLcO+veHX/wiPEpr6NC6rwkToKAg6j9PJGv2tY1+PjDMzAYD64BpwDkZ/uKewKfuvt3MegNHATdkVrZIeDLiwQeH1w9+EFpqHn88hP6cOXDXXX3o2PEsjjvuLE47E0672xmwY3UIeQhfFXr2hFdegQceCGeJDh1qn7Dys5+Fk8OQIXVPBCUl0f3RIlmW0Th6M/sycAtQANzl7v9hZjOACnefa2aHA38AegLbgA3uPsrMjgR+A9QQ+gNucff/bup36YpeMrVzZxirn7raX5VsEBw3Dk4+GY45Bo48EvbfP3nAjh3w9tvhav+LXwzr/u//DWeM1atr2/kHDYI1a8L7a64JQ4VSJ4IDD4RevWDgwLDdHc3qJrmgqSt63TAlseAOy5fXhv5LL8Hu3WE8/6GHwsSJta8+fRr4gN27oaoqTNX56adw2mlh/emnw1NPwda0Sdu++EV47rnw/uCDw4mjW7fa14knwvXXh+2XXBKewL7ffrXbR4+GSZPC9pdegq5dYcQI6Nixhf7pSD5Q0Eve2bwZ5s0LV/wvvBDep7J62LC6wX/QQc1clLuHJ69UVoa2o+7da4P6ppvg3XfDL9y8GT75BA4/PHxTABg1Kmz/5JNwMgH45jfhrrvC53boENYXFkIiEfoOzjwTjjqqxf7ZSDwp6CXv7dgBr75aG/wvvhhaZAD69asb/KNHt0A/rXsoYvPmcFY54ICw7k9/go8+Cv0E8+bB/PkwY0bokHj/ffjud0P4H3FEGE3UuXOWC5O4UNCL1FNTA0uX1gb/Cy+ElhsIF+xHHVUb/OXlrdiqsnNnOCF07RrOTGedVdv50L596ID45S9D+O/aFc5I6iMQ9n3UjUjstGsXrtxHjw4Xze6hnzY9+B97LOzbsSN84Qsh9I84IrTMHHhgCxXWoUN4QWjKqawMV/Yvvxza8+fNC6OIAO65B370o1DUhAnhdfjh4SQhkkZX9CKNqK4OTTyp4F+4sLaZvbQ0ZOr48eGVSIR+1lb1/POhrX/ePHjzzbCuoCAU3rNnuOlgw4Yw7Kh799qf/fq1cqHSGtR0I5IFmzeH1pRXXgmv+fNrR2G2awcjR9aG/+GHw5gxrTglz4cfhqv+pUvhssvCurPOggcfrLtf//5hlBDA2WfD3/4WTgCp1/DhcOutYfu994b+g/QTRd++4Q+TnKOgF2kh778PFRV1w/+DD8K2jh2hrKzulf/QoeGk0Co2bAgFbtoU7hbetCm050+bFrb/4hfhzLVpU+1rwAD4/e/D9vLy0Emc7qijwtccCDcrbNkSvt4MHhx+pqahkFanoBdpJe7hKn/+/NrwX7Cg9kbc7t1D8KeHf+om3pyza1cYFpp+IigsDMUDXHghLF4c/uCqqvDHT5sGs2aF7WVlYb6h0tLak0F5efjqI1mnoBeJ0K5dsGxZ7RX/K6+EfNy1K2zv3z8Mt09N9TBiRPjZv38bGlCzY0dt2A8ZEv64888PJ4G33gr3EkAYNnrDDeGbwNixtSeB0tJwx/HEiaFpaOvWcIbs1i10LnftWvu+NeYo+uQTWL8+1Jm6R2LLlvAtpmtXWLky3E09YkS4S7rVvqY1TkEvkmO2boVFi0LwV1SEE8Hy5SFPUrp1qxv8qdfQodCpU2Sl753t2+Gdd0LhJSWhw/jii2tPBBs2hP1uuSWsX7IkDImq74474J/+KfSMf+UrdU8AXbvClVeGeS9WroS77w73HWzbFkJ6y5Zwp/LBB8PTT8NVV9UGeCrM//a30LN+++1hOFZ9y5eHfyHXXlt7U1znzuEuvIMPhpkzw9e2Dz4If2sr9tBreKVIjuncOYyKPOKI2nXu4SJy+fLa14oVYcTPfffV7mcWWkEaOgkUFeXot4COHUMYphQVwf331y5v2xb6Ebp0CcsDB8KTT9YGdCqQJ0wI27t2DVNRpF9tV1WFz4Fw78H119fOg5E6IZx7bvgH1bFjuGmtpKT2RNGtG/TuHY4//vjwwOP0bV27hm8eAP/8z2EypdS/pOXLw9e0VLBfdRX85jehzyP1L2nkyHBcBHRFL9IGbNkSLlLrnwRWrKg7DU+PHnWbgFKvIUPycCod93ADWocOrX/2+8tfQqd16iSwfHlomlq9Omw/66zwLzT9bJ26sWMvqelGJKZqamDt2rp5knqfahaHcFFbWlo3/IcPDz/bVF9AW+UevrGkHoZz3XXhZLB8eWi+cg/9E3/5y17/CgW9SB7atCncR/Xmm7VX/ytWhOXUKCAIrQ3Dh9cGf+o1bFiYdFNa2Nat4Q7o7dvDqKS9pKAXkc+4h3um0oM/9T51cZnSv3/dq//UCWDgwDxsCspx6owVkc+YhT7CAQNCn2O6bdtCP2b6N4AVK8LDuVKzfaY+o3//ukPk09+XlNRO2SPRU9CLyGc6dQpj+keN+vy2Dz4Iob9qVe2oyDVrQp/jrFmhvyClXbtwImnsRFBcHCbjlNahf9QikpHevcOroWei7NwZmoNS4Z/6uWYNPPts2JbeJFRQEK76008AAwfWftMoLlb/QDZlFPRmNhm4lfDM2Dvd/bp6248hPFN2LDDN3eekbTsPuCq5eK2735OFukUkh3ToUBvYDdmxI9wvlQr/9BPCE0+E+wfq23//2uBPPwGkL/fsqRFDmWg26M2sALgN+BJQBcw3s7nuvjRtt3eA84HL6h17APBjoBxwYEHy2I2ISN4oLAx39A4d2vD2bdvCVX9VVe0rfXnx4nDzbP2xI506NX8yKCpqnVkTclkmV/TjgUp3Xw1gZrOBqcBnQe/ua5LbauodexLwtLt/lNz+NDAZmLXPlYtIbHTqFG7qGjKk8X127gxh39jJ4IUXwr0DO3fWPa6gIEzB379/7au4+PPLPXrE99tBJkFfDKxNW64CvpDh5zd0bHH9ncxsOjAdYODAgRl+tIjkkw4dQrt+SUnj+9TUhGl00k8G775b+1q5MjyvZWMDbQqdOjV+EkhfbosP8MqJzlh3nwnMhDCOPuJyRKSNatcO+vQJr6amxd+6te4J4N13w7eD1PsFC2Du3LrTS6SkHtLVt2/dV58+dZdzqckok6BfB6SfQwck12ViHfDFesf+OcNjRURaROfOzTcVuYe7i9NPAOknhPfeC7OPvvde3VlHU8xC2Dd2IkhfPuCAlm02yiTo5wPDzGwwIbinAedk+PlPAj81s+TTjDkRuHKPqxQRaWVm4eq9e/fmn5WyeXMI/A0ban/Wf79iRfi5ffvnj+/QITxw/uijYfbs7P8tzQa9u+8ys4sIoV0A3OXuS8xsBlDh7nPN7HDgD0BP4DQz+3d3H+XuH5nZNYSTBcCMVMesiEhcdOsWXk19Q4Dauc0aOyH07dsy9WmuGxGRGGhqrpvon38lIiItSkEvIhJzCnoRkZhT0IuIxJyCXkQk5hT0IiIxp6AXEYk5Bb2ISMzl3A1TZlYNvL0PH9Eb+CBL5bS0tlQrtK1621Kt0LbqbUu1Qtuqd19qHeTuRQ1tyLmg31dmVtHY3WG5pi3VCm2r3rZUK7StettSrdC26m2pWtV0IyIScwp6EZGYi2PQz4y6gD3QlmqFtlVvW6oV2la9balWaFv1tkitsWujFxGRuuJ4RS8iImkU9CIiMReboDezyWa2wswqzeyKqOtpipmVmNlzZrbUzJaY2cVR19QcMysws4Vm9mjUtTTHzHqY2RwzW25my8zsiKhraoyZfT/538AbZjbLzDpFXVM6M7vLzN43szfS1h1gZk+b2crkz55NfUZraaTWG5P/HbxuZn8wsx4RllhHQ/WmbbvUzNzMemfjd8Ui6M2sALgNOBkYCZxtZs085TFSu4BL3X0kMAG4MMfrBbgYWBZ1ERm6FXjC3Q8GDiVH6zazYuBfgXJ3H014VOe0aKv6nLuByfXWXQE84+7DgGeSy7ngbj5f69PAaHcfC7xJbj2z+m4+Xy9mVkJ4vvY72fpFsQh6YDxQ6e6r3X0HMBuYGnFNjXL39e7+avL9J4QgKo62qsaZ2QDgFODOqGtpjpl1B44B/hvA3Xe4+98jLapp7YHOZtYe6AK8G3E9dbj7X4D6z3meCtyTfH8PcHpr1tSYhmp196fcfVdycR4woNULa0Qj/2wB/hO4HMjaSJm4BH0xsDZtuYocDs50ZlYKlAEvR1xKU24h/IdXE3EdmRgMVAP/k2xqutPMukZdVEPcfR3wc8KV23rgY3d/KtqqMtLH3dcn328A+kRZzB74FvB41EU0xcymAuvc/bVsfm5cgr5NMrNuwEPA99x9U9T1NMTMTgXed/cFUdeSofZAAvi1u5cBW8idpoU6km3bUwknp/5AVzM7N9qq9oyH8dk5P0bbzP6N0GR6X9S1NMbMugA/Aq7O9mfHJejXASVpywOS63KWmXUghPx97v77qOtpwlHAFDNbQ2gSO87MfhdtSU2qAqrcPfUNaQ4h+HPRCcBb7l7t7juB3wNHRlxTJt4zs34AyZ/vR1xPk8zsfOBU4Oue2zcODSGc9F9L/v82AHjVzPru6wfHJejnA8PMbLCZFRI6tOZGXFOjzMwIbcjL3P3mqOtpirtf6e4D3L2U8M/1WXfP2atOd98ArDWzEclVxwNLIyypKe8AE8ysS/K/iePJ0Y7jeuYC5yXfnwc8HGEtTTKzyYRmxynu/mnU9TTF3Re7+4HuXpr8/60KSCT/m94nsQj6ZGfLRcCThP9RHnD3JdFW1aSjgG8Qro4XJV9fjrqoGPkX4D4zex0YB/w02nIalvzWMQd4FVhM+P8xp27XN7NZwEvACDOrMrN/BK4DvmRmKwnfSq6LssaURmr9L2A/4Onk/2e3R1pkmkbqbZnfldvfZEREZF/F4opeREQap6AXEYk5Bb2ISMwp6EVEYk5BLyIScwp6EZGYU9CLiMTc/wcImtX4aKutOAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "optimizer = optim.SGD(model_mlp_test.parameters(), lr=0.01, momentum=0.5)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "model_mlp_test, losses, accuracies = train_val_model(model_mlp_test, criterion, \n",
    "                                                     optimizer, dataloaders,\n",
    "                                                     num_epochs=15, \n",
    "                                                     log_interval=5)\n",
    "\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rpmgachOUCnX"
   },
   "source": [
    "We can see a drop in performance compared with the previous MLP model. So we can understand that although we have less learnable parameters, due to properties of CNNs (e.g., invariance and parameter sharing), which allow them to have fewer weights as some parameters are shared.\n",
    "\n",
    "CNNs are expected to be invariant to the location where important features occur in the input. In fact, it's not unusual that there is a dataset shift where the data acquisition process suffers some modification. We will do this by applying a transformation with horizontal translations to our validation dataset and see how robust each model is to these shifts.\n",
    "\n",
    "We can do this by going back to **0.1 - Create Dataloaders -\n",
    "MNIST dataset** cell to define the test transform using the following code \n",
    "\n",
    "```\n",
    "mnist_transform_test = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "     transforms.RandomAffine(0, translate=[0.1, 0]),\n",
    "     transforms.Normalize((0.1307,), (0.3081,))])\n",
    "```\n",
    "\n",
    "and replace\n",
    "\n",
    "`mnist_val_dataset = datasets.MNIST('../data', download=True, train=False, transform=mnist_transform)`\n",
    "\n",
    "with\n",
    "\n",
    "`mnist_val_dataset = datasets.MNIST('../data', download=True, train=False, transform=mnist_transform_test)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-5gcf_gMlcqI"
   },
   "source": [
    "After rerunning the different models we can see that the accuracy of the Logistic Regression drops from ~90% to ~72%, the MLP drops from ~98% to ~87%, and the CNN drops from ~99% to ~97%. This shows that the learned features are more robust to variances in location, as expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nU3NwQ7Nuvhv"
   },
   "source": [
    "# Bonus Case - Attention with small images and CNNs.  (And how to create a dataset that takes numpy arrays)\n",
    "\n",
    "In this case we will use the Scikit-Learn's digits dataset\n",
    "\n",
    "## Scikit-Learn Digits\n",
    "\n",
    "This dataset is provided by scikit-learn and the digit images are returned as numpy ndarray. We will use PIL (Python Image Library) to convert the numpy ndarray to a image, tranform it to a tensor and normalize it.\n",
    "\n",
    "In this case we don't have a predefined Digits Dataset provided by torchvision so we will need to write a custom Dataset class and implement three functions: \n",
    "\n",
    "`__init__`, `__len__`, and `__getitem__`.\n",
    "\n",
    "Scikit-Learn return the digits images and labels as ndarrays. Each digit image is an 8x8 array.\n",
    "\n",
    "To use the previous CNN, we will use a transform to resize the images to the MNIST image size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "A4v-XFzcv9If"
   },
   "outputs": [],
   "source": [
    "SKLEARN_DIGITS_TRAIN_SIZE = 1247\n",
    "SKLEARN_DIGITS_VAL_SIZE = 550\n",
    "\n",
    "class NumpyDataset(Dataset):\n",
    "\n",
    "  def __init__(self, data, targets, transform=None):\n",
    "    self.data = torch.from_numpy(data).float()\n",
    "    self.targets = torch.from_numpy(targets).long()\n",
    "    self.transform = transform\n",
    "\n",
    "  def __getitem__(self, index):\n",
    "    x = np.expand_dims(self.data[index], axis=2)\n",
    "    y = self.targets[index]\n",
    "    if self.transform:\n",
    "        x = self.transform(x)\n",
    "    return x, y\n",
    "\n",
    "  def __len__(self):\n",
    "    return len(self.data) \n",
    "\n",
    "digits_transform = transforms.Compose([\n",
    "                                       transforms.ToPILImage(),\n",
    "                                       transforms.Resize(28),\n",
    "                                       transforms.ToTensor(),\n",
    "                                       ])\n",
    "\n",
    "# Get sklearn digits dataset\n",
    "X, y = load_digits(return_X_y=True)\n",
    "X = X.reshape((len(X), 8, 8))\n",
    "y_train = y[:-SKLEARN_DIGITS_VAL_SIZE]\n",
    "y_val = y[-SKLEARN_DIGITS_VAL_SIZE:]\n",
    "X_train = X[:-SKLEARN_DIGITS_VAL_SIZE]\n",
    "X_val = X[-SKLEARN_DIGITS_VAL_SIZE:]\n",
    "\n",
    "digits_train_dataset = NumpyDataset(X_train, y_train, transform=digits_transform)\n",
    "digits_val_dataset = NumpyDataset(X_val, y_val, transform=digits_transform)\n",
    "digits_train_dataloader = torch.utils.data.DataLoader(digits_train_dataset, batch_size=64, shuffle=True)\n",
    "digits_val_dataloader = torch.utils.data.DataLoader(digits_val_dataset, batch_size=64, shuffle=True)\n",
    "\n",
    "dataloaders = dict(train=digits_train_dataloader, val=digits_val_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dhQU3v7Zv9Ih",
    "outputId": "030f1fa0-62a0-4dc0-d61f-013e09e2457d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Datasets shapes (before transformations): {'train': torch.Size([1247, 8, 8]), 'val': torch.Size([550, 8, 8])}\n",
      "N input features: 784 Output classes: 10\n",
      "Train batch: torch.Size([64, 1, 28, 28]) torch.Size([64])\n",
      "Val batch: torch.Size([64, 1, 28, 28]) torch.Size([64])\n"
     ]
    }
   ],
   "source": [
    "# Get some examples of images and targets\n",
    "_, (example_train_imgs, example_train_targets) = next(enumerate(digits_train_dataloader))\n",
    "_, (example_val_imgs, example_val_targets) = next(enumerate(digits_val_dataloader))\n",
    "\n",
    "# Info about the dataset\n",
    "D_in = np.prod(example_imgs.shape[1:])\n",
    "D_out = len(digits_train_dataloader.dataset.targets.unique())\n",
    "\n",
    "# Output information\n",
    "print(\"Datasets shapes (before transformations):\", {x: dataloaders[x].dataset.data.shape for x in ['train', 'val']})\n",
    "print(\"N input features:\", D_in, \"Output classes:\", D_out)\n",
    "print(\"Train batch:\", example_train_imgs.shape, example_train_targets.shape)\n",
    "print(\"Val batch:\", example_val_imgs.shape, example_val_targets.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 284
    },
    "id": "Vx78pb7Ov9Ih",
    "outputId": "207e26b2-c2f7-41e7-ae13-9421b398027e"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAELCAYAAADgPECFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAu8klEQVR4nO2de7BdV33fvz+/JNnyQ7IkS/fqLfkB5mF7mEDGJrg0CRMSCiGhaZsONiEJaTMwGWihBUMhMRlICdApIRDi8AiBPBqS0HagLW2Z1oBNCCGmwWDLkqzn1cuSbEvye/ePvXX0Xd97z9rn3HvWOefqfj8zGu111n6ss9fv7HX39/dbvxVVVcEYY4wpxTmjboAxxpizGw80xhhjiuKBxhhjTFE80BhjjCmKBxpjjDFF8UBjjDGmKB5o+iQivhoRvzjqdpj5R0R8KiJuH3U7zPxjvttOdqCJiEfp3zMRcYrKPz+sRkbEzc31uT23ZPavIuJEs9/eiPhgRJw7rPZSO97dtOUf02fnNZ9tHHZ7hskY2c7bpS2nmvas6LL/TmrrgeYHvnRY7aV23NrYyVvl8z0RcfOw2zNMxsh21kTEFyNiXy+/WdtOd7IDTVVVS0//A7ALwCvosz86vV9EnFe6oQD2cXuqqvp0y/7Pb9r9DwH8MwC/pDsMqd0PAXjPKAa6UTIutlNV1W9KW94P4KtVVR3OHPaKZt8bALwAwG26wxBt560RcfEQrjU2jIvtAHgGwJcB/Ewfx9h2ZmBW0lnzhrEnIt4WEVMAPtmMonfKflVEbG22F0XEByJiVzPafywilgzgO2Spqur7AP4vgOdExMamTa+PiF0A/lfTtl+IiHsj4mhE/LeI2EDf4cci4vsRcTwiPgIg+mzClwE8AeCfz1QZEZdGxGci4lBEPBgRt0XEOU3drRFxZ3PfjkbEjoj4CTn2jojY37y53T7uA9oobSciAsBrAbT9kQIAqKpqL4AvAXgOtelXI+J+APc3n/1URHwnIo5FxNcj4nl0vesj4tsR8UhE/AmAxX02+V4A3wDw5i7fZ1FEfLj5i3tfs72oqTt9n98SEQcbG3mdHDv03+NcGLbtVFV1oKqqjwL4637battJmYuPZjWA5QA2APjlHvZ/H4CrAFwHYCuASQDvOl3Z3OybMsevar7Ujoj4UERc1EsjI+LZAF4M4G/p45cAeBaAl0XEKwG8HcCrAaxEPSh9vjl2BYAvoP6rZAWABwDcSOde37R7faYJFYB3Avh3EXH+DPX/EcClADY37XotgNdR/QsB/KC5/m8BuKN5YALApwA8hfp+Xg/gxwHMB//RsG3nNC8GsArAn/fSyIhYB+DlSG3nVaj75NkRcT2APwDwBgCXA/g4gC82P8QLAPwlgD9E/V3/DPKXcY/tfieAX4uI5TPUvQPAi1Dfl+cD+CGkf0GvRm1bkwBeD+B3ImJZU5e9p2PMqGynL2w7QlVVPf0DsBPAjzbbN6P+K30x1d8K4E45pmoaEgBOANhCdT8MYEeP114N4NmoB8ZNAP4PgI9n9q8APAzgKOrB4fbm2I1N3Wba90sAXk/lcwCcRG3IrwVwF9UFgD0AfrHHdr8bwGeb7bsB/AsA5zVt2Ajg3OY+PpuOeQNqaef0Pd1GdRc2x64GcAWAxwEsofp/CuB/99qnw/o3StuRc94B4FM9tPVRAMcAPAjgo6fvcdOml9K+vwvgN+T4H6D+g+FHAOwDEFT3dQC399jWzj0B8KcA3t9s7wFwc7P9AICX0zEvA7CT7vMpAOdR/UHUD5eB3dOFYDv8m7XtzO6ezkUrPFRV1WM97rsS9UPyb878MY5A/aBtpaqqKQBTTXFH1E6u/4L6odyNG6qq2sYf0LV308cbAPyHiPht3hX1KD3B+1ZVVUUEH9sPtwH4JOq/Uk6zAsD5qI3yNA821z7N6e+NqqpONt9hKeq/dM4HsJ++1zlIv9u4MjTb6RwQcSGA1wB4ZQ+7v6qqqq90qVPbuSUi3kifXYDabioAe6vml9jA/dwP7wLwzYj4oHw+gem2M0HlI1VVPUXlk6htZyD3dEQM3Xb6xLYzA3MZaDTt84mmAfWVI1ZT3WHUI+S1Va1dzpUKc5P9uO27Aby3IifjaSLiSgDrqBxc7uuCVfU/ImIbgH9JHx8G8CRqo/te89l6AL3co92o32hWiEHMB0ZhOz+N2kH61TmcA5jZdt6rO0XESwBMRkTQA2M96r8k+7tgVX0/Ir6AWu5g9qG2nb+n8+/r4ZSD/j0Ok1E+d+bKgrWdQc6j+TsA10bEdRGxGLVsBACoquoZAJ8A8KGIWAUAETEZES/r5cQR8Q8iYkPUrEOtEf7VgNr9MQD/NiKuba51aUS8pqn7r813enXUkSJvQi1bzZZ3AOiEHFZV9TTqV9v3RsTFUQchvBnAZ9tOVFXVfgD/HcBvR8QlEXFORGxpjHS+Ucx2iFsAfEb+SpwrnwDwKxHxwsY2L4qIn4w60ucbqP1nb4qI8yPi1ah18NnyHtS+u8vos88DuC0iVjb+xHehN9sZ1D0dB4raTnPORU1xUVMeBAvKdgY20FRVdR+AXwfwFdRRFXfKLm8DsA3AXRHxcLPf1acro449f3GX01+PWqM80fz/XdQP/UG0+y9Qh7z+cdOu/wfgJ5q6w6jllvcBOALgSgBfozavb9qdCwbga30NwDfl4zei/l7bUd+zz6F2EvbCa1G/bn8PtT/qPwFY0+OxY0Nh20FETAJ4KYDPDLjd30IdNv8R1Pd/G2p9HFVVPYE6wORW1G9SP4c6sITblW23XGsHatmVg2BuB/AtAPeg/k18u/msF7L3dL5Q2nZQ//X+aLP9/aY8iHYvKNuJwf6BZ4wxxqQ4BY0xxpiieKAxxhhTFA80xhhjiuKBxhhjTFE80BhjjClK3xM2I6LnMDWaOTqtfO656UTSRYsWzbgNAIsXL+667wUXXJDUafn882dKL9Y/Gp3H5aeeSudLnjhxYsbtmcpPPvlkZ/vpp59O6p555plsWdrTb7LPodKP3ZSCbaEfG1MbytnYOef0/reb9nfObh599NGkzMe22UlLZOnhqqpW9tDckTEOttMPbANLl6arBGj54ovPJFdWO1Nb0mcmc+rUmajrkydPdq0DgCeeeGLGbSB9HgHTn21Cz7Yz0HTVeiPOOy89Pd9IvsEAsGXLls72lVdemdRpefPmzZ3tjRs3JnVavuKKK/KN7oI+BPSGc4ccPXo0qbv77rtn3AaAu+66KylPTXUyzEx7mDz88MNJWQ3I9MeqVas622pTW7duTcpsj5OTk0nd2rVrkzLX6wCW45FHHknKbCtqJ1o+cuRIZ1sfJGon+jARZpvaxHSBB5Mbb7wxqbvpppu6ltesSafAXXLJJUmZn5n6x8R3v/vdGbcB4J577knKO3fu7Gzv3ZtO7t+zZ09SPnw4t5pG77Zj6cwYY0xRPNAYY4wpigcaY4wxRfFAY4wxpigDDQbQqImLLkoXwWQn2bJly5I6doStXp0mSNbypZde2vWa6vhkh7o69HNlrcsFB6jT/vjx453txx7rdekMU5pcpKBG2zz++OOdbXW2a0QYO/XVTtQ+uax17OxduTIN5lm3Ll2dggNtNBhF29ASDLBg4eAlDWTKRcXmIhQB4PLLL+9sa7+tWLEiKfMzUc+bi0LLRRZq/+fKfUYozhq/0RhjjCmKBxpjjDFF8UBjjDGmKAP10ahWqX4Y1id1chJPtNy0aVPXOiD10egsbfWJsPauE9l0X9biVcNXLZN1TtXsDx061NnWSZgtM21NQbgPtR/YJwOktqJ9eOzYsaTM2rrq4RdeeGFSzmUnWL58eWdbtX21Vb6mToxWn5JODF2oaKaSXF9omSdPqp9Fn3PsU9aJwRMTE0n5sssu62yrT1ufbew3yj2f2iab20djjDHmrMMDjTHGmKJ4oDHGGFOUoj4a1h+B1C+TS4bZliiTNWnVrlVrZ72a57cA0+e/cFnnVeSyN2sbDh482NlW/00pDdS0w3p0bt4MkPpo1MfRljGcUf8J93/Op6mJO3NZodUnwz5Cc4acj2bJkiVJnfpLODmv+s+0r7h+w4YN2X25z7UNbXN7GLYr9bvkMjKr/8Y+GmOMMfMSDzTGGGOKMlDpTGUCDe1kKY3XBgHSV1NdQ0ZT0PCroL76qTzG63bo2goPPfRQ17Km7chJZ7ovSxcqz6lEk3vltcw2WDisU/tBJU62I5WttF/4vFqn8lgufRKnI9EQWpV9uH379u1L6ga12N98ILe4IpD2XS7lj8r8WmY5TKdf8NpFQCqXaSohLXOfz6Xf+glv5udn7rk2SPxGY4wxpigeaIwxxhTFA40xxpiiDNRHoz4G1QbZl6EhmVynGqPSj9bOPhJNHcL+GyANS861D0i/m7aB9fO2kGoOo9XztN0H0x/ch9oP6l/kfTUNP+vqWtZQaPUZ8L7qw2TtPBeSCuTTiCwk1K+hqfa5zD4ZAJicnOxsa2oYrgNSP7H6jNWnzH4YTl0zU/tyYev9kFsmoJ90NfbRGGOMmZd4oDHGGFOUgUpn+tqlchNLQzqbnvdtm606W+lMJRCdQX3gwIHOtmbsVSmN25+r0++Z+94ql1g6Gyx8r1Xi0nvN/d+WCYBDmNXmVWZj2UVDmHMztnNS2kIOi29b1ZfDyfV+c1jyVVddldRp1mVeNVMlOC3nVs3UskqrvaJ93E948yhsx280xhhjiuKBxhhjTFE80BhjjCnKUMOb2Z/ST3hzbnXLflKJqI9GU9Cwz0ZDoVXT57Ku3JlLK5PTRIeVDmKhkvMDqk+Ow077SXOidRoKm8sQzm3qJ7x5WCGq44iGpauPhlct1RDmrVu3draf97znJXXXXXddUuYwZQ1J1qzKXK/75mxnLuR8NDnfr300xhhjzgo80BhjjCmKBxpjjDFFGaiPRvXH3Apxqq1yneqWWubY+dxSBEA6b0Xbp/Mh+FycjgZI59gA+WUCWE/PzQEyNW26NZfVprgP2+a78LFtqxdyuc1Hw/AqskDqIwBSH0IuLbzaifoiuZxLMXK204/t5Pqtza/BPuU2XypfR5eJyJVnO6dGafO7jMKH5zcaY4wxRfFAY4wxpigDlc5yEheQpl/QVAwsc6iMofC+KpUpS5Ys6WxrCoq1a9cmZQ493b17d1K3ffv2pMzSn76K8mu2hnG3hTsvRNrshu+1Sg+c/oPTjWgdkNpCm6TBbciFr2p548aNSZ2WOZWJ/gb4OirlaOoiS2c1uVQsQHpvVOLmKQqayV1XLeW+yk23ANJ+bFu5M2dnZxN+ozHGGFMUDzTGGGOK4oHGGGNMUYqGN2sIM+vgrJcDeR+NnpfPo5qnhjuzJq4abW4ZA14lD5j+XVgz1xQ0/WjtZrrPI2c3mmKE/W7aZ6tWrUrKnEZE7UTPy/aZC8XXer2m+gE53Fn9Qnxe9QPklpvQUNyFlIJGv2suJf5cfDTsS2lLw8/2oH2jtsR+RPtojDHGmFnigcYYY0xRPNAYY4wpSlEfTa5e92Wdsy31v84/YHQOBvt+2tJ759Lg8BwbADh8+HBnW/VdRr+LpqN3Sprp/aC+C/afqE+O/TLr1q1L6tQ/smzZss52bvldvabq6rmyzuXha2r79Tz8m2hLQcPa/0KeR9M2p4X9MupL5aVApqamkrrcs6Jtme3cM0d9gexDztnDTOX5hN9ojDHGFMUDjTHGmKIMVDrTV0iViXhFSw1vzoV2ahoXfsVsW9WTX105vBWYLnNwWc+j7WXJZsOGDeiGvq7rqp5mepZllbVyIcyc4mX9+vVJnfYL96/Kr9q/XN+WgobrNWxaz8vSbi6rbls2YbbPhbzCpv5O9VmRe65wWeV5lcNzUxa0zH2utqL2wdKq9pu6AXLZvscdv9EYY4wpigcaY4wxRfFAY4wxpihFfTQ5/4Tq3qyXqs7KYYht19Q0E3ydiYmJpG5ycrJrG1R3VU1/9erVXdvA7T969GhSp5qtme6jUd9ZLoSZ/TKbN29O6jZt2pSUOYS5Tf/OrcaZW9VRz6Oh2rn0JDn7y6U9GYcVFEdFm4+G75uGiPO+6pNRXwr3edvSBBzCnEuJBaQrsmp4s/r35vOzY/623BhjzLzAA40xxpiieKAxxhhTlKHOo2ENMqetHj9+PKk7ePBgUs75czSdOuvrV111VdfzAKkmqjq8aq2cDl73ZV/U3r17u7bH1LTNo+F7rWlleK6M+mS2bt2alLkP29J79JPuI5daKVdWfyLbY1uaE/Y9eB7NGXLLK+jziH0eOb+b1rfdX56vx8tCANP9xJzaqs0no7+T+YTfaIwxxhTFA40xxpiiDFQ6a1tNkle001dclgb0FVfDm/lYvYaGMHK4qUoyGkbLmXVVKtPvxmGrel5+BdZw1/mcgbUUbWHx3P8qo3I/qKShtqGhxjlY4tSMu7my9ncu03Pue+eyhQNp2Lz+XvS8C4lc5uphZUrnPs6tjKrlnDw6U3k+4TcaY4wxRfFAY4wxpigeaIwxxhSlqI9GwzcZDUtkrVK1dNWrWYfNpU8HUs1cde5Dhw4lZQ5FVP+NhhZyGzUskfd1OHM7qlvnllLQdO7chzt37kzqNLS0nzTr3IcaksppQ7Sc89cBqe9Pfx/sw9SUKPv27UvK7KtSH03ud2eGS9sKoPy80mfXfPbJKH6jMcYYUxQPNMYYY4rigcYYY0xRivpodE4La5Bax+kW1K+R83PklmcF0vT+6qPhZVSBdB6NzsngOiDV2nUJgZyPxvNopqM+GvVPsF9G/WrsK9N+0HI//jL2rVxzzTVJnZbZ5nSpafX1cf+rJs++FrXVnI9G5x3ZRzNa+NnRj49G63JzguYbfqMxxhhTFA80xhhjijJQ6axNxsqlxphtJtW2bLksl6ikoJINZ4JWaS8XaqiSDLd3Pq+KNyzULrTMYb9zoR95ljPwari9yqgc/qzhzDnb0O/JqXY0xFslQ05BoxKcls1gaZO/uc/19z+XZ1mvbWqz7dwzqZS076egMcaYonigMcYYUxQPNMYYY4oyUB9NP6hOyKGobWGqHD6sadg1VczSpUs721u2bEnqtMwrNGoqEdXpWQdXXw+Hm6oOv5BWPxw32H/CPhhgehgyr+q5efPmpG79+vVJefXq1TNeA5ju65uamups6/IX+/fv71qX8xmqL9Q2Nnj68e+xDbQtTcJpktQm1ZZyKxRzeiU9TtvAy1po+qJSKbP8RmOMMaYoHmiMMcYUZWTSmYbRsTzWJmvwq59KWroyJp8rJ5UBwMaNGzvbKsmpdMGh0ppdOrdqnmWN0cE2pjP4Jycnk/K6des621u3bu1aB6TSmfa3zvDnss7253KbdMbSiaWz8rB0plnAVa7nZxBL98D0ZxlnJ9G63Gqt/Uhn2gZunz4/LZ0ZY4yZl3igMcYYUxQPNMYYY4oyluHNqlXmMuJq6J6WWQPN+WS0rOlqVGvnEGYNEbSPZjxh7VptSm3hyiuv7Gz3E96smadz4c3bt29P6voJb3aameHCzyv136qfg+2szUeTC2/OpeLS52fOR6Pn5TbZR2OMMeaswAONMcaYonigMcYYU5Sx8dGwrqgrX65ZsyYpX3755Z1tTdmei1Pn4/SaQDofQf0u6qPZs2dPZ3vXrl1JHa9+qOc5m1bNm2+wtq7zrdSO2FZ43paeB0j7VFe75HT+QOqHYRsCUv+O2k1umQozWtTvymX1pamvjf25vEwJkF9iIDevT/3CuWUkhjUHy280xhhjiuKBxhhjTFHmpXR2xRVXdLZVDuNwQSCV0lTy0DYcP368s33gwIGkTuUxDk3duXNnUsfyCJ8TsAQySrj/21IXcQio2o1KEbwCqK6MyTIqkIY3awoals5UgrPkOlpYUtK+UHtg6UrlMF0tVu2FyaWgUemMr6PXVFtimU3bXsrO/EZjjDGmKB5ojDHGFMUDjTHGmKKMjY+GQ0jVzzIxMZGUOaW7phJRnw2nXzhx4kRSp2VO968+mt27dyflbdu2dbY1lQjrsLqEgH00o4PTa6iPRlOF5Hw02odsR5o6RsPi2X/H23oe1dntoxkt7KPR/tcy+0DUP6I+Gg5/12ei2iiXNYQ556PRFYDZv6PnsY/GGGPMvMQDjTHGmKJ4oDHGGFOUsVnKmfVHTWutfpdVq1Z1tnlOjdYBqdaufhfVT1kjb5sPsXfv3s62+m84Nn1YGqhph21O5yjwMhVa1tTpqsmzJq6pY3QeFc+VUf9Nzm68vMRoyc2jUXvgftT5LjmfjS4JreftNbVNLs0NkJ9H4xQ0xhhj5iUeaIwxxhTFA40xxpiieKAxxhhTFA80xhhjiuKBxhhjTFGi33C2iDgE4MEyzTGzZENVVSvbdxsdtpuxxbZjZkvPttP3QGOMMcb0g6UzY4wxRfFAY4wxpigeaIwxxhTFA40xxpiieKAxxhhTFA80xhhjiuKBxhhjTFE80BhjjCmKBxpjjDFF8UBjjDGmKB5ojDHGFMUDjTHGmKJ4oDHGGFMUDzR9EhGfiojbR90OM/+w7ZjZMt9tJzvQRMSj9O+ZiDhF5Z8fViObtqyMiM9FxPGIOBoRf5TZdye19UDTSUuH2d6mHbdGRBURb5XP90TEzcNuzzAZF9uJiJ+MiDsj4lhETEXE70fExZn9bTsjZoxs5+3SllNNe1Z02d+204XsQFNV1dLT/wDsAvAK+qzzoI+I80o3FMAXAEwBWA9gFYAPtOz/iqbdNwB4AYDbdIchtfshAG/NPdzORsbIdi4FcDuACQDPAjAJ4N+3HGPbGSHjYjtVVf2mtOX9AL5aVdXhzGG2nRmYlXQWETc3o+PbImIKwCebUfRO2a+KiK3N9qKI+EBE7GpG+49FxJIer/fjANYB+NdVVR2vqurJqqr+tpdjq6raC+BLAJ5DbfrViLgfwP3NZz8VEd9p/ur9ekQ8j659fUR8OyIeiYg/AbC4l+sS9wL4BoA3d/luiyLiwxGxr/n34YhY1NSdvs9viYiDEbE/Il4nx87qno6KYdtOVVWfq6rqy1VVnayq6iiATwC4scdjbTtjxLBtR84ZAF4L4NO97G/bSZmLj2Y1gOUANgD45R72fx+AqwBcB2Ar6r8s33W6srnZN3U59kUAfgDg0xFxJCL+OiJe0ksjI2IdgJcD4IHpVQBeCODZEXE9gD8A8AYAlwP4OIAvNjfzAgB/CeAPm+/6ZwB+Rs6fa/dp3gng1yJi+Qx172i+33UAng/gh5D+FbQa9V/lkwBeD+B3ImJZU5e9p2PMMG1H+REAf9/LjradsWRUtvNi1ErKn/fSSNuOUFVVT/8A7ATwo832zQCeALCY6m8FcKccUzUNCQAnAGyhuh8GsKPHa/9ec67XAzgfwD8BcAzAikxbH232eRDARwEsoTa9lPb9XQC/Icf/AMBLUD+U9qFZ8rqp+zqA23tsd+eeAPhTAO9vtvcAuLnZfgDAy+mYlwHYSff5FIDzqP4gagOZ0z0d5r9R2o6c88cAHAVwVUtbbTtj8m+MbOcOAJ/qoa22nRn+zUUrPFRV1WM97rsSwIUA/qZ+AwWaBp/b4/GnUN+EO5ryH0fEO1BLIH/V5ZhXVVX1lS51u2l7A4BbIuKN9NkFqDX9CsDeqrmbDQ/22GblXQC+GREflM8n5JwPNp+d5khVVU9R+SSApZj7PR0lw7Sd+oCIFwH4HICfrarqvpbdbTvjyyhs50IArwHwyh52t+3MwFyks0rKJ5oG1FeOWE11h1EPFtdWVXVZ8+/Sqnaa9cI9M1xPy/3Ax+4G8F5q12VVVV1YVdXnAewHMBl0R1EHI/R/war6PuqAhndI1T7URsfn39fDKed6T0fJMG0HjUzxRQC/UFXV/5xDuwHbzqgZqu00/DRq5/pXZ9FeZsHaziDn0fwdgGsj4rqIWAzg3acrqqp6BrUT9kMRsQoAImIyIl7W47n/AsCyiLglIs6NiJ8FsBbA1wbQ7k8A+JWIeGHUXBR1SOzFqJ1pTwF4U0ScHxGvRq1lzpb3AHgdgMvos88DuC3q8O0VqP8C+WzbiQZwT8eJYrYTEc8B8GUAb6yq6j8PuN22ndFT8rlzmlsAfEbeMObKgrKdgQ00jRzx6wC+gjqq4k7Z5W0AtgG4KyIebva7+nRl1LHnL+5y7ocA/CMA/wrAcQD/BsArq3yYYa/t/haAXwLwEdT6/TbUGieqqnoCwKub8kMAfg71Xwcdcu2e4Vo7UDv4LqKPbwfwLdRvbd8F8O3ms17I3tP5QknbAfAW1K/7d8SZ+RA9BQP00G7bzogpbDuIiEkALwXwmQG3e0HZTgx2kDbGGGNSnILGGGNMUTzQGGOMKYoHGmOMMUXxQGOMMaYoHmiMMcYUpe/MABEx8jC1RYsWdbYvvPDCpE7LS5acyfV2zjnpuJrOh0rrn3zyyaTu8ccfT8qPPfZY17qnnjozoVbPw3UAMKiov6qqon2v0ZGzm1w/aPm881KTXbp06YzbM5X5WL2mlp955pnONvf1TOVTp051ttUWnnjiiaTM9WoLI+JwVVUrR92IHOPwzMnZYK7MzyoAuOCCC5Iy17ed99xzz+1ax/Rj22qDWn766ae7Xue+++7r2XaGka564Kxbt66z/YIXvCCpu+GGG5Lys571rM42Dzpt5f379yd1999/f9fy9u3bk7qDBw92tg8cOJDUaVkHooXI+eefn5RzfyysXJna9Y03nknEfNNNaY5BLS9ffia3oF5Tf7g8INx7771JnZa/973vdbZ37NiR1O3cubNreWpqCmPAbFObLCguuujMFJTLL788qWO7AoBVq1Z1tjdu3JjU5cp83Ezn5eteeumlSR3bb5ttP/roo53thx56KKk7cuRIUn744YfRjZtvvrln27F0ZowxpigeaIwxxhTFA40xxpiieKAxxhhTlLEJBuCICt6eqczRRJdccklSd9lll3Utt0V1cHQGR2YA06Mv2ImfiyzT85jpaCTOxReny5wvW7assz0xMZHUscNUgwg0eo37RaPBNPqPI8m0DzWabe3atZ1tjUg7fvx4Uj58+EweWP3efB29pu1o8LB95CIdgdT5vnr16qSO+x9IbVSDV9iWgdQGtI/Vlo4dO9bZVvvl82gwgNoZ2zZvA+UiIf1GY4wxpigeaIwxxhTFA40xxpiijMxHo7NVWVfU2bRaZp1TJzWpJsrlNr9LTiPPZQpQnTM3+9vr/0xn8eLFSVknxE1OTna2169PV7RlvVx9O6qzs66tfXby5MmkzPq42onq7OwbUrt55JFHkvKhQ4c62zyxF8hnlFC7sR31jz5z2PfbNsmRbVInXfKkcADYtGlTZ7vtWcZltR2eWAmkNqt+az5P2zXZznJZTQaJ32iMMcYUxQONMcaYonigMcYYU5Sx9NFosktOaAekc2PUR7NixYqkzD4a1eFVn2QNVDXwnI9G4915X9X3ra1PR3006gPhJKpbtmxJ6q644orOtvpoVMdmH40mCzx69GhSZlvReTPaPq7X/lY/DCfVVDtnm1K9fkwyPZ9VsB8ml1UZSJ8z6qN57nOfm5SvvvrqznYu67vW555HWq92xr8h/T1pWX2Xw8BvNMYYY4rigcYYY0xRRiad6esbv6qqVKFpZfg1NrdmA5CmjlA5Ql9NT5w40dnWsFSu07LWjeHiVmONyhaaVojlUJbKgNQ2VO5QeYH7m9N5ANMlLpbO9Lya6obDr1WSy4VCawgt20rbIn2WYPsnt8Ceyqwa7szyvT6PdEoF26iu96ISPEtpajv6DOKypqBh2VglZC3zd2tbJE3Ls8VvNMYYY4rigcYYY0xRPNAYY4wpytj4aFgD1RDlNWvWdC2rXprT6VUvza3nnqsDUk0/lyrCKWjaaUuJz32oGjf7UlTjVt2dNW5O1w9M99FwH6rPSDV57tO2sORcWDzr7g6LHzx6D7mv9H5rv/Gz44EHHkjqNHyYbUl9gbpsBJd1+oX6kLmsvhOeAtDmU+T7oM+u3LNsLviNxhhjTFE80BhjjCnK2EhnHNKs0oRm7M1JZxoqy6/AbdLZd77znc723r17kzqVVg4cONDZ1jDEfrLwmumobJG7n/xqr/2g8gLLGJxFGZjevyxrqT2q5NVP1m8+VmUJlm8snZUn12856Wz79u1JndoDS2tqkypN5UKW1Xa4nMvQrFMANAMFX0flOv0dHDlyBIPAbzTGGGOK4oHGGGNMUTzQGGOMKcrY+Ghy4c3qo5mYmOhszyW8eceOHUn5nnvu6WxPTU0ldZpmhrVW1dpZT/fKiO203SPWy1XHVo2ZUT8H939beDPr7pxiRuuA/rT+XHhzzm7M4OF7rLai/hHO7q39pv5cTi2UC1EGUvtt88txWUPu2S+jvxHNfs82qu1h3zMA7NmzB4PAbzTGGGOK4oHGGGNMUTzQGGOMKcpAfTTqd9E06Fzm9P1Amt5/9erVSd3atWuT8qpVq7qeR+fR8FwKjSdX/w5fV+dgaDw8p53Irdypq+Zp2Vr89HuiaTv279/f9djcyoLqr2PtWrVptRs+l9apfs+pbzQNjtoGH6v+HFOWflLQKFzf1m/8nFP/iD6D2Neiz89cWX007EfUpSn0mvws09+BzpvJ/fb6wW80xhhjiuKBxhhjTFEGKp2pVKZZQ/kVTlfCZDlMpTMOZ9Z99RUyJ53papybNm1Kyiyt6CsjhzcCaaishk2z9KMyUC69ykJFX9819JjR7Ldsc2p/uVUSVXJVO2JZVeUPlVlYblBb0LB4ld3M6MhJZyqz8XNEn2v6XGHbyqWK0bLaa66sqxBv3ry5s63PS5XOcuHN+rzScOfZ4jcaY4wxRfFAY4wxpigeaIwxxhSlqI8mF8qnPhpOxa4ramp4M4fvqU9Gy6y1qpa6cePGpMwhrZpqW1PScFk1fL4PuVBYU6M6sabw53Q/6jtj7VxX1MytQnjllVcmdWpjXM6lNQLyPhpNC6/pQczoyKWg0RBmDi3WMHoNJ2Yfs06hyJX1eal2x9dVPxG3oS28mb+3/vbUF20fjTHGmHmBBxpjjDFF8UBjjDGmKAP10ahGrhpjbh4D64rqS9FlAziGPJdWAki1V9VW1U/Eceqqa6omyt9Fz8ttUI1e/Q+5eRULJT2N+i20rHNnGNbO1SejKTy4D7du3ZrUqT2yn7At1Tvr2tq/2nbPo5mfsO9XbYXn9QGpL1CXAdcyH6u+Xn2u8DNJfdE5NMUTp6BRn7HOo1GfzWzxG40xxpiieKAxxhhTlJGtsDkXOG2Lyhi6+h3X62tirqyvmyqlcbbUiy++uGv79Bq6kiPLY5qOpi3kciGiclgue7OWWaZQOVZlVA7FV/lTU+RwyLWu2qrZcNVezfBQaZWlck33olM12D40xQunfwFSWVZltlzqI7XXXPZ7ldVzIfZavvfeezvbu3btSuo0S/2g8BuNMcaYonigMcYYUxQPNMYYY4oyL300HCKqadg1nDS3+mGurKHZmkaeU9SoL4XPoykcNEyaQ3lVQ24L3V6I6D1iXVv7SPVwDi3VMFP12fC5du/endSpj+b+++/vuq/65OyjGR05H00ulBhI7YPDl4Hp6YyuvvrqrufNXUf9jwq3X+2I/TBsjzOVd+7c2dl+8MEHkzr7aIwxxsxLPNAYY4wpigcaY4wxRZn3Phqd46Ax4xxf3o+PRjV8XTaA59Gonst+GdZDgelLsLLWqj4YL/M8ndw8mpxPBkj7sJ95NOqDUx/Nfffd19nW5STUh6jzvMzwUB8Nz0vR37D+Ttle+LcPAFu2bEnK11xzTddrqv1yWe1My7n5gzkfzd13352U2UbVJ6PP00HhNxpjjDFF8UBjjDGmKAOVzvRVT9O4sIygr2wnT57sepxKSLlXSD0vv1LmZDUt6zVVWuHQYw2F5hBmTU+jK+yxlKLhzAsl06/KCZoFnMsqcbA8tn79+qROV83k1CF6HrUjDkvWjMxqN5zhVu2vzZbN8MhNF9B+0d8eT5tQeXT79u1JmSU5nc6QK6t0rjIrT4XQZxm3SUPqtcz2qvZZ6pnjNxpjjDFF8UBjjDGmKB5ojDHGFGWgPhrVOXPpNjQVA4cWs78GyK/AqPtqeB5rq6pV7tu3LylzunfVSzW8ma+jqU9yGi2vJKrt7+f+nU20rczKqwnqveaVMDVdO6cCAVL/mKaFV78L24Laie7Lvsc2zdtphEZHzgeqvzXtJ7YHtU89du/evZ1tfW7kyto+fbbxddQGeUqFroqp52Eb1WdOKfv0G40xxpiieKAxxhhTFA80xhhjilLUR5PztahGzvMPVPPM+Wh0X/XRHDt2rLOtPpo9e/YkZU4Xo36DTZs2JWXW5VXX5GPbfDT8vfV+6VKuZyt6r9knAwAXXXRRZ1vnIeV8NNdee21S5nQgupyEat5sK6y5z7RvzkejtmEfzejI+Wi0X/SZw346feaw/wYAtm3b1tnmZZ2B6elq+Jmp6Wr0WcZltUGeR6NzbDQNEtvosOzTbzTGGGOK4oHGGGNMUQaqzbS9fnJZMyezhKSvjCop8etmWzgp76uvkCyrAalcotl9VWrh9BCaeodTqqgMpFIap0LRsEmVlM5W9Htq6Dun8dFUQBweqilncnKn2piGhLKsqilo1I64/3MZgmeqZ1jaaZM0eF+VhLRsZob7TX/DSu7ZpeHvnEVc+0JTH61atarrviqBsY2qG4DDm/VZpVLfKFJb+Y3GGGNMUTzQGGOMKYoHGmOMMUUZWfysas7sh9HQPdVA1e/BaPgra6+awj0X4qqpTtS3wtq7aqt8TfVTaepvLi/UdCUa6q6rGy5fvryzratmcj+pb0f9ITnfnvpdcisN6nXYn6d9pj4atl1tH9tCzk6A1K7UbrTc5n8w7XBf6bIWuT7W54auAsu2rTanzw72Fe7atatrnZ5nHJ4jfqMxxhhTFA80xhhjijKW0pmGHevMW3791NBYlc44ZFjPqxIdv37qK25OOlNy0lku64HuOw6vvMNApTO91znpjPtJJS2VOPh+tklnXFZplDMVAKl0pnaRW31VbZelXJV1NaSWfy9tmYctnc0dls5yK8ACqXSm4cy64i5nCtHMKmqjHMK8e/fupI6fbWrL4/Ac8RuNMcaYonigMcYYUxQPNMYYY4oyNj4a1pk19YKGN7NmrilJcuGDel5NLcIaqGqpqrWyFq/fhbXWtvDmUax2N26UCm9WHw37KuYS3qw+pFw4q9oRl9Wfw/ao30Vhn0Fb6iczd/oJb86tsKvTJthHo344tdHDhw93tjW8mffV48bBR+c3GmOMMUXxQGOMMaYoHmiMMcYUZWx8NKxPagpsnbfAmqOeR+ctsN6v2v/k5GTX87JfAJg+P4f9RJrahstal5sPodr6OGirw0C/p6ZQYf+dLhmR83Hp/Be2DV3pVG0hl84/h/pW1Ha5rFo/26f6BDXtEvtztK2q0es8G9M/PFdG+yLX5zrHRm00t3SFlvlZkluqou13MAr8RmOMMaYoHmiMMcYUZWTSmcol/Jo4NTWV1OXCh/W1UENluV5fcdetW5eUOZVIbpVHbb+msuEwak0lolIavwIvVOlMpTKVBfh+argor1CooeO51DETExNJndrNmjVr2po9I3oelVlYvsu1L7cSK5CG0KpUovKsmTssgeXkeWB63zEq/XLIsj5HtB/ZvnNTIcZxWoTfaIwxxhTFA40xxpiieKAxxhhTlLEJb1bfBZNLe92WYp61bdVS2Sejx6p+riGjrJcePXo0qeOU3W3p3vm7jWPqiGGgPhoN62QfjfrO+P6qj0ZtjH0gajcazq4aeK9oyHIupbx+b7Y/9cmoX4CP1d8Op1IygyHno9EQdn7OaOh5Pz4afXbkfDT8vBqHcGbFbzTGGGOK4oHGGGNMUTzQGGOMKcrY+GhYu1TtWv0auVQi6nfhtPKqw2vK+bVr187YHmC67s1pPVQjZx+D+hvU39RrbPzZjPqi1NfCWrX6LnLzkFSrZv+OzovSVO/qWylBLp2/tl3vEfsFdalz9T+ZucO+N72/6jdkG1Ufjdo2+3P1OZdb9n2++W/9RmOMMaYoHmiMMcYUxQONMcaYonigMcYYUxQPNMYYY4rigcYYY0xRot90BRFxCMCDZZpjZsmGqqpWtu82Omw3Y4ttx8yWnm2n74HGGGOM6QdLZ8YYY4rigcYYY0xRPNAYY4wpigcaY4wxRfFAY4wxpigeaIwxxhTFA40xxpiieKAxxhhTFA80xhhjivL/AdU5jpTUN8kJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_img_label_prediction(imgs=example_train_imgs, y_true=example_train_targets, y_pred=None, shape=(2, 3))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xbBAH9OTv9Ii"
   },
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W46ofUE6v9Ii",
    "outputId": "5e698e3a-c465-45c3-9f72-07f566ef8055"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1247, 8, 8)\n",
      "Test score with penalty: 0.8909\n"
     ]
    }
   ],
   "source": [
    "scaler = StandardScaler()\n",
    "print(X_train.squeeze().shape)\n",
    "X_train = scaler.fit_transform(np.reshape(X_train, (X_train.shape[0], -1)))\n",
    "X_val = scaler.transform(np.reshape(X_val, (X_val.shape[0], -1)))\n",
    "\n",
    "# Turn up tolerance for faster convergence\n",
    "clf = LogisticRegression(C=50., multi_class='multinomial', solver='sag', tol=0.1)\n",
    "clf.fit(X_train, y_train)\n",
    "#sparsity = np.mean(clf.coef_ == 0) * 100\n",
    "score = clf.score(X_val, y_val)\n",
    "\n",
    "print(\"Test score with penalty: %.4f\" % score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 339
    },
    "id": "E_G2Rt0Gv9Ij",
    "outputId": "59791f8a-e8ac-41f9-81e9-8cdc92cadd42"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAj8AAAFCCAYAAAAe+Ly1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnIElEQVR4nO3deXxddZ3/8fcnN0n3PaWlhaQsRXYoIKuFiigDIiigAyhYREfG7ScgKooziiKOjoozrj9BUREQQRgEFHSwllXZylJApKULBbomTbekWT7zxzkdQqZpz6e9l9vwfT0fj/uguXnne86537u8c+4lX3N3AQAApKKm2jsAAADwWqL8AACApFB+AABAUig/AAAgKZQfAACQFMoPAABICuUH/Z6ZfdHMrq7g+LPNbFr+bzOzn5pZs5n91cymmtnfKrDNRjNbbWalco9dCWb2BjObZWarzOwTZjbIzH5rZivN7Ndm9l4zu7PAOJ8zsytei31+PTCzfzazxfl9ZUy19wfoL4y/84P+wMzOkHS+pN0lrZI0S9Kl7n6PmX1R0q7u/r7XYD+mSrpW0hvcfU0Zx50n6YPu/sdyjflaMrMrJbW6+3n512dK+rikw929swr7M0nS85LqXsvtv8b3xTpJrZIOdffHKr094PWEMz/Y5pnZ+ZIul/RVSeMkNUr6vqSTqrA7TZLmlbP4vE40SZrd6+tnq1F8+jMzqw3Ex0kaqFff7kW3Y2bG8z/S5e5cuGyzF0kjJK2W9O5NZL4o6eoeX/9a0suSVkqaKWmvHt87XtJTys4eLZL0qfz6Bkm3SmqRtELS3ZJq8u/Nk3SMpHMktUnqyvfpS5KmSXqhx/g7SvqNpKWSlkv6bn79LpLuyq9bJumXkkbm3/uFpG5J6/JxPy1pkiSXVJtnJki6Jd+35yR9qNfxXy/p5/lxzZZ00CZur70k/SEfa7Gkz+XXD1BWMl/ML5dLGtDj505QdsatRdJ9kvbNr78rv03a8v2/VtJ6SR351+dImi7pngL70HsuD8231SLpMUnTenxvhqQvS7o3P+47JTXk31uQ336r88thvW6DCfntPbrHdVPyuanLv/6ApKclNUu6Q1LTpvZf0j/0Ou7HCs7dDZKuVnYW54OSDpb0UP71Yknf2sgc7iZpTY9jvCu//nBJDyq77z+o7Mxbz9vr0vz2WqfsDFXVH+NcuFTjUvUd4MJlU5f8BaVTeQnoI9P7BfMDkobplRfzWT2+95Kkqfm/R0k6IP/3ZZJ+KKkuv0zVK28Lz5N0TP7v6Xr1i/g05eVHUil/gf62pCHKfit/U/69XSW9Nd+nscpK2eU9xvnfbeRfT9Kry89MZWe7BkraX1m5OrrH8bcpK3al/Fge6OO2GpbfBhfkYw2TdEj+vUskPSBpu3wf75P05fx7UyQtkXRIvo335/s8IP/+DGVv2/U1J/97u21mH/735yRNVFYWj1d2lvqt+ddje2xzjrIiMCj/+msbu/36uC3u0quLyDck/TD/90nKisoekmolXSzpvsj+9xh3c3PXIemd+TEOknS/pDPz7w9V9rbWxvb/VccoabSyonZmvs+n51+P6XF7LVBW3GqVlzwuXFK8cNoT27oxkpZ54O0Td/+Ju69y93ZlLy77mdmI/NsdkvY0s+Hu3uzuj/S4fntlv913uPvd7h79QNzByn7Lv9Dd17h7m7vfk+/Tc+7+B3dvd/elkr4l6agig5rZjpKOkPSZfMxZkq6QdFaP2D3ufru7dyk7k7RfH8OdIOlld/9mPtYqd/9L/r33SrrE3Zfk+/glZS+kkvRPkn7k7n9x9y53/5mkdmVnZqI2tQ89vU/S7flxdbv7H5SdETm+R+an7v6su69TdvZr/8B+XKOsIMjMTNJp+XWSdK6ky9z96fy+91VJ+5tZU2D/i87d/e5+c36M65TdF3c1swZ3X+3uDxQ8nrdL+ru7/8LdO939WknPSHpHj8xV7j47/35HwXGB1x3KD7Z1yyU1FP0shJmVzOxrZjbHzFqVnZ2Qsre1JOkUZS+e883sz2Z2WH79N5T9pn+nmc01s89uwb7uKGn+xoqamY0zs+vMbFG+X1f32KfNmSBphbuv6nHdfGVnRjZ4uce/10oa2MdttqOysyV9bWd+r21MyP/dJOkCM2vZcMnHmqC4Te1DT02S3t1rm29SVlI36H3cQwP7caOkw8xse0lHKnvr8e4e2/5Oj+2ukGTKbvOi+y8Vm7uFvX7mHGVns54xswfN7ITAtub3um5z2wKSRPnBtu5+ZWcY3lkwf4aytyyOUfZ5oUn59SZJ7v6gu5+k7K2dm5WdLVD+2/sF7r6zpBMlnW9mbwnu60JJjX2Ujq8qe4tiH3cfruyshvX4/qbOMr0oabSZDetxXaOyzyxFLZS08ya209RrGy/2+LlL3X1kj8vg/OxCOfehd+4XvbY5xN2/VuBnN3vWzt2blX1O6B+V3W+u63G2b6GkD/fa9iB3v28z+997u0Xm7lU/4+5/d/fTld1H/03SDWY2ZHPHo/87f5vdFpAqyg+2ae6+UtK/SPqemb3TzAabWZ2ZHWdmX9/IjwxTVpaWSxqsrHRIksysPv97MyPyU/6tyn7bl5mdYGa75m9/rFT2Ad7u4O7+VdlnQb5mZkPMbKCZHdFjv1ZLWmlmEyVd2OtnF6uPF1R3X6js8zeX5WPuq+zswJb8baNbJW1vZp80swFmNszMDsm/d62ki81srJk1KLvdN2zjx5LONbND8v9TaIiZvb3Xi3o59qGnqyW9w8yOzc/oDTSzaWa2Q4FtLFU2f5srWdcoewvqVL3ylpeUff7rIjPbS5LMbISZvbvA/i+WNGnD/0m1JXNnZu8zs7Hu3q3sg95Ssfvi7ZJ2M7MzzKzWzP5R0p75/gLogfKDbZ67f1PZ3/i5WNmL2kJJH1N25qa3nys71b9I2f/V1fvzEmdKmpe/9XSuss+5SNJkSX9UVlDul/R9d/9TcD+7lH2+YldlHyx9QdlZBSn7/MwByorVbcr+j7CeLlNWPFrM7FMbGf50ZWexXpR0k6R/9S34m0D52y9vzffzZUl/l/Tm/NtfUfaZmsclPSHpkfw6uftDkj4k6bvKPkT7nLIPMYdtZh965hYqO4v3Ob0y7xeqwPOWu69V/n825bdpX59NukXZ3L/sPf5WjrvfpOysy3X5feVJSccV2P9f5/9dbmYbPk8Wnbt/kDTbzFZL+o6k0/LPAm3umJcr+zzSBcrK/6clneDuyzaWN7Pfmdnneny9Ov87VrLsj3eu3tw2gf6KP3IIAACSwpkfAACQFMoPAABICuUHAAAkhfIDAACSQvkBAABJofwAAICkUH4AAEBSKD8AACAplB8AAJAUyg8AAEgK5QcAACSF8gMAAJJC+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApFB+AABAUig/AAAgKZQfAACQFMoPAABICuUHAAAkhfIDAACSQvkBAABJofwAAICkUH4AAEBSKD8AACAplB8AAJAUyg8AAEgK5QcAACSF8gMAAJJC+QEAAEmh/AAAgKTURsINDQ3e2NhYOG/eFdwdC6XXe6y71dd4KC9Jal8TiltN6CaV1w0snJ2/YL6WLVseu5H6EJ7L7s5ybLbv8bti43evbglvo3PE+FC+NvirgXl34ez8BQu1bHl55lKKz2dHd+yx0FX80CRJtTWxQyttwa9hNV3r4z8UUaorHJ2/YEHVHpvtXbG5LFlsN7s8Nv6q9vhzxZD6Uig/OPrgVPFjmD9/Qdkem+Hn2cB+SpK3LovtUPAl0IaPif2ApPaF80L5AaNHhvLdQ0eH8rMefXSZu4/tfX3olbqxsVH33ntv4XxNW2tkeClYHF5YXx/K7zAw/qAszX0wlI/eWTq2261w9oipR4XG3pToXJZWL41tIFAEJKm0Znkov3bmLaG8JLW848JQfvTA2BNybcfawtnDp70lNPbmROdz0aqO0Pgr22O/yGw3pHhxkKSh9fH2M7T1hfDPRHQNH1c4e8SR08q23ehczmmJlcARA2L36+jcz3h+RSgvSQfvMCKU33dM7P4V+eXt8KOODo29KeHXzM620Phdf7wqtkPdsbksHXN2bHxJcz8Z+5md3vuuUH79YaeF8iOGDp6/set52wsAACSF8gMAAJJC+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkhJaT8IUW32rZm1zaGdeHrRDKL+2I/anup9qjy/XsmzQ/qH8kaNi24itmbUFa5P1tV3F5rLrgdhyEg999dpQ/rDr/38of8u+HwrlJWlacPrrVi8O5b1+SGwDZWTerZr1xZfX+OafF4bG/8gRO4XyDYNiSyh854H4UhXn7zc8lPfgmlbVYt6tmvbVhfNzVsSW7Tlmp5Gh/Pj5d4fyk/c5NJSXpP+aW/y+K0lDg2uB7Tws8lJXvvtJ+DWzNfacs+TR2aH8+o9/O5Sf9PLjobwkta1sD+Vrt58Uyrd0BBca7ANnfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApFB+AABAUig/AAAgKZQfAACQlNDaXq7Y6lIvDJgY2pnnlsXWdzlqVFso3/HHn4fykjRvn3NC+d8vio1/bFN97AeqpDRmfCh/wPknhvIN51wfyo+dvH8oL0lPX7R77AdC665JXQOGxcYvI7caddcPLpz/xNSdQ+PvMu+/Q/nOBxeE8js2nhLKS1LHwBGhfKmiS3uVb/Auq9GaUvF14t62c2zbV47fP5Q/4cOHhPK/ecduobwkfWRKQyhfWrM8lO+ycaF8uURfM1UKvSSr/oLvhPL/NXtJKH/2/geE8pK0z7e/HsrPHRRbN3DVqvWhfF848wMAAJJC+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApMQWEgn687zmUP6ACbG1ep74wBmh/G1/nBfKS9Jnlnw0lD/1mmdC+WOb9gzlyyqwdlX3fseGhr5z+OpQfsjY20L5z599YCgvSd0DY12/1PpSKO91A4uHrby/d5hiq0tNXvxAaPy7pn8llB8wPLZm3SG3nh3KS1J98/xQ3jrbQ/nO7eJrVJVDjaRBtYHZ/OOVofGfao3dDh84IvZYGzmwLpSXpLbrYutB1Q6M3b9qT/l0KF8uXS61ru8unL9uTvR5YXEo/bHxS0P5zvVDQ3lJenbApFD+ukdfDOVP2GO7UL4vnPkBAABJofwAAICkUH4AAEBSKD8AACAplB8AAJAUyg8AAEgK5QcAACSF8gMAAJJC+QEAAEmh/AAAgKRQfgAAQFIqurbXM4tj6ztNGjU4lK9tbgvlJwyMH26pJbbuyOihsTVnvFQ876HVmzY3luQ1xW+P6JanbB9bE+amy04O5ZvXdYTyklSz+KlQvnPpolDet9+7eDY0crHxImNafWAdMkmT375HKD/xs18N5dcPia8H9bcPXRDK73Tim0L5mhOLr+1V7vmMqN1hl1D+49P3DeVbnoitV9gxvvhaVhtcOu70WP6IhlC+K5At51yWTBpeX/wcw+VXPRwa/5PTY+uurR63Tyh/x5zY+pySNHdFSyi/cu36UH6fsYNC+b5w5gcAACSF8gMAAJJC+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApIRW+jTFFrg8bNKo0M5c89DCUP4//mV6KL/vovmhvCS9MLgxlP/CW2PL4pkXXwSwfMuaxnXd+h+h/ITtdwrlF044OpT/wcznQ3lJOmqneaF8V3Bh01JX8QX6rKpLYUorJsYWRBxw8Q9D+SO/d38of89ZK0N5SVpw7wuh/K5nj49toLuzcLSaj83unQ4K5RvfdnAo/8h//C6UP/voh0J5STrlxYmh/HtujC2a/avT3lA4W87Hpktq6yo+XsPEEaHx37ZLbIHXy++PvcY+Oi++sOmP3hNbPHX0updC+cgitZvCmR8AAJAUyg8AAEgK5QcAACSF8gMAAJJC+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJCW0tpfkUmD9omObBodGf3rJ0FC+/Y1nhPLN7fFVQeauaAvlj6pdEMp31gfWDgusA7Y50XXa6nfdNzS+t8dutxED60L5xUtia/tIUs3Jx4bypbmxNYp++Uzx9alWtJVrhZqMS1ofWENozor20Pgr2jpC+fXBx9qVLw0L5SXpoLtnhPIPdsT2acyq4o+39sBtvznRx2Zb7ZDQ+AOP+3Aof+huU0J53ym2bpwk/fPo2OporW3F112TpM7SwMJZL/NKbZEzDLeed0Ro7LprLgnlP3987DXzt9s3hfKStOoL54TyI089KZSv2TvWE/ocpyyjAAAA9BOUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApFB+AABAUig/AAAgKZQfAACQFMoPAABISnBtL5NK9YXTNeuKr3UkSefvF1vf5/k1sfVdxgwqhfKSdFTn07EfqAuuURRar6t86wdJkrqL3342dlJoaAuuQ7ZX+3Oh/I/OfmMoL0n3rCi+Lp0kTd7l6FD+6mtmFc4uXx3bl83pdmldZ/H7x5D62GOhJvhr0ozPHhXKL1kTWztMkr7557mh/DOLWkP5f33HnoWzHd1lfmwGPPzSmlB+1KDYOnpDdpwayre1xW+LPRpid7AJdcHHT3vxuTcv37p7NZIGlIqvFTZAscdB9/jxoXznUw+E8scffmooL0nLtxsVytfusGsoX67Z4cwPAABICuUHAAAkhfIDAACSQvkBAABJofwAAICkUH4AAEBSKD8AACAplB8AAJAUyg8AAEgK5QcAACSF8gMAAJJi7sXXYTGzpZLmV253sBlN7j62HAMxl1VXtrmUmM9tAI/N1w/m8vVlo/MZKj8AAAD9HW97AQCApFB+AABAUvpN+TGz8WZ2nZnNMbOHzex2M9vNzCaZ2ZMV2uYAM/uVmT1nZn8xs0mV2E5qqjSXR5rZI2bWaWanVmIbqarSfJ5vZk+Z2eNm9t9m1lSJ7aSmSnN5rpk9YWazzOweM9uzEttJUTXms8e2TzEzN7ODKrmdLdUvyo+ZmaSbJM1w913c/UBJF0kaV+FNnyOp2d13lfRtSf9W4e297lVxLhdImi7pmgpvJylVnM9HJR3k7vtKukHS1yu8vde9Ks7lNe6+j7vvr2wev1Xh7SWhivMpMxsm6f9J+kult7Wl+kX5kfRmSR3u/sMNV7j7Y+5+d89Q3mbvzn/Df8TMDs+v397MZua/WTxpZlPNrGRmV+VfP2Fm521kuydJ+ln+7xskvSW/Q2HLVWUu3X2euz8uqbvSB5iYas3nn9x9bf7lA5J2qOAxpqJac9na48shkvi/cMqjWq+bkvRlZScL2ip1cFurtto7UNDekh4ukFsi6a3u3mZmkyVdK+kgSWdIusPdLzWzkqTBkvaXNNHd95YkMxu5kfEmSlooSe7eaWYrJY2RtGzrDidp1ZpLVMa2MJ/nSPrdlu0+eqjaXJrZRyWdL6le0tFbeRzIVGU+zewASTu6+21mdmFZjqQC+kv5KapO0nfNbH9JXZJ2y69/UNJPzKxO0s3uPsvM5kra2cz+U9Jtku6sxg6jT8zl60tF5tPM3qfsifqoSu48XqXsc+nu35P0PTM7Q9LFkt5f4WPAK8o2n2ZWo+xty+mv0b5vsf7yttdsSQcWyJ0nabGk/ZQ9IdZLkrvPlHSkpEWSrjKzs9y9Oc/NkHSupCs2Mt4iSTtKkpnVShohafnWHAiqNpeojKrNp5kdI+nzkk509/atOwxo23hsXifpnVuw7/i/qjGfw5SdcZphZvMkHSrpFtsGP/TcX8rPXZIGmNk/bbjCzPY1s6m9ciMkveTu3ZLOlFTKs02SFrv7j5VN1gFm1iCpxt1vVPabxgEb2e4teuU3kFMl3eX8VcitVa25RGVUZT7NbIqkHykrPksqcFwpqtZcTu7x5dsl/b2Mx5Sy13w+3X2luze4+yR3n6Ts83gnuvtDlTnELdcv3vZydzezd0m63Mw+o+xDVPMkfbJX9PuSbjSzsyT9XtKa/Pppki40sw5JqyWdpezzPD/NT9NJ2afge7tS0i/M7DlJKySdVq5jSlW15tLM3qjs/3wYJekdZvYld9+rjIeWpCo+Nr8haaikX1v2/yAscPcTy3RYSariXH4sP4vXIalZvOVVFlWcz36B5S0AAEBS+svbXgAAAGVB+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApFB+AABAUig/AAAgKZQfAACQFMoPAABICuUHAAAkhfIDAACSQvkBAABJofwAAICkUH4AAEBSKD8AACAplB8AAJAUyg8AAEgK5QcAACSF8gMAAJJC+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApFB+AABAUig/AAAgKZQfAACQFMoPAABISm0k3NDQ4I2NjYXzre1doZ15uWVdKL/LdkND+fVd3aG8JM1ftjaUr62N9cnJYwYW35cFC7Vs+XILbaAP0bm07thc+uoVobwNbwjl1bk+lpekrs5YvrYuli/VF47OX7BAy5YtK8tcSlswn12x288DxyZJ7V0eym+J6ON5XUcsPyDwWF764kK1Nq+ozmOzqyO2AYvtpnXHHjfdtcWf0zaI3ltqVi2P/cDgYYWj8xcu0rIVzdWZy8722AZqY4/LLo8dVql9VSgvKb5PpQGhfEmxx/Ejsx5b5u5je18fKj+NjY269957C+fvnNsSGV6X3fRkKH/zxw4P5ee1BO9Ykj5y5V9D+ZFjh4Tyv3v/noWzh097S2jsTYnOZWltc2j8zpm/CuVLx0yP5ZtfCOUlSSuXhOI+emIo3zVqx8LZI940NTT25kTns7Z5YWj8zsCxSdKclmC52oKuND/4y9LTS1eH8juNHlw4+5nTjguNvSnhuWx9KTS+14Se9lVaEysa67bbPZSXpO7g/NfP/HkoX9rvzYWzhx53cmxnNiE6l3XL5obG7xo5IZRv9VgxGTlnZigvSTY6tk8to3YN5Yd1x05IDBy13fyNXc/bXgAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApFB+AABAUig/AAAgKaE/9Wn5pajRg2LLA0ybEvuLusGVJHRQzaLYD0i699DnQ/muN0+PbaAj9tcqyyU6lz77z6HxB+x5cCh/64K2UH7qzKtDeUla8tDfQvnJF38hlO/y+PIp1VKzfk0ov3RtbHmTofWxB+fcFbH5l6RJowaF8rsE/mKzJL3QWnyfSsElIzZlfbdr4ariS1Y0DhsXGv+Wv7eE8ic1zw7lS7MfCOUl6fPrjwjlL3rz+0L5MWtif9G8XEyums7i96O1o3cOjT+nObZqwd61wb8G3jQllJekGctij/2mtthzy7BBsb9Q3hfO/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApFB+AABAUig/AAAgKfFFMgLrF81ZEVu36oVgvjO4lNLcy74U+wFJP73ykVD+M99/IZSvO/WToXy5eH4pyrpj66/c2DYplD9wwtBQfuVz8XXaWua3hPK+tjW2Aes/v0t0vTgnlK+bcUUoXz88to7WwR+4JJSXpHXBJ4ABpdj8TBpRfG3C6FpmmzKgRmocXPzRObe1MzT+srXF1w2TpLUHnRzKD+5aF8pL0l+/+2goX3rLLrEN1NYHwuVbp02SPPC8sHRtbC5v/9uSUH6f3WJz3zkitt6mJJ1+/ndC+a//6xmh/IS9xobyfek/z9YAAABlQPkBAABJofwAAICkUH4AAEBSKD8AACAplB8AAJAUyg8AAEgK5QcAACSF8gMAAJJC+QEAAEmh/AAAgKQE1/ZyWdf6wumfzJgbGn30iIGh/DPLY2vIvOlz8bW9vvyRl0P57lXNsXxd4JjLuHaUKbaCTXR9n5VPLQ3lm9pja6LN64itgSNJO/3DfqF896qW2Ab609pezbH5GThmeCy/Q2MoH1s5LvPE4thagHc+GzvmSw4pfszWHb8/bmK00H2p6e7vh0Y/520fDOUnf/SGUP7zHzkylJekP5+7ZyjfduPXQvnud32ieLimFBp700wqFV9XrLH7pdDo2w0bEMp/+pH22PjDY8/LktS66NlQfs+xsXUda7uLd5BN6T/P1gAAAGVA+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkkL5AQAASaH8AACApITW9nKZumuLr0V184cPDu2MRRabkjTkyTtC+fWL5sQ2IKnl6HND+dEDY+vCeIWyRcaKjDdj3srQ+Dc9vCiUnzxmt1j+S1eE8pK0tit2C46yVaF8teZyS5TeeFwov3DKqFD++ebYulvDFq0O5SXpiSWx+enq7o7lh4wpnPWa4DKJmxorON5z1/8pNP4eR/1jKN8878lQ/m8vTwnlJem6498Tyq/vjj2C3nva5wpnvYxr9EWfZ1t//YPQ+O/dcYdQfuBeh4TyT19yUSgvSXv/MvbcPLg++JoZWCttUzjzAwAAkkL5AQAASaH8AACApFB+AABAUig/AAAgKZQfAACQFMoPAABICuUHAAAkhfIDAACSQvkBAABJofwAAICkUH4AAEBSQqvxWX4pauhjt4Z2Zt4vrw/lu3ZvCuVHnPqhUF6SFq/pDOVHBRc2NS++2GJw3dfNjhUZ7+CJw0LjX7Y0tlBld3ChwrHdraG8JD2xblAof9l9i0P5S44dUTgbXGO1kMhN2DV0XGjsN3TEFip9vjkUV9PI4gsmbzC19a+xHzhwcigee+SXj8lV09lWOL/Xt74ZGv/IK2MLPN/+00+H8qvWd4XykjRu1n2hfNNvvhLK17QVf74wj+9/n2Mp9jw7ZPc9Q+Pf8b5/D+XH7X1TKN84bY9QXpJ2nVgXyv899tSile2xBYr7wpkfAACQFMoPAABICuUHAAAkhfIDAACSQvkBAABJofwAAICkUH4AAEBSKD8AACAplB8AAJAUyg8AAEgK5QcAACQltLZXVPcBbw/ldxkzPpTvWrk8lF80cGIoL0l7DI7dRPWLHgvlu4Y2BMIdobE3xfNLUWM7V4TGv+jkfUL5JWvWh/LWOiuUl6RVIw8J5UcPHRDKj7Li6zHVqjzr0/RUE1hEqG7Js6GxOx6/O5Tv3umUUH7UDbG1miTp+cdja1Tt/NkvhPLWVfw+aaFHU3ktGzYplC/VLgnlp9z3vVDeBsTXafvTXu8P5fc5/eOhfOnlvxXOWkd7aOxyWjLl3aH8sT+PrUPW/NCjofyAUUNDeUmy9lWh/KDaMaH8yNryrLrHmR8AAJAUyg8AAEgK5QcAACSF8gMAAJJC+QEAAEmh/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJKWia3vNXh5br+m822LrHU3da/dQ/kur54byktTxwH2hfNf4xlDe6ocUD3v514MqvOna2DpXxy+5I5QvTT4wlH/EDwvlJempRa2h/AfeuEMoX1q1qHi4qzzr02xg+aWo6HyunHp2KH9cfSiua4/8r9gPSDrp0pNC+egxW2SOvIxre7lL3cUf62PaF4eGn3FCKZTvaPpUKP/0ivgahHsPiu1TzbrmUN47i++Tl3MugyasWxjKdx3yrlC+Ye8jQ/n2kTuG8pLU3BF7nWooRZ65pNLq2P29L5z5AQAASaH8AACApFB+AABAUig/AAAgKZQfAACQFMoPAABICuUHAAAkhfIDAACSQvkBAABJofwAAICkUH4AAEBSLLKOiZktlTS/cruDzWhy97HlGIi5rLqyzaXEfG4DeGy+fjCXry8bnc9Q+QEAAOjveNsLAAAkhfIDAACS0m/Kj5mNN7PrzGyOmT1sZreb2W5mNsnMnqzQNqeb2VIzm5VfPliJ7aSmGnOZb/c9ZvaUmc02s2sqtZ3UVOmx+e0ej8tnzaylEttJTZXmstHM/mRmj5rZ42Z2fCW2k6IqzWeTmf13PpczzGyHSmxna9VWeweKMDOTdJOkn7n7afl1+0kaJ2lhhTf/K3f/WIW3kYxqzaWZTZZ0kaQj3L3ZzLar1LZSUq35dPfzeuzDxyVNqdS2UlHF59mLJV3v7j8wsz0l3S5pUgW3l4Qqzue/S/q5u//MzI6WdJmkMyu4vS3SX878vFlSh7v/cMMV7v6Yu9/dM5S32bvN7JH8cnh+/fZmNjP/LfFJM5tqZiUzuyr/+gkzO094LVRrLj8k6Xvu3pxvc0kFjzEl28Jj83RJ15b9yNJTrbl0ScPzf4+Q9GKFji811ZrPPSXdlf/7T5JOqtDxbZV+ceZH0t6SHi6QWyLpre7elv+mf62kgySdIekOd7/UzEqSBkvaX9JEd99bksxsZB9jnmJmR0p6VtJ57l7pM02vd9Way93y790rqSTpi+7++608FlT3sSkza5K0k155ssWWq9ZcflHSnfkZvCGSjtnK40CmWvP5mKSTJX1H0rskDTOzMe6+fCuPp6z6S/kpqk7Sd81sf0ldyl/wJD0o6SdmVifpZnefZWZzJe1sZv8p6TZJd25kvN9Kutbd283sw5J+JunoSh8EJJV/LmslTZY0TdIOkmaa2T7u3lLRo8AG5Z7PDU6TdIO7d1Vu19FLuefydElXufs3zewwSb8ws73dvbviRwKp/PP5qXy86ZJmSlqUj7tN6S9ve82WdGCB3HmSFkvaT1lzrZckd58p6Uhlk3CVmZ2Vv/2xn6QZks6VdEXvwdx9ubu3519eUXAfsGlVmUtJL0i6xd073P15ZWfyJm/doUDVm88NThNveZVLtebyHEnX52PcL2mgpIatORBIqt7r5ovufrK7T5H0+fy6lq09mHLrL+XnLkkDzOyfNlxhZvua2dReuRGSXsp/YzhT2dsbG06NL3b3HyubrAPMrEFSjbvfqOwDdwf03qiZbd/jyxMlPV3GY0pVVeZS0s3Kzvooz+8maW4ZjytV1ZpPmdnukkZJur/Mx5Sqas3lAklvycfYQ1n5WVrWI0tTtV43G8xsQ7e4SNJPynxcZdEv3vZydzezd0m63Mw+I6lN0jxJn+wV/b6kG83sLEm/l7Qmv36apAvNrEPSaklnSZoo6ae9Jqm3T5jZiZI6Ja2QNL1Mh5SsKs7lHZLeZmZPKTsFe+G29h50f1TF+ZSysz7XOX+mviyqOJcXSPpx/uFZlzSdOd16VZzPaZIuMzNX9rbXR8t0SGXF8hYAACAp/eVtLwAAgLKg/AAAgKRQfgAAQFIoPwAAICmUHwAAkBTKDwAASArlBwAAJIXyAwAAkvI/ufxFMvosDVkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x360 with 10 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "coef = clf.coef_.copy()\n",
    "plt.figure(figsize=(10, 5))\n",
    "scale = np.abs(coef).max()\n",
    "for i in range(10):\n",
    "    l1_plot = plt.subplot(2, 5, i + 1)\n",
    "    l1_plot.imshow(coef[i].reshape(8, 8), interpolation='nearest',\n",
    "                   cmap=plt.cm.RdBu, vmin=-scale, vmax=scale)\n",
    "    l1_plot.set_xticks(())\n",
    "    l1_plot.set_yticks(())\n",
    "    l1_plot.set_xlabel('Class %i' % i)\n",
    "plt.suptitle('Classification coefficient vectors for...')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PfYdXpde4bg0"
   },
   "source": [
    "### Feed-forward using digits dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "UDKn4WS636Bg",
    "outputId": "754a52a5-d33e-4a7f-9431-df84fb38b3d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/19\n",
      "----------\n",
      "train Loss: 2.2229 Acc: 20.93%\n",
      "val Loss: 2.1097 Acc: 35.64%\n",
      "\n",
      "Epoch 2/19\n",
      "----------\n",
      "train Loss: 1.1401 Acc: 74.66%\n",
      "val Loss: 0.8908 Acc: 74.55%\n",
      "\n",
      "Epoch 4/19\n",
      "----------\n",
      "train Loss: 0.3788 Acc: 88.69%\n",
      "val Loss: 0.4712 Acc: 84.91%\n",
      "\n",
      "Epoch 6/19\n",
      "----------\n",
      "train Loss: 0.1971 Acc: 94.39%\n",
      "val Loss: 0.5091 Acc: 83.27%\n",
      "\n",
      "Epoch 8/19\n",
      "----------\n",
      "train Loss: 0.1401 Acc: 96.07%\n",
      "val Loss: 0.4645 Acc: 87.64%\n",
      "\n",
      "Epoch 10/19\n",
      "----------\n",
      "train Loss: 0.1273 Acc: 95.99%\n",
      "val Loss: 0.3447 Acc: 90.18%\n",
      "\n",
      "Epoch 12/19\n",
      "----------\n",
      "train Loss: 0.0914 Acc: 97.67%\n",
      "val Loss: 0.3651 Acc: 89.27%\n",
      "\n",
      "Epoch 14/19\n",
      "----------\n",
      "train Loss: 0.0922 Acc: 97.11%\n",
      "val Loss: 0.3925 Acc: 87.64%\n",
      "\n",
      "Epoch 16/19\n",
      "----------\n",
      "train Loss: 0.0628 Acc: 98.08%\n",
      "val Loss: 0.3407 Acc: 90.18%\n",
      "\n",
      "Epoch 18/19\n",
      "----------\n",
      "train Loss: 0.0547 Acc: 98.64%\n",
      "val Loss: 0.3234 Acc: 90.36%\n",
      "\n",
      "Training complete in 0m 6s\n",
      "Best val Acc: 90.36%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmvklEQVR4nO3deZxT5dn/8c8NDKuAIojsi1pbcYUREZlxYaqAVgcfba1tXarVttLW+nNt+7i2dXvaqnWhbnW31ip1KQpurYgKDgoqbuDGIgICguwMXL8/rqSTySQzA5PkJJnv+/U6ryTnPkkuDpkrd+5zL8HMEBGRwtci6gBERCQzlNBFRIqEErqISJFQQhcRKRJK6CIiRaJVVG/ctWtX69+/f1RvLyJSkGbMmPGFmXVLVRZZQu/fvz9VVVVRvb2ISEEKIXyarkxNLiIiRUIJXUSkSCihi4gUCSV0EZEioYQuIlIklNBFRIqEErqISJEouIQ+dy6cfTZs2hR1JCIi+aXgEvr778P118M990QdiYhIfim4hD5mDAwdCldcARs3Rh2NiEj+KLiEHgJcfjl8+inceWfU0YiI5I+CS+gAhx8Ow4fD734H69dHHY2ISH4oyIQegje5LFgAt98edTQiIvmhIBM6wKGHwsEHw+9/D+vWRR2NiEj0CjahhwCXXQaLFsH48VFHIyISvYJN6OA19JEj4aqrYM2aqKMREYlWQSd08B4vS5bAzTdHHYmISLQKPqEPHw6jRsHVV8NXX0UdjYhIdAovoX/xBVx6Kcye/d9dl10Gy5bBjTdGF5aISNQKL6GbeZ/Fv/3tv7uGDoWjjoJrr4VVqyKMTUQkQoWX0Lt1g/JymDCh1u7LLoMVK+C666IJS0QkaoWX0AHGjvUmlw8++O+uwYOhshL++EdP7CIizU1hJvTKSr9NUUtfuRL+9KfchyQiErXCTOh9+0JZWZ2q+N57w/HHe7PLsmXRhCYiEpXCTOgA//mPjyhKcsklsHo1/OEPEcQkIhKhBhN6CKFPCOGFEMI7IYTZIYRfpDgmhBBuCCHMDSG8GUIYnJ1wa72p327YUGv3oEFwwglwww2wdGnWoxARyRuNqaFXA//PzPYAhgFnhRD2SDpmNLBbbDsDuCWjUaZz4oneXzHJJZf4hF3XXJOTKERE8kKDCd3MFpnZ67H7XwHvAr2SDjsGuMfcq8D2IYQeGY82Wf/+8MILdRrMd98dvv99uOkm+PzzrEchIpIXtqoNPYTQH9gPmJZU1AuYn/B4AXWTfuYdeyxs3gxPPlmn6H//15eou/rqrEchIpIXGp3QQwjbAY8AZ5vZNo3HDCGcEUKoCiFULc1EA/eQIdCnDzz6aJ2iXXeFk0+GW26Bzz5r+luJiOS7RiX0EEIJnszvN7O62RMWAn0SHveO7avFzG41s1IzK+3Wrdu2xJscmA8ymjzZu7Yk+c1vvAL/+983/a1ERPJdY3q5BOAO4F0z+2Oawx4HTor1dhkGrDSzRRmMM71TT4Xrr6/p9ZJgwAA47TS47TaYNy8n0YiIRCaYWf0HhDACmAK8BWyJ7f4V0BfAzMbHkv6NwChgLXCqmVXV97qlpaVWVVXvIRkxbx7stpvnfa1sJCKFLoQww8xKU5W1aujJZvYSULf6W/sYA87atvAyYOlSnwbglFOgdetaRX37wo9+BH/5C1x4oXeMEREpRoU7UjTRtGlw5pnw/PMpiy+6CFq2hN/+NsdxiYjkUHEk9IoK2G67OpN1xfXqBT/+Mdx1F8ydm9vQRERypTgSetu2MGYMPPaYd2tJ4cILvTXmiityHJuISI4UR0IH7764eDG88krK4p13hrPOgvvug/ffz3FsIiI5UDwJfcwYaNMGpk9Pe8j550O7dnD55TmMS0QkR4onoXfq5ENCzzkn7SHdusHPfgYPPggff5zD2EREcqB4EjpAly4NHnLKKb7O9HPPZT8cEZFcKq6EvnmzL0935ZVpD/na12CnnWDKlNyFJSKSC8WV0Fu29GXpHnww7SEhwIgR8OKLOYxLRCQHiiuhg/d2eeutejucl5fDJ5/AggW5C0tEJNuKM6FD2kFG4OtLg5pdRKS4FF9C79cPBg+uN6Hvsw907KiELiLFpcHJuQrSuHHenmKWclrdli1h+HC1o4tIcSnOhH7qqQ0eUl4Ov/61L0e64445iElEJMuKr8klbt06mDo1bXG8Hb2eQ0RECkrxJvQrroBDDoHly1MW77+/T9aldnQRKRbFm9ArK6G6Gp58MmVx27YwdKja0UWkeBRvQi8t9YnQ6+ntUl4Or78Oa9bkMC4RkSwp3oTeooXX0idNgrVrUx5SVuaV+FdfzW1oIiLZULwJHeDYY/3iaJqZuIYP97yvdnQRKQbF2W0xrrwcXnsNhgxJWdypkw8yUju6iBSD4q6ht2rlbekpBhfFlZd7k8vGjTmMS0QkC4o7oQMsWQJnnJG2Gl5W5q0yr7+e47hERDKs+BN6x47wwANpp9QdMcJv1ewiIoWu+BN6u3YwahQ89hhs2VKnuHt32H13XRgVkcJX/AkdvLfLokUwbVrK4rIynwIgRb4XESkYzSOhH3kklJSkHWRUVuYLHc2eneO4REQyqHkk9M6d4cQTvZ9iCvGJutSOLiKFrLj7oSe66660Rf37Q+/e3o5+1lk5i0hEJKOaRw09bssW+PzzOrtD8Fr6lCm+JoaISCFqXgn9f/7He7ykUFYGn30GH3+c45hERDKkeSX0sjKYNStl1lY7uogUuuaV0MeO9dsUvV322AO6dFF/dBEpXM0roQ8YAPvuC48+WqeoRQsfNaqELiKFqnkldPBa+ssvw+LFdYrKymDOnJTXTUVE8l7zS+gnnwxPPOF905PE29FVSxeRQtT8Enq/fj5ytG3bOkWDB0P79kroIlKYGkzoIYQ7QwhLQghvpyk/JISwMoQwM7ZdnPkwM+zdd+Haa+t0Oi8pgQMPVEIXkcLUmBr6XUDqzts1ppjZvrHt8qaHlWVTpsD558MHH9QpivdsXLkygrhERJqgwYRuZi8Cy3MQS+5UVPjts8/WKSor84r71Kk5jklEpIky1YZ+YAhhVgjhqRDCoHQHhRDOCCFUhRCqli5dmqG33gYDB3oXxmeeqVM0bJivXKdmFxEpNJlI6K8D/cxsH+DPwD/THWhmt5pZqZmVduvWLQNv3QQVFfDCC1BdXWt3+/a+DKkSuogUmiYndDNbZWarY/cnAiUhhK5NjizbvvlN2LTJO54nKSuD117ztUZFRApFkxN6CGHnEEKI3R8ae81lTX3drDv6aF/V4hvfqFNUXg4bN8L06RHEJSKyjRqcDz2E8CBwCNA1hLAAuAQoATCz8cBxwE9CCNXAOuAEswKYhLZNm7RFBx3kU+pOmQIHH5zDmEREmqDBhG5m322g/EbgxoxFlEsTJ8Lll8Nzz0GHDv/dvcMOsOeeakcXkcLS/EaKJmrVyheOfumlOkVlZT7lS9I1UxGRvNW8E/qIEdC6dcr+6OXlsHo1zJyZ+7BERLZF807o7dvD8OHe5JJEE3WJSKFp3gkdvD/6G2/AF1/U2t2zp48/UkIXkUKhhD5qlM+RnmLyFi0cLSKFRAl9yBBfwWiXXeoUlZd7xf299yKIS0RkKymhxy1aVGeX2tFFpJAooQPcc483mn/0Ua3du+4K3bsroYtIYVBCBzjgAL9N6r4YgtfSX3wxgphERLaSEjrA174GvXun7Y8+b55vIiL5TAkdvCpeUeH90bdsqVWkdnQRKRRK6HEjR8Ly5XWGhu61F3TqpIQuIvlPCT3uiCPgr3+F/v1r7W7Z0mdfVDu6iOQ7JfS4bt3glFOgS5c6ReXl8O67dQaTiojkFSX0RIsWwfjxsH59rd3xdvQUkzKKiOQNJfREM2bAT37i8+YmKC319TDUji4i+UwJPdHBB3ujeVL3xTZtYNgwtaOLSH5TQk/UsaNn7hT90cvKfFLG1asjiEtEpBGU0JNVVHjTy4oVtXaXlcHmzfDKKxHFJSLSACX0ZBUVPl9uVVWt3QceCC1aqB1dRPJXg4tENzvDhsGSJdC1a63dHTvC4MFqRxeR/KUaerJWreok87iyMl9TesOGHMckItIISuipzJgBo0fD/Pm1dpeVeRf1GTMiiktEpB5K6Km0bg1PP11n8egRI/xWzS4iko+U0FPZc0/Yaac63Re7dYNvfEMXRkUkPymhpxKfTvfZZ+usEF1WBlOnehdGEZF8ooSeTkUFLF4Ms2fX2l1eDitXwqxZEcUlIpKGEno6I0f6vLlffVVr9+GHe3/0xx+PKC4RkTSU0NPp29enVzzwwFq7u3XzPP/Pf0YTlohIOkroDVmzBqqra+2qrPQml48/jiYkEZFUlNDr85//wA47wKuv1tpdWem3qqWLSD5RQq/P3nt7d5ak7osDB3qRErqI5BMl9PrssAMMGVJngBF4Lf2ll2Dp0tyHJSKSihJ6QyoqvMklqbfL2LGwZQs88UREcYmIJFFCb0hFhV8UTRrvv88+0K+fml1EJH8ooTdk+HC45hqfDiBBCN7sMnmyVjESkfzQYEIPIdwZQlgSQng7TXkIIdwQQpgbQngzhDA482FGqG1bOO88r44nGTvWp9KdNCmCuEREkjSmhn4XMKqe8tHAbrHtDOCWpoeVZ776Ch55BL74otbugw6CHXeECRMiiktEJEGDCd3MXgSW13PIMcA95l4Ftg8h9MhUgHlhzhw47rg6VfFWreBb34Inn4RNmyKKTUQkJhNt6L2AxJUgFsT21RFCOCOEUBVCqFpaSP399t0XunSp0x8dvNll5UofgyQiEqWcXhQ1s1vNrNTMSrt165bLt26aFi18sq4U0+l+85vQvr2aXUQkeplI6AuBPgmPe8f2FZeKCliwAD74oNbudu1g1Ch47DHvly4iEpVMJPTHgZNivV2GASvNbFEGXje/jBzptymWK6qshIULtdaoiESrVUMHhBAeBA4BuoYQFgCXACUAZjYemAiMAeYCa4FTsxVspAYO9Nr5rrvWKTrySGjZ0ptd9t8/gthERIBgSW3CuVJaWmpVVVWRvHc2VFTAZ5/BO+9EHYmIFLMQwgwzK01VppGiW+OTT+Ckk+CNN+oUVVbCu+/C++/nPCoREUAJfet06AD33gtPPVWn6Jhj/FZzu4hIVJTQt0a3bj4rV4r+6H36QGmpErqIREcJfWtVVMDUqb40XZLKSp9p97PPch+WiIgS+tY65hjYuNE7nyd1PI8vTff447kPS0RECX1rlZXBo4/CKaf4CNIEe+wBu+2mZhcRiYYS+rYYOxZOO83vP/wwnH8+bNr03znSn3/e53cREcklJfSmeu01uPZaOOwwWLiQsWN95sWJE6MOTESaGyX0prrmGrj/fu+bvu++HLDqGbp3V7OLiOSeEnomnHgiVFVB9+60GH0EPx3xJhMnwvr1UQcmIs2JEnqmfP3rMG0a/PWvDD19b1avhheeqY46KhFpRpTQM6lDBzj5ZA49FIa1f5Mh39tdK1+ISM4ooWdBmzZwUFkLVq1thR12GFx5pSZLF5GsU0LPkv1P3ZP9NlfxxaHfhl/9Co46CpYtizosESliSuhZMno0bCjpyNX7PAA33wzPPQc33RR1WCJSxBpc4EK2TadOvsjRPx8LXDvnJ4QRI/zCKfhkLz16QAjRBikiRUU19CwaOxY+/BBmzwb22gtKSuDLL+GAA3waXhGRDFJCz6Kjj/ZK+IQJCTs7dYLeveHcc2H58shiE5Hio4SeRTvvDMOGJY0abdECxo/3ZH7RRVGFJiJFSAk9y8aOhddfh3nzEnbusw+cfTbceiu8/HJUoYlIkVFCz7L4HOl15na59FJvernvvtwGJCJFSwk9y3bbDQYNSpHQt9sOXnlFXRlFJGOU0HOgshJefDHFuKLevf2q6cKFWrdORJpMCT0HKith82Z48skUhevWwZAhMG5crsMSkSKjhJ4DQ4Z4ZTzlHOnt2vkF0gkT0mR8EZHGUULPgfjSdJMmwdq1KQ445xxfkHTcuDQHiIg0TAk9RyorvXVl8uQUha1bwy23wKefwhVX5Do0ESkSSug5Ul4OO+xQz9J05eXwwx/6lVOzXIYmIkVCk3PlSEmJz6D7xBNQXQ2tUp35W2+Fli1zHpuIFAfV0HOostJH/E+ZkuaAeDJ/4w34179yFZaIFAkl9Bw64gho27aeZpe4X/4STj5ZC2KIyFZRQs+hDh3g8MM9odfbTP7nP/s0uxdckKPIRKQYKKHnWGWlT9T16KP1HLTXXt6V8Y47YOrUXIUmIgVOCT3HTjgB9t8ffvADmDatngMvuQT69oUf/xg2bcpZfCJSuJTQc6xdO+/p0qOH93qZOzfNgR06+MRdxx+vbowi0ihK6BHo3h2eesrz9OjRsHRpmgOPOgouvtgHHomINEAJPSJf+5rX1BcsgG99q4ER///6F/z0pzmLTUQKU6MSeghhVAjh/RDC3BDChSnKTwkhLA0hzIxtp2c+1OJz4IHwwAMwfTqceKLPyJjSO+/41ACPPZbT+ESksDSY0EMILYGbgNHAHsB3Qwh7pDj0ITPbN7bdnuE4i9bYsXDDDZ6rf/7zNM3lZ58Ne+4JP/sZrF6d6xBhxQrvbTNzZs2+v/zFv2jUvi+SNxpTQx8KzDWzj8xsI/A34JjshtW8jBsH550HN98M116b4oCSEl9Yev58uPzy7AWyfn3N/fPOg4oK6NkTunSBESPg//7PyxYs8N43gwbBLrv4N9Ezz8CGDdmLTUQa1JiE3guYn/B4QWxfsv8JIbwZQvhHCKFPqhcKIZwRQqgKIVQtTXslsHm66irv0njBBfDggykOOOggOP10+OMf4YMPmv6G06b5N8hZZ8Ghh/qV2lGjaspffhm++sqHt15zjbfjX3WVl/Xu7V8u48d7Ur/tNh8x9fe/e/mKFbB4cdNjFJGtEqyBn8whhOOAUWZ2euzxD4ADzGxcwjE7AqvNbEMI4UzgO2Z2WH2vW1paalVVVU3+BxSTDRs8f778sk+ze8ghSQcsW+bdY773PZgxA95+25NufNu0qaYW/dvf+oIZ8bLVq6FTJ/jkEy8fM8Zfq1Mnn4t90CA44AD40Y+2PvC1a+H552H4cK/NX3+9T18wdKj31PnWt2DvvX1ieBFpkhDCDDMrTVnWiIR+IHCpmR0Re3wRgJldmeb4lsByM+tc3+sqoae2YoW3bixc6M3WgwalOfDnP/cpAuJKSmD77b1mHAL8/vfw739Dx441W7du3g0SYM4caN/em1QynWjff99r608+6Vd8Afr1g/fe88ls4p+5DRv8i2bNGmjRAvrEfthNnuxfXmvW+LZ6Ney6K3znO15+5pn+i+XYY32xbZFmpL6EjpnVu+FT7H4EDABaA7OAQUnH9Ei4PxZ4taHXHTJkiElqn35q1qOHWZ8+ZgsWpDlo0SKzjz4yW7rUbP36nMa3VRYtMrvjDrOLLqrZN2qUWYsWZp7afRs5sqZ84MDaZWBWWVlTPmiQ72vf3uz73zebPNmsujp3/yaRCAFVliavNlhDj30jjAGuA1oCd5rZ70IIl8de+PEQwpXA0UA1sBz4iZm9V99rqoZev5kzoawMBg706XY7dYo6ogwxg8su80nhO3TwGvZ22/k0ByNH+jGzZ/tUwvHyDh1qD64yg1degXvugYce8onMbrzRrweYqWknnQ8/9Gswp54KpaXwxRfw0Ud+v4WGpBSKJtXQs7Wpht6wSZPMWrUyq6gw27Ah6mjy1Lp1Zg8/bLZ4sT++6y6z/fYz+9OfzD7/PNLQ8kZVldm3v+2/ilq3Nhs/3vffeKP/0unZ0+zMM80mTszvX3tiZvXX0PW1nMcOP9w7kDz7rF+rVJfvFNq2heOOg5128sedO3tt85e/hF69/KLsQw8135N3/PFeA3/6aTj3XPj4Y78GAfDd7/qvnAMPhPvu8wvlO+3kF9EBtmyJLm7ZJlqCLs+dcor3ELz4Yr+umM1u6EWhstK32bPh3ns9US1cWHNBdc4cv8AaVbPMhg3eXNS3LwwYkPk4qqth4kT/ImvRAoYN8+k9zzzTv+wSdeni037+4Ac+BuGFF2DWLL+ADnDMMX5B+phjfBswILOxSualq7pne1OTS+Nt2WJ2+un+6/jWW6OOpsBUV9dcWV61yi+kDhhgdt55ZlOnmm3enP0Ytmzx9rNTTzXr3LnmQu+4cTXl06ebbdy47e+xdq3ZTTfVXFCeNKnpcV92Wc0FaDDbay99AJvq7bfN/va3Jr0ETb0omg26KLp1qqvh6KO9R9/jj/uvY9lK69fDww97rf355/2kdu/ui3MffXRm32vLFv81sPvu/ni//fwC5Nixvn32GXz96z6o64MP/Lh27bzv/vDh3i1zxIi6tepka9fCH/7gXViXLvUa+QUX+L8nUxc6P/zQP3SPPebTg15wgV9QHTrURwrvsov/6tllFx/L0LNnZt63WGzc6Nt228GECT6Vx0cfbfOC8LooWiS++sps8GCzDh3M7r3XK3ayjVasMHvgAb9YOGuW75s0yR8/8IDZl19u/Wtu2eIXIM891/uctm9vtnq1l82Z4xdwU1m50uzvfzf7xS/MSkvNWrb0GvEjj9Q89667/Db+nx5/rY0bzfr1MzvySLMXX8z+hyL++gsWmJ14otnQoWY77lhTi7/9di9/6y2zESPMTj7Z7Ior/JxOn262Zk1248snn3/uv3J23tnskkt836ZN/v/dBNRTQ1dCLzCLFpkNG+b/c2PGmM2bF3VEReTuu8122slPbkmJ2RFHmN1yS+O6GE2ebLbbbjXPPeoos/vuS5/E67N6tdnzz/uXjpn32IknzJ12MjvkELO+fWt6pDQxQWTEihX+ZRbvbTR9ullZmfegSRxPEG8Keukls1NOMbvySrMJE8zefbdpTU75ZPp0Hx9RUuL/5lGjzJ57LmMvr4ReZKqrza67ziuAHTua3XxzbpqCm4Xqam9bP+88s113NevVq+bkPv20Jx4zH9R15ZWemMzMZs70wVG33Wa2bFlmY9q82dte//IXs5NOMtt7b7Nf/nLbfkVEYc0ar7FPmFBzbh580EfPJSb7Vq3MPvjAy195xQekTZ2a+fOZDYkD2yor/Q/zZz8ze//9jL9VfQldbegF7OOP4YwzvFtjWRncfrsvnCEZYubt0jvt5Pd79/a27549/RZ8kFR8OgXZeqtW+VQR773n28UXQ5s23u30uutqjuvWza8zPPecDzJ76SX/P2jf3geetW/vo+++8Q0/fv16nw5jG9upG23RIp9K+tZb4T//gd1281Xgt98+a6MBmzSXS7YooWeGGdx1F5xzjn+GL7vM77dSh9TMmz/fLw7++98wZIhPj9m/f9RRFafqap9I7r33ahL+kiU1i7x85zs1s3vG9ezpXVQBjjzSu2+2aVOT8AcN8v74ABdd5DWizp1rtoEDvW8+wOuv+23nzp6YO3euPVp52jRfyODhh31SvDFj4Oqrfd2CLFNCbwYWLfKR7xMmwODBcOedsM8+UUclkiWff14zgdvatX4bQk33r4ce8i+BeNnatV7Lv/pqLz/pJHj1VVi50n8lrF/vPYteesnL99gD3n239nuOHu1fEitW+Crvbdr4NApnneU18xxRQm9GHnnEP1/Llnnvst/8xgdTikg9Nm70pB5vJnnlFf9FsHJlzda3r38RgDf9DB1aMwgrh5TQm5nly73Z5e67vavzHXd412YRKXz1JXTN5VKEunTxdvWnn/ZfmiNG+PTpUSxHKiK5o4RexI44whc1GjfOZ5fdc08faSoixUkJvch17OgX46dM8bb0I47wCb8ysSypiOQXJfRm4qCDfNGMX/8a7r/fu/SOGOG9YeKzpYpIYVNCb0batvW1o+fPh2uu8Z4wp53mPbB++EOvxUd0jVxEMkAJvRnaeWc47zx45x14+WUfS/Hww1Be7jX3K6+sGZ8hIoVDCb0ZC8EXq7ntNh+ncffdPtjuV7/yLrdjxsA//uFrMohI/lNCF8BHR590ko9qnzPHR0a/+aavYNarl0/hPGtW1FGKSH00sEjS2rwZnnnGL5w+9pgPphs82BP/wIG1p8GIT3mR7bmQRJq7+gYWaQonSatlSxg1yrdly+CBB3zU6dlnp3/OdtvVzGWUbuvSxRfW2XffzC2qIyKqocs2+OQTn1U2cZqLVatqP07e4uXr1tW8zo47wmGHwciRUFHhtf6o1m4WKRSqoUtG9e+/7bPGbtwIixfDiy/6PO7PPus9bOKvW1Hh22GH+eR4ItJ4qqFLpMx81Opzz3lyf/55r8mDN8nEE/yIEX7hVqS502yLUjCqq31tgXjtfepUr9W3bu1dLCsq4OCD/XF8quvEKbET76cr37DBa/99+0KfPr4l3s/SQjMiGaGELgVr7VpfcyBeg3/jjcaNZm3Xzmv08cVqEm9bt/aprufN84VBtmyp/dzOnWuSe3Ky79vXV6Jr0yY7/16RhqgNXQpW+/Zw+OG+AXzxha/+FULdRB2/365d43vPVFf70pTz5/s2b17N/fnz4bXX/D2Tde7sF3W7dvUt8X7y4x139K2kJHPnRSQVJXQpKF27+nKRmdKqlde6+/ZNf8y6dbBgQU3CX7DAe/ksW+bJfvFimD3bH9c353znzjVJvk8fGDCg5gLzgAHQr59/IYlsKyV0kQa0a+dLRjZm2cj16z2xx5N9fEt8vGQJvPUWPPFE3WkVunevSfCJyb5/f0/4auqR+iihi2RQ27Y+VUKvXg0fu2WL1+4//tj79ifevvaarw+7aVPt5/Ts6Yl9++29Nh9vYorfT7cv8XG7dv5FEr9I3Jgt8aLyunU+OKxXL7+e0Lt37ftdu2o8QVSU0EUi0qKFT13co0fqNV83b/b2/eRk/+mn3uSzbp0n2sRt8+bMxJZ4jSJxa9/em46WLfML1Z99VveicuvW6ZN9/Mtuxx39iyVXid/MB7etWOHdYjt39jiK7bqGErpInmrZsqZ3TVlZ456zaVNNck9O+PHH69Z50k1O1okXmBubbKur/VfGwoV+bSF+G78/fbrfrl9f97mtWtU/RUS6rVMn/zesWJF6+/LL1PuSv3hC8Kmk+/TxL5v4uU6836OHx1ko1G1RRLLKDJYvr53w4zXlVNuXX9ZMF7E16amkBHbYIfW2/fY19zt39veI92SKX/CeP7/uRe34r6jkZN+5c00zVrt2qbd4WUlJZn+JqNuiiEQmhJqum/vs0/jnbdniCTbVvEDt2tVN2u3bNy1xxptlErutJib7N9+Ef/3Lf+VsjRYt6ib7M8+Ec87Z9ljTUUIXkbzUooU3r3Tq5LXibAuhpllnzz1TH2PmtftVq7zZJ77Fm7Lq2xKP6d49O/+GRiX0EMIo4HqgJXC7mV2VVN4GuAcYAiwDvmNmn2Q2VBGRaIVQ84sgHzU4ni6E0BK4CRgN7AF8N4SwR9JhpwErzGxX4E/A1ZkOVERE6teYAdJDgblm9pGZbQT+BhyTdMwxwN2x+/8ARoagnqgiIrnUmITeC5if8HhBbF/KY8ysGlgJ7Jj8QiGEM0IIVSGEqqVLl25bxCIiklJOFwAzs1vNrNTMSrtp9QIRkYxqTEJfCCReY+4d25fymBBCK6AzfnFURERypDEJ/TVgtxDCgBBCa+AE4PGkYx4HTo7dPw543qIasSQi0kw12G3RzKpDCOOASXi3xTvNbHYI4XKgysweB+4A7g0hzAWW40lfRERyqFH90M1sIjAxad/FCffXA8dnNjQREdkakc3lEkJYCny6jU/vCqRYRyZv5Ht8kP8xKr6mUXxNk8/x9TOzlL1KIkvoTRFCqEo3OU0+yPf4IP9jVHxNo/iaJt/jSyen3RZFRCR7lNBFRIpEoSb0W6MOoAH5Hh/kf4yKr2kUX9Pke3wpFWQbuoiI1FWoNXQREUmihC4iUiTyOqGHEEaFEN4PIcwNIVyYorxNCOGhWPm0EEL/HMbWJ4TwQgjhnRDC7BDCL1Icc0gIYWUIYWZsuzjVa2Uxxk9CCG/F3rvOAq7B3RA7f2+GEAbnMLbdE87LzBDCqhDC2UnH5Pz8hRDuDCEsCSG8nbCvSwjhmRDCnNhtyuUNQggnx46ZE0I4OdUxWYrv2hDCe7H/wwkhhO3TPLfez0MW47s0hLAw4f9xTJrn1vv3nsX4HkqI7ZMQwsw0z836+WsyM8vLDZ9m4ENgINAamAXskXTMT4HxsfsnAA/lML4ewODY/Y7AByniOwR4MsJz+AnQtZ7yMcBTQACGAdMi/L/+HB8wEen5A8qBwcDbCfuuAS6M3b8QuDrF87oAH8Vud4jd3yFH8R0OtIrdvzpVfI35PGQxvkuBcxvxGaj37z1b8SWV/wG4OKrz19Qtn2voeb2whpktMrPXY/e/At6l7jzx+e4Y4B5zrwLbhxB6RBDHSOBDM9vWkcMZY2Yv4vMRJUr8nN0NVKZ46hHAM2a23MxWAM8Ao3IRn5lNNl+HAOBVfEbUSKQ5f43RmL/3Jqsvvlju+DbwYKbfN1fyOaFnbGGNbIs19ewHTEtRfGAIYVYI4akQwqDcRoYBk0MIM0IIZ6Qob8w5zoUTSP9HFOX5i+tuZoti9z8HUi3xmy/n8of4r65UGvo8ZNO4WJPQnWmarPLh/JUBi81sTpryKM9fo+RzQi8IIYTtgEeAs81sVVLx63gzwj7An4F/5ji8EWY2GF8P9qwQQnmO379BwadkPhp4OEVx1OevDvPf3nnZ1zeE8GugGrg/zSFRfR5uAXYB9gUW4c0a+ei71F87z/u/p3xO6Hm/sEYIoQRP5veb2aPJ5Wa2ysxWx+5PBEpCCF1zFZ+ZLYzdLgEm4D9rEzXmHGfbaOB1M1ucXBD1+UuwON4UFbtdkuKYSM9lCOEU4Cjge7EvnToa8XnICjNbbGabzWwLcFua9436/LUCjgUeSndMVOdva+RzQs/rhTVi7W13AO+a2R/THLNzvE0/hDAUP985+cIJIXQIIXSM38cvnL2ddNjjwEmx3i7DgJUJTQu5krZWFOX5S5L4OTsZeCzFMZOAw0MIO8SaFA6P7cu6EMIo4HzgaDNbm+aYxnweshVf4nWZsWnetzF/79lUAbxnZgtSFUZ5/rZK1Fdl69vwXhgf4Fe/fx3bdzn+wQVoi/9UnwtMBwbmMLYR+E/vN4GZsW0M8GPgx7FjxgGz8Sv2rwLDcxjfwNj7zorFED9/ifEF4KbY+X0LKM3x/28HPEF3TtgX6fnDv1wWAZvwdtzT8OsyzwFzgGeBLrFjS4HbE577w9hncS5wag7jm4u3P8c/h/GeXz2BifV9HnIU372xz9ebeJLukRxf7HGdv/dcxBfbf1f8c5dwbM7PX1M3Df0XESkS+dzkIiIiW0EJXUSkSCihi4gUCSV0EZEioYQuIlIklNBFRIqEErqISJH4/wyOiz9DpF6dAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = MLP([D_in, 512, 256, 128, 64, D_out]).to(device)\n",
    "\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "model, losses, accuracies = train_val_model(model, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=20, log_interval=2)\n",
    "\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 724
    },
    "id": "R9ImEBeo4MW6",
    "outputId": "c402c8da-dd25-4212-86f6-d3768e1619ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/49\n",
      "----------\n",
      "train Loss: 2.4328 Acc: 16.28%\n",
      "val Loss: 2.0682 Acc: 44.91%\n",
      "\n",
      "Epoch 10/49\n",
      "----------\n",
      "train Loss: 0.4297 Acc: 85.89%\n",
      "val Loss: 0.3002 Acc: 89.82%\n",
      "\n",
      "Epoch 20/49\n",
      "----------\n",
      "train Loss: 0.2546 Acc: 91.98%\n",
      "val Loss: 0.2488 Acc: 92.73%\n",
      "\n",
      "Epoch 30/49\n",
      "----------\n",
      "train Loss: 0.1890 Acc: 93.42%\n",
      "val Loss: 0.2325 Acc: 93.64%\n",
      "\n",
      "Epoch 40/49\n",
      "----------\n",
      "train Loss: 0.1606 Acc: 93.99%\n",
      "val Loss: 0.2394 Acc: 93.64%\n",
      "\n",
      "Training complete in 0m 20s\n",
      "Best val Acc: 94.55%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlb0lEQVR4nO3dd3xW5fk/8M+VQWSHEVYgSyIiZQQCP6y2IrYVRXHUgRVx8cVSFa3aFttaV33ZWlt3SyngqkWRIjiwgoKKiEDYG8MIElYYIUAg8/r9cT0PT/Z8Rs55Pu/X67yS55xnXCeEK/e57+u+j6gqiIjI+SJCHQAREfkHEzoRkUswoRMRuQQTOhGRSzChExG5RFSoPrhjx46alJQUqo8nInKklStXHlLVuKqO1ZrQRaQHgDcAdAagAKao6gsVnjMMwFwAOz27ZqvqEzW9b1JSEjIyMmoNnoiIfEQkq7pjdWmhFwN4UFVXiUhrACtFZIGqbqrwvMWqekVjAiUiooartQ9dVfep6irP98cBbAYQH+jAiIiofuo1KCoiSQDSACyr4vD5IrJWRD4WkT7VvH68iGSISEZOTk79oyUiomrVOaGLSCsA/wVwv6rmVTi8CkCiqvYH8BKAOVW9h6pOUdV0VU2Pi6uyT5+IiBqoTgldRKJhyfwtVZ1d8biq5qnqCc/38wBEi0hHv0ZKREQ1qjWhi4gAmAZgs6r+rZrndPE8DyIyxPO+h/0ZKBER1awuVS4XALgFwHoRWePZ91sACQCgqpMBXAdggogUAzgFYLRyGUcioqCqNaGr6lcApJbnvAzgZX8FVZP164EZM4Bf/xqIjQ3GJxIROYPjpv7v2AE8/TSQmRnqSIiImhbHJXTvagE7d9b4NCKisOO4hJ6cbF+Z0ImIynNcQm/TBmjfHti1K9SREBE1LY5L6IB1u7CFTkRUniMTenIyEzoRUUWOTehZWQAr3YmIfByZ0JOSgNOngf37Qx0JEVHT4ciEzkoXIqLKHJnQvbXorHQhIvJxdEJnC52IyMeRCb1FC6BzZ7bQiYjKcmRCB1iLTkRUkWMTOmvRiYjKc3RC370bKCkJdSRERE2DYxN6UhJQXAxkZ4c6EiKipsGxCZ216ERE5Tk+obPShYjIODah9+gBiLCFTkTk5diEHhMDxMezhU5E5OXYhA6wFp2IqCxHJ3TWohMR+Tg+oWdnA4WFoY6EiCj0HJ3Qk5KA0lLgu+9CHQkRUeg5OqGzFp2IyMcVCZ2VLkREDk/o8fFAZCRb6EREgMMTelQUkJDAFjoREeDwhA6wdJGIyMvxCZ2Ti4iIjOMTenIysH8/cOpUqCMhIgot5yX03Fxg0SLg2DEAvhtGZ2WFLCIioibBeQl95Upg+HBgzRoArEUnIvJyXkLv1s2+7t0LgLXoRERetSZ0EekhIotEZJOIbBSR+6p4jojIiyKSKSLrRGRgYMKFL6F77j3XpYstpcsWOhGFu6g6PKcYwIOqukpEWgNYKSILVHVTmedcBiDVs/0/AP/wfPW/Nm2Ali3PtNAjIoDERLbQiYhqbaGr6j5VXeX5/jiAzQDiKzztKgBvqPkGQKyIdPV7tIDdpqhbtzMJHWAtOhERULcW+hkikgQgDcCyCofiAZRd83CPZ9++xgRXrX/9C+jQ4czDpCQgIyMgn0RE5Bh1Tugi0grAfwHcr6p5DfkwERkPYDwAJCQkNOQtzEUXlXuYnAwcPgwcPw60bt3wtyUicrI6VbmISDQsmb+lqrOreEo2gB5lHnf37CtHVaeoarqqpsfFxTUkXrNtG/Dmm4AqAFa6EBEBdatyEQDTAGxW1b9V87T3AYz1VLsMBXBMVQPT3QIAH34IjB1rk4zgm1zEfnQiCmd16XK5AMAtANaLyBrPvt8CSAAAVZ0MYB6AywFkAsgHcLvfIy0r3jMmu3cv0K4dW+hERKhDQlfVrwBILc9RAHf7K6halZ1c1KcPOnYEWrRgC52IwpvzZooClWaLilg/OlvoRBTOnJ3Qs33jrqxFJ6JwV6869CajeXNg+XIgJeXMrqQk4MsvrfBFauwgIiJyJ2cmdAAYPLjcw+RkIC/PCl/atQtNSEREoeTMLhcAmD8fePXVMw+5jC4RhTvnJvT//Ad49NEzD1mLTkThzrkJvVs3YN8+oLQUgK87ffv2EMZERBRCzk7oxcVATg4AoG1bIC4OyMwMcVxERCHi3IRedraoR2oq8O23IYqHiCjEnJvQK0wuAoCePdlCJ6Lw5dyEPmCATSwaMeLMrtRUYM8eID8/dGEREYWKcxN6TIy10iMjz+xKTbWvHBglonDk3IQOAC+8YOWLHj172ld2uxBROHJ2Qn/tNWDGjDMPvQmdA6NEFI6cndC7dSu3QBdLF4konDk/oZepcgFYukhE4cv5Cf3gQaCo6Myunj2Z0IkoPDk/oUdEnJktClgLPTubpYtEFH6cndBvvx0oKPBNMgJLF4kofDk7oTdrVq4OHWDpIhGFL2cn9Px84K67gHnzzuxi6SIRhStnJ/SYGGDaNODrr8/s8pYuMqETUbhxdkKPjAS6di1Xiw5YPzq7XIgo3Dg7oQOsRSci8nBlQu/Zk6WLRBR+nJ/QExOtFr0Mli4SUTiKCnUAjfb885V2la106ds3uOEQEYWK81voVWAtOhGFI+cn9E2bgJEjgdWrz+xi6SIRhSPnJ/TSUptYVCF7s3SRiMKN8xN6FTeLBli6SEThx/kJvV07mzHK0kUiCnPOT+gi1U4uAli6SEThw/kJHQDS062lXgYX6SKicOP8OnQAmDmz0i6WLhJRuKm1hS4i00XkoIhsqOb4MBE5JiJrPNsf/B9m/bF0kYjCTV26XF4DMKKW5yxW1QGe7YnGh1VPs2YB/foBeXnldrPShYjCSa0JXVW/BHAkCLE0XEEBsH49sG9fud2sRSeicOKvQdHzRWStiHwsIn2qe5KIjBeRDBHJyClzY+dGq6YWnaWLRBRO/JHQVwFIVNX+AF4CMKe6J6rqFFVNV9X0uLg4P3y0Rw2TiwCWLhJReGh0QlfVPFU94fl+HoBoEenY6Mjqo4YWOsB+dCIKD41O6CLSRUTE8/0Qz3sebuz71kvr1sCPf2xlLWV4W+hM6EQUDmqtQxeRGQCGAegoInsAPAogGgBUdTKA6wBMEJFiAKcAjFZVDVjE1Zk/v9KuNm2ATp04MEpE4aHWhK6qN9Vy/GUAL/stIj/r2ZMtdCIKD+6Y+g8AkyYBaWmVdrN0kYjChXsSuqrd7KJCbw9LF4koXLgnoXfrBhQWAkfKz4HyDoyylU5EbueuhA5UW7rIhE5Ebuf6hM7SRSIKF+5J6ElJwOjRQPv25Xa3aQN07gxs2RKasIiIgsUd66EDQHw8MGNGlYf69wfWrAluOEREweaeFrpXUVGlXQMHAhs22KKMRERu5a6E/qMfAVdcUWl3WhpQXAxs3BiCmIiIgsRdCT02Fti9u9LugQPt6+rVwQ2HiCiY3JXQExKA776rNLkoJcXW71q1KkRxEREFgbsSeo8ewMmTwNGj5XZHRFi3C1voRORm7kvogLXSK0hLA9auBUpKghwTEVGQuCuh9+8P/OpXQNu2lQ4NHGjruWzbFoK4iIiCwF0JPTUVeOYZm2RUgXchRvajE5FbuSuhA8CJE8ChQ5V2n3suEBPDfnQici/3JfS+fYH776+0Ozoa6NePLXQici/3JfQePaqsRQd8lS4huEEeEVHAuTOhV1HlAtjAaG4ukJUV3JCIiILBfQk9IcFuUVRaWukQB0aJyM3cl9B79LAFug4cqHSob18gMpIDo0TkTu5L6MOGAS++CDRvXulQ8+ZA795soRORO7lnPXSv886zrRoDBwILFgQxHiKiIHFfC10V2Ly5xkqXffuA/fuDHBcRUYC5L6EDwODBwPPPV3mIS+kSkVu5L6GL1FiLPmCAfWU/OhG5jfsSOlBjLXqbNkDPnmyhE5H7hF1CB7g2OhG5k3sT+v79QGFhlYcHDgR27LBZo0REbuHOhH799cDs2dUe9s4YXbMmOOEQEQWDOxN6nz7A1VcDzZpVeZhLABCRG7kzoRcWAvPnA5mZVR7u1AmIj2c/OhG5izsTelERcOmlwLvvVvuUtDS20InIXdyZ0Fu2BNq3r7HSZeBAYMsWu88oEZEb1JrQRWS6iBwUkQ3VHBcReVFEMkVknYgM9H+YDVDD5CLAWuilpcC6dUGMiYgogOrSQn8NwIgajl8GINWzjQfwj8aH5QcJCbW20AH2oxORe9Sa0FX1SwBHanjKVQDeUPMNgFgR6eqvABuslslFPXpYrwz70YnILfzRhx4PoGzm3OPZV4mIjBeRDBHJyMnJ8cNH1+C++6zSpZobiIoAgwYBX33Fe4wSkTsEdVBUVaeoarqqpsfFxQX2w845B0hPt8xdjRtvtIHRJUsCGwoRUTD4I6FnA+hR5nF3z77Qys0Fpk+vthYdAEaPBtq2Bf7RNHr9iYgaxR8J/X0AYz3VLkMBHFPVfX5438bJzQXuvBP44otqn9KyJXDrrcCsWUCge4CIiAKtLmWLMwAsBdBLRPaIyJ0i8nMR+bnnKfMA7ACQCeBfAH4RsGjrIz7eultqKF0EgJ//3CaWTp8epLiIiAKk1nuKqupNtRxXAHf7LSJ/iY4GunatsdIFsJtGDxsG/POfwEMPAZGRwQmPiMjf3DlT1KuW0kWvCROAnTuBTz4JQkxERAHi/oReS5cLYAszdu7MwVEicjZ3J/TnngMWL671ac2aAePGAR99BGRlBSEuIqIAcHdC797d1sqtg/HjbQx1ypQAx0REFCDuTuhZWcDjj9ep2Z2QAIwcCUydWu2d64iImjR3J/SDB4HHHgPWrq3T0ydMsJe8915gwyIiCgR3J/Qengmsdah0AeyeGMnJHBwlImdyd0Lv1Mnq0etQ6QIAERHAXXfZ5NJNmwIcGxGRn7k7oUdE1LkW3euOO6zqZfLkAMZFRBQA7k7ogCX0vXvr/PS4OOC664DXXweO1LQKPBFRE+P+hP7++8DChfV6yaRJwIkTViBDROQU7k/obdpY10s99O1rdemvvAJs3hyguIiI/Mz9CX3lSpsGevBgvV72xBO2vO5DDwUoLiIiP3N/Qj9wAJg2Ddixo14vi4sDHnkEmDcP+N//AhQbEZEfuT+he2vR61i6WNa99wJnnw08+CBQXOznuIiI/Cx8Eno9She9YmKAZ5+1mnSu8UJETZ37E3rbtkDr1vXucvG66irg4ouBP/wBOHrUz7EREfmR+xO6CHDBBcDhww1++XPPWU36k0/6OTYiIj9yf0IHrBb97bcb/PL+/a1Q5qWXgG3b/BgXEZEfhUdCj462r6WlDX6LJ58EmjdnGSMRNV3hkdABm/554YUNfnnnzsDvfw988AGwaJEf4yIi8pPwSegdOwJLlwK7djX4LSZOBOLjLbGr+i80IiJ/CJ+Efs019nXOnAa/xVln2WSjr7/mZCMianrCJ6GffbYt0tLI2xHdfrvdBIOtdCJqasInoQPWSv/qq3qv61JWs2bAo48Cq1Y1qrFPROR34ZXQR4+2NXHrufpiRTffDPTqZd0vJSV+io2IqJHCK6H37m19JR07NuptoqLs78LGjcDMmX6KjYiokcIroQNAfr71ox8/3qi3uf5665J/9FEu3EVETUP4JfSVK4FrrwU+/rhRbxMRYWumf/st8OabfoqNiKgRwi+hf//7tth5I6tdAFu4a9Ag634pLPRDbEREjRB+CT0y0jLxRx8BBQWNeisR4I9/BLKy7B4aREShFH4JHbDyxePHgc8+a/RbXXqpLeb4xz8Cp075ITYiogYKz4R+ySW2RrofFmXxttL37gW6dwcuu8zWTv/wQ7v7HRFRsESFOoCQiIkB1q8HEhL88nbDhgGzZtk464oVwFNP+RZ2TEwE/u//gIcfbnT5OxFRjeqUYkRkhIhsFZFMEZlUxfHbRCRHRNZ4tnH+D9XPEhOtee0nP/0pMHUqsHYtkJcHfPkl8Ne/+krfr7sOOHHCbx9HRFRJrQldRCIBvALgMgDnAbhJRM6r4qnvqOoAzzbVz3H6nypw993WX+JnLVsCP/gB8MADwLx5wPPP2z02zj+/wXfCIyKqVV1a6EMAZKrqDlUtBPA2gKsCG1YQiNiNo//6V2D37oB+zH332eqM2dnA4MHAwoUB+zgiCmN1SejxAL4r83iPZ19FPxWRdSIyS0R6VPVGIjJeRDJEJCMnJ6cB4frZc8/ZYixjxgR8UZYf/cj617t0AX7yE+Dll7laIxH5l7+G6T4AkKSq/QAsAPB6VU9S1Smqmq6q6XFxcX766EY4+2zg738HFi+2kcwgfNzSpcDIkcC999paYRkZAf9YIgoTdUno2QDKtri7e/adoaqHVdU7S2cqgEH+CS8IxowBbrnFWutHjwb849q0sUmqjz9uc5sGDwbS021A9eTJgH88EblYXRL6CgCpIpIsIs0AjAbwftkniEjXMg9HAdjsvxCD4JVXrD+kXbugfFxEhNWqZ2fbRxcUWGljt27APfdYpQy7Y4iovmpN6KpaDOAeAJ/AEvVMVd0oIk+IyCjP0yaKyEYRWQtgIoDbAhVwQLRuDfTsaVn0k0+Clk3btgV+8Qtg3TpgyRJbkWDqVGDAACApCZgwwW5KzZY7EdWFaIiagunp6ZrR1DqQZ8+2gvIpU6zJHAKHD/smKX36qSXzZs1s8tLllwNXX20l9EQUnkRkpaqmV3mMCb2M0lJbnGXJEltmt3fvkIZTUGB3zJs3z7YtW2z/4ME2UemnP7WBViIKH0zo9bFvH9Cvn/WH3HcfcOedQIsWoY4KAJCZaRcRs2ZZlz8ApKVZYh8/3lYFJiJ3qymhc3WRirp2Bd59F4iOtjn7UZ7lbhYtsvVfQjha2bMn8OtfA8uXA7t22Zyo5s0tzH79gAULQhYaETUBTOhVGTYM2LwZ2LbNOrABYOJEy5qpqVZzmJUV0hATE21pgSVLrCqmfXubsPSb39R8sw1Ve81bb9nFCBG5BxN6TTp39n2/YIENliYnW0JPTgYeeSR0sZXRr591wdx1F/DMM8CFFwLbt5d/zsmTFn5amh0fM8bKJAcNshLKb74J+GRZIgow9qE3RFYW8PrrwNCh1izOygL+/GdrvUdF+barrgI6dQpqaLNnW7d/SQnwj3/YAOrf/w689hpw7BjQv7+tSTZwIDB/vg22fv21jQd37GjruV9/vZ1WTExQQyeiOuCgaKDNng3cfDNw+nT5/cuXW0Z9803gnXeABx+07hw/Lttbld27LZyvvrLH0dFWFXP33XZL1Yoff+SIld/Pm2ezV48eBWJj7cZOo0cDw4f7hhKIKLSY0IOhsNDuQVdcbFtRkZWdxMTYDUcffhjIybGm8YMPWjM4Ojpg4RQXWwv95Eng9tvL9x7Vdhqffmp/f+bMsbXdO3a0PwiTJrEGnijUmNCbglOngH//20pTtm61WUIffWSJ/8knLbl7t/btgSuvtEwaQqdP27K/77wDzJ1rrfRnn7U5VwG+yCCiajChNyWlpda30ayZdVQfP24rdlX0u9/ZzTeKi63ZHOJa+F27rG9+4UJbCnjq1Npb6yUlVkmze7cNM+zebVtkJDBunA3mElH9MKE3daqW/YqKbNu+3QZT4+PtbtM33QRce62Vplx8ccg6tFWtUuahh6yFXrG1npNjt977/HPgiy+s8rO4uPx7tGtnFyunT9twwsSJwKhRluSJqHZM6E62fj3wwgs22SkvD+jQwRZ0ee45W1QsBHbtshb2Z59Za71XL0viGzfa8RYtgAsusJLIxETbEhJsa93aBmGnTbObfOzebQuR3X23XQEEacFLIsdiQneDU6dsxa7//tdmEq1fb03j11+3LpvYWODQIdu6dbOSSQCYPBno29duaBrhv2kHZVvrqlbbftFF1uoeNMg3H6smxcV2r9UXX7QWfYsWwK23Wqv93HOrf93x41aGOX26/TF46ingvKruclsfpaV+/fkQBQoTutuoWjJXBb73PWDTpvLHvQOu+flWaZOfb2Uuo0ZZLeLw4ZWLzAsKrOl89Kg1pVu1so7vJUuAEyfsPVq1sib0xRfbwO2pUzh9sgSRbVs1umBn7Vq7EPnPfyyUyy4D7r8f+PGPfV06mZnWqp8+HSg9fgJjU7/B4n09sSk/CXfeCTz2mP0tq05JCXD8QD5i355slxlZWbbt2mVjFr/6lf1BHDMG6NHDtsREG+vo2rX6N6amqaDA1qbu0yfkY1D+VFNCh6qGZBs0aJCSHxQWqn7xherCharr1qnu26daUOA7npurOmOG6g03qLZqpQqoPvusHVu4ULV7d9UWLWy/d/vsMzv+9tvl93u3jAw7/s9/2uNevVQfflh1xQrV0tK6x56To7pggeozz6j+7Geq3/++5j3xnD7xhGrnzqqJ2Kl9epfoX/6ies2IfL0En2pCVLaOGaOa+dibqoCWRkXp4n4TNCEqW1u0UH3kEdW8PHv7khLV1atVX/rTCZ140RqNjVWNRoGWQDQ/urUeiu+reRddoaW/uNviUFXdvl110CDVTp185xsdrTprVqP+mQLu9GnVp59WHTvW9+8Tjk6fVi0qsu8ffdT+/c46S3XkSNXJk1W/+y44cWzaZP8nFi2q3/+JOgCQodXkVbbQw8np09bxPWCADbhu3GhllO3b+7bYWGuBd+5sU0v377eWefPm1lI/etRmxLZoYa2fefOscP3zz60JnJgIbNhgrzl4ENi7195j3z772qGDLQ2pat97b/vXvTuQkmJdRQ88gIKcPMR0aotT0hw7NBk9kYkYFOLYH19C29/dYwvHL1tmg8b/+hdKI6PwYfJEXL3laXSMi8D55wMrFp/GDUcn42E8jaKo5nhybCYSz47CxiW5WLC8LXIOWdO/XTvr87/2WtvatvX8vAoKbD2f6dNtkZwuXWyRtj17gBtu8F3leK+YAOtDKi62PqgAzBLOzbU7Jp48aRcUI0Z4PnrSJJut3KKFXU1deaVdsgwc6PcYgqakxO7wsn27bzt1ymbN3XWXdZO99Zb9u5w6Zd2Rc+favpEjgZ07bRr08uX2Pjt3Ai1b2pVos2Z29Slil3VduzZsavTJk3Z5uXSpfda4cXZ5+c031s0J2FX0PffYlV/Llo3+sbCFToF36JDqq6+qPvCAb9/gwZVb9xdd5Ds+Y4a1jHNyKr/f8eOqU6dq6X336/GLr9CiXz6k+tFHvuZ3Wdu3q95yi+qNN+ry5aojhhfoYx1f1MMt4lUBPfX94apLlpR7SWmp6tatqtOnq955p2pysoUXE6N6zTWq776rmp9fxXneeqs9sVMn1QsuUE1IUL3kEt/x3r1959q7t+qECarz59fjB+mRna36wQeqy5efueLascPeMjpaNT5etSuy9cpzt+qMGapFew+q/u9/qseOqT75pGpsrOq4cfX/XC9vq7K42C5zVq2ybeVK+8FV+cNppIMHVf/yF9UXX/TF0LKl/SxjY+3K6Yc/VH3hBTuek1P+dys2VvW22yzeqs5n48byV1oDB5Z/fYcOdqXoNXWq6ptv2hXwzp22bd9uxw4fVj3nHFUR3+vPPlv1rbfseEmJ/Q5Pm6Y6YIAdb9/erpgbCWyhU0h89JFdFXTtaq2oLl0C25dZUmL1j5s22UDw0KFWy3/xxbW+VNUacjNmAG+/DRw4YBU5V19trfdzzrGtW1eFfPapTcM9csT62dPSkP/zB5CVBRz4Zidicg+g39Ev0HLF57b+wrhxVpV0+rSNYaSk2OJu3i011T5s7VprVS9fblc2Xl98gaXRP8RvRm7A2YWbcdfUwUjfMwelj/wBqyMGYWj+IqSkWIv9ttuAs86CXV0VFtoYyrJlNnlt1Cj7WfTsWfXMsOxsW2z/nXds8OLxx62y6swlSxm//72959GjVp6UkGCXOnl5dhkxdqyNkq9ebZcU/ftbi/X8820CgnfQpbTUrnqmTLG7pxcV2QL/s2bZ8Q0brAXdvn3V/947dtjVX0mJ/UPVZTTea9MmK7Pau9e3JSTY1Q5gV1g5OeVfM3asFSKoWov7nHPs3IYOtd/vqqhnidOvv7b1rxuJLXQKL9u2WauqgX2XxcV24XDHHapt25ZvxLVsqZqWpnrjjarXX28XIWW728tuKSmqN99YpH9/OleXLlU9tjlbC/qkaVHr2HJP/N+lf9MFC1QLM9aqpqaq3nyztUIXL1adOVNnvXFSY2JUX479XfkPuOwyLdmWqe+9pzpkiO1q1051zBi7wjhzMTNjhmqXLr7XxcerjhmjJUdydf9+1ZLJU1QvvNB3vH9/a52q2hjN7Nmq772nOmeO6ty5qm+8Ya11VWux9unjG59p1cre/9//tuM7d6peeaVqt26+92/e3MZvVFXvvdfXev3lL60V3VScPKm6ZYtdYU2dan3wa9aEOiq20IkaqrTUGq5bt1p3+rZtvu8jIqyBnZRUfisqskb2smW27dlT+X3bIhfJ2IkU7MA69EMmUtG2re++sSNGWKP9T38Cfvtba3zOmVmIjnvX2ZsmJ1tfraelrWrDGK+9ZsMK3m7iSy6xYYkfXKg4tGQrCj75HG1WfY64/evRP2I9TuRHYG7UtfhezLfIHHgjIm68Hn2u7XWmqCc31853yxb7unWrNcRvvdWGCc409FXth1XdDDFV4LvvrJW6dCkKH5iE4626ov3OlZBtW23w4qyz/PcP52IsWyQKob17LQdnZloyjIuzZXri4mxr1szGlefOtbG7Q4esR6J3bxt3/tnPbCJWXfNdcbH19Myda9vOneWPd+lidft9+tg9aXduzMdXq1pg7VrfzN7u3a3H5uBB3+uioqy3aO9eGx9PTQXuuMN6ISqWix49CmRk2B+29eut5+LIEd924oQ9b8gQ+4N15ZWcBlBXTOhEDlFSYo3YuXNt3ZxrrrHu6oYuhqZq3dArV1ryPu88Ky6qyunT1uW9bJndMKV5c5sF7N1SUuwPzcmT1sU9fbot9RARYRcLP/yhDQOsWAF8+63vfVNSbBilbDFVhw52TtOmWTf4975nC5LecEP9V7YoLfVNk2jqDh60+YG9elm3e0MwoRNRQHz7LfDqqzZOuHevtdSHDLFt8GAgPd0qYatTXGxjsE8/bVW0KSlWIXrDDTW/7uhRu0HLxx/biqA5OTaOe8st1mXV0OrA06dtnLTsvDPvFh1tXWKjRtnVSV2o2lXWhx/atmyZ7Zs40SbSNQQTOhEFVHGxJdm4uIa9vrTUupueespa+IAl9LLFQMnJ1qf/8cdW9l1aal1Yl15qxUYzZ1ribdXK1u/3Ftp4u3IKCuz1ubnW7ZOVZaXtO3b4ytyzs8vfBz4y0qZsJCZa4dC6dba/Vy9L7KNGWeHO8eP2By0721cws3273TjGO4YyeDBwxRVWIp+W1vAuJiZ0InIEVevGWbHC+v7LbgUF9pxBg6yL5/LL7UrAOw5bWgosXmw3CJs505JsXJwdz82tfEMxr65drTsqJcW+Jif7FpWLjy/fBZSVZX94PvjAqi2Liuz9q7ofb/v2trbRyJEWa3VVjfXFhE5EjlZaahONo6PrdhWQn2+Tdj/+2CaAxsaW39q1s1Z9cnLDu2fy8qwFvnKlxRQfb11O8fH2RyJQUy6Y0ImIXKKmhM5CISIil2BCJyJyCSZ0IiKXYEInInIJJnQiIpdgQicicgkmdCIil2BCJyJyiZBNLBKRHABZDXx5RwCH/BiOk4TrufO8wwvPu3qJqlrlfNmQJfTGEJGM6mZKuV24njvPO7zwvBuGXS5ERC7BhE5E5BJOTehTQh1ACIXrufO8wwvPuwEc2YdORESVObWFTkREFTChExG5hOMSuoiMEJGtIpIpIpNCHU+giMh0ETkoIhvK7GsvIgtE5FvP13ahjDEQRKSHiCwSkU0islFE7vPsd/W5i8hZIrJcRNZ6zvtxz/5kEVnm+X1/R0SahTrWQBCRSBFZLSIfeh67/rxFZJeIrBeRNSKS4dnXqN9zRyV0EYkE8AqAywCcB+AmETkvtFEFzGsARlTYNwnAZ6qaCuAzz2O3KQbwoKqeB2AogLs9/8ZuP/cCAMNVtT+AAQBGiMhQAH8G8Jyq9gRwFMCdoQsxoO4DsLnM43A574tVdUCZ2vNG/Z47KqEDGAIgU1V3qGohgLcBXBXimAJCVb8EcKTC7qsAvO75/nUAVwczpmBQ1X2qusrz/XHYf/J4uPzc1ZzwPIz2bApgOIBZnv2uO28AEJHuAEYCmOp5LAiD865Go37PnZbQ4wF8V+bxHs++cNFZVfd5vt8PoHMogwk0EUkCkAZgGcLg3D3dDmsAHASwAMB2ALmqWux5ilt/358H8GsApZ7HHRAe560A5ovIShEZ79nXqN/zKH9GR8Gjqioirq05FZFWAP4L4H5VzbNGm3HruatqCYABIhIL4D0A54Y2osATkSsAHFTVlSIyLMThBNuFqpotIp0ALBCRLWUPNuT33Gkt9GwAPco87u7ZFy4OiEhXAPB8PRjieAJCRKJhyfwtVZ3t2R0W5w4AqpoLYBGA8wHEioi34eXG3/cLAIwSkV2wLtThAF6A+88bqprt+XoQ9gd8CBr5e+60hL4CQKpnBLwZgNEA3g9xTMH0PoBbPd/fCmBuCGMJCE//6TQAm1X1b2UOufrcRSTO0zKHiDQH8GPY+MEiANd5nua681bVh1W1u6omwf4/L1TVm+Hy8xaRliLS2vs9gJ8A2IBG/p47bqaoiFwO63OLBDBdVZ8KbUSBISIzAAyDLad5AMCjAOYAmAkgAbb08A2qWnHg1NFE5EIAiwGsh69P9bewfnTXnruI9IMNgkXCGlozVfUJEUmBtVzbA1gNYIyqFoQu0sDxdLk8pKpXuP28Pef3nudhFID/qOpTItIBjfg9d1xCJyKiqjmty4WIiKrBhE5E5BJM6ERELsGETkTkEkzoREQuwYROROQSTOhERC7x/wFJO1SUkY7qCgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = CNN().to(device)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n",
    "criterion = nn.NLLLoss()\n",
    "\n",
    "model, losses, accuracies = train_val_model(model, criterion, optimizer, dataloaders,\n",
    "                       num_epochs=50, log_interval=10)\n",
    "\n",
    "_ = plt.plot(losses['train'], '-b', losses['val'], '--r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uIhei09Ruvf-"
   },
   "source": [
    "# Bonus Information - Visualizing CNN filters\n",
    "\n",
    "Some work have been done to demonstrate the type of features learned by different filters in different layers. \n",
    "\n",
    "For instance, considering a known CNN called **VGG16** which has the following architecture\n",
    "\n",
    "![image](https://media.geeksforgeeks.org/wp-content/uploads/20200219152327/conv-layers-vgg16.jpg)\\[taken from: https://www.geeksforgeeks.org/vgg-16-cnn-model/ \\]\n",
    "\n",
    "these would be some of the filters from some of the layers: \n",
    "<table border=0 width=\"500px\" >\n",
    "\t<tbody> \n",
    "\t\t<tr>\n",
    "\t\t\t<td width=\"19%\" align=\"center\"> Layer 2 <br /> (Conv 1-2)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l2_f1.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l2_f21.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l2_f54.jpg\"> </td>\n",
    "\t\t</tr>\n",
    "\t\t<tr>\n",
    "\t\t\t<td width=\"19%\" align=\"center\"> Layer 10 <br /> (Conv 2-1)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l10_f7.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l10_f10.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l10_f69.jpg\"> </td>\n",
    "\t\t</tr>\n",
    "\t\t<tr>\n",
    "\t\t\t<td width=\"19%\" align=\"center\"> Layer 17 <br /> (Conv 3-1)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l17_f4.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l17_f8.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l17_f9.jpg\"> </td>\n",
    "\t\t</tr>\n",
    "\t\t<tr>\n",
    "\t\t\t<td width=\"19%\" align=\"center\"> Layer 24 <br /> (Conv 4-1)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l24_f4.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l24_f17.jpg\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/results/layer_visualizations/layer_vis_l24_f22.jpg\"> </td>\n",
    "\t\t</tr>\n",
    "\t</tbody>\n",
    "</table>\n",
    "\n",
    "or obtain the class activations:\n",
    "<table border=0 width=\"500px\" >\n",
    "\t<tbody> \n",
    "    <tr>\t\t<td width=\"27%\" align=\"center\"> Input Image </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> Layer Vis. (Filter=0)</td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> Filter Vis. (Layer=29)</td>\n",
    "\t\t</tr>\n",
    "<tr>\n",
    "\t\t\t<td width=\"27%\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/pytorch-cnn-visualizations/master/input_images/spider.png\"> </td>\n",
    "\t\t\t<td width=\"27%\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/cnn-gifs/master/spider_layer_graph.gif\"> </td>\n",
    "\t\t\t<td width=\"27%\" align=\"center\"> <img src=\"https://raw.githubusercontent.com/utkuozbulak/cnn-gifs/master/spider_filter_graph.gif\"> </td>\n",
    "\t\t</tr>\n",
    "\t</tbody>\n",
    "</table>\n",
    "\n",
    "\\[examples taken from: http://www.github.com/utkuozbulak/pytorch-cnn-visualizations \\]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VU21L86BvAXK"
   },
   "source": [
    "# Bonus Information - Predefined architectures, pre-trained models and transfer learning\n",
    "\n",
    "Packages like [torchvision](https://pytorch.org/vision/stable/index.html) and [timm](https://rwightman.github.io/pytorch-image-models/) offer you the possibility of using predefined architectures or even use pre-trained models that can be used to fine tune the models for that same task or used for transfer learning.\n",
    "\n",
    "Besides datasets, transforms and others, **Torchvision** has a large number of predefined architecture with the possibility of loading the pre-trained weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w_1YpmkV-PbU"
   },
   "source": [
    "#### Torchvision classification models examples\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "l5yOWA9l4lrF"
   },
   "outputs": [],
   "source": [
    "import torchvision.models as models\n",
    "\n",
    "# construct a model with random weights to be trained\n",
    "resnet18 = models.resnet18()\n",
    "\n",
    "# load a pre-trained model\n",
    "resnet18 = models.resnet18(pretrained=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nAsfqUIyDBD7"
   },
   "source": [
    "For examples of different models and how to use pre-trained weights please visit https://pytorch.org/vision/stable/models.html#\n",
    "\n",
    "\n",
    "\n",
    "Another possibility is **timm** which contains models for classification only.\n",
    "In **timm** you are not restricted to have inputs only with 1/3-channels, allowing you to use architectures or pre-trained models using images that have 2 or > 3-channels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4zS8Ykbo-ZUg"
   },
   "source": [
    "#### timm classification models examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "w0FQYnKQ-lr_",
    "outputId": "0d3ef180-ef41-4653-99dd-6721c41c2475"
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'timm'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-53-6451e42f7d0e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;34m'google.colab'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m   \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msystem\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'pip install -q timm'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mtimm\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m# list all models\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'timm'"
     ]
    }
   ],
   "source": [
    "if 'google.colab' in str(get_ipython()):\n",
    "  !pip install -q timm\n",
    "import timm\n",
    "\n",
    "# list all models\n",
    "print(timm.list_models())\n",
    "\n",
    "# list pre-trained models\n",
    "print(timm.list_models(pretrained=True))\n",
    "\n",
    "# list models architectures by wildcards\n",
    "print(timm.list_models('*resne*t*'))\n",
    "\n",
    "# construct a model with random weights to be trained\n",
    "model = timm.create_model('resnet18')\n",
    "\n",
    "# load a pre-trained model\n",
    "model = timm.create_model('resnet18', pretrained=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OSEwqB8ADIao"
   },
   "source": [
    "For more details on how to use this package visit https://rwightman.github.io/pytorch-image-models/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kaank-9kDI72"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "P8-CNNs.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}