{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Practical 12 \n",
"\n",
"This practical has two parts (jupyter notebooks), **Part I: Word embeddings** and **Part II: Transformers**.\n",
"\n",
"# Part I: Word embeddings\n",
"\n",
"This Jupyter Notebook consists of the following parts:\n",
"1. [Word Embeddings](#word_embeddings) \n",
" 1. [Word2Vec](#word2vec) \n",
" 2. [Word2Vec Architectures](#word2vec_arch) \n",
" 3. [Word Embeddings Visualisation](#visual)\n",
"2. [Exploring Word Vectors with GloVe](glove)\n",
" 1. [Loading Word Vectors](#loading)\n",
" 2. [Finding Closest Vectors](#finding)\n",
" 3. [Word Analogies with Vector Arithmetic](#analogies)\n",
"3. [Motivation for Part II: Transformers](#transformers)\n",
"\n",
"\n",
"Check out a supplementary jupyter notebook for Part I, if interested in Skip-Gram implementation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"# Word embeddings"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Word2Vec\n",
"\n",
"There are two classes of vector models: count-based (TF-IDF, Bag-of-Words) and neural-based. In this practical we will be focusing on neural word embeddings, i.e. word embeddings learned by a neural network.\n",
"\n",
"**Main idea:** to use neural architectures that are predicting (not counting) the next word or a context of a given word.\n",
"\n",
"One of the most known such models is **Word2Vec**. It is based on a neural network that is predicting the probability of a word given it's context. It was created by Mikolov et al. (2013). Here are the main papers on the topic:\n",
"\n",
"* [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf)\n",
"* [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546)\n",
"\n",
"These vectors are ususally reffered to as **_distributed representations of words_** or **_word embeddings_**.\n",
"\n",
"_As word embeddings are a key building block of deep learning models for NLP, word2vec is often assumed to belong to the same group. Technically however, word2vec is not be considered to be part of deep learning, as its architecture is neither deep nor uses non-linearities_ \n",
"\n",
"*a quote from [Sebastian Ruder's blog](https://ruder.io/word-embeddings-1/) \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Word2Vec Architectures\n",
"\n",
"There are two Word2Vec architectures: Skip-Gram and CBOW.\n",
"\n",
"**Skip-Gram** predicts context words given the central word. Skip-Gram with negative sampling is the most popular approach.\n",
"\n",
"**CBOW (Continuous Bag-of-Words)** predicts the central word from the sum of context vectors. This simple sum of word vectors is called \"bag of words\", which gives the name for the model.\n",
"\n",
"\n",
"\n",
"\n",
"*the image is taken for [Lena Voita's blog](https://lena-voita.github.io/nlp_course/word_embeddings.html#main_content) \n",
"*if interested, please, check it out, it is very informative and illustrative"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How does it work?\n",
"\n",
"Word2vec takes a large text corpus as input and maps each word to a vector, producing word coordinates as output. It first creates a dictionary by training on the input text data, and then calculates a vector representation of the words. The vector representation is learned on contextual proximity: words that occur in the text next to the same words (and therefore, according to the distributive hypothesis, have a similar meaning) will have close coordinates in the vector representation. \n",
"\n",
"To calculate the proximity of words, usually the cosine or euclidean distances between vectors are used.\n",
"\n",
"Using distributed representations you can build semantic proportions (also known as analogies) and solve examples like:\n",
"\n",
"*king: male = queen: female*\n",
" $\\Rightarrow$\n",
"*king - man + woman = queen*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Word Embeddings Visualization\n",
"\n",
"Go to https://projector.tensorflow.org/ and visualize Word2Vec embeddings. \n",
"\n",
"Original Word2Vec repository: https://code.google.com/archive/p/word2vec/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"# Exploring Word Vectors with GloVe:\n",
"\n",
"As we have seen, the word2vec algorithms (such as Skip-Gram) predicts words in a context (e.g. what is the most likely word to appear in \"the cat ? the mouse\"), while GloVe vectors are based on global counts across the corpus — [see How is GloVe different from word2vec?](https://www.quora.com/How-is-GloVe-different-from-word2vec) on Quora for some better explanations.\n",
"\n",
"The best feature of GloVe is that multiple sets of pre-trained vectors are easily available for [download](https://nlp.stanford.edu/projects/glove/), so that's what we'll use here.\n",
"\n",
"Part II of this notebook is taken from [practical-pytorch tutorials](https://github.com/spro/practical-pytorch/blob/master/glove-word-vectors/glove-word-vectors.ipynb)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Loading Word Vectors\n",
"Torchtext includes functions to download GloVe (and other) embeddings"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"\n",
"from dataclasses import dataclass\n",
"from typing import List, Dict"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Downloading embeddings\n",
"\n",
"Run the following python cell to automatically download, store and unzip the glove embeddings. Note that approx 1G of storage is required.\n",
"\n",
"Alternatively, you can download the files from [Stanford NLP](https://nlp.stanford.edu/projects/glove/)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Archive: glove.6B.zip\n",
" inflating: glove.6B.50d.txt \n",
" inflating: glove.6B.100d.txt \n",
" inflating: glove.6B.200d.txt \n",
" inflating: glove.6B.300d.txt \n"
]
}
],
"source": [
"# execute only if you want the glove embeddings to be downloaded and unzipped here.\n",
"\n",
"!wget https://nlp.stanford.edu/data/glove.6B.zip \n",
"!unzip glove.6B.zip -d glove.6B"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"glove_file_path = \"glove.6B/glove.6B.50d.txt\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"@dataclass(frozen=True)\n",
"class GloVe:\n",
" stoi: Dict[str, int]\n",
" itos: List[str]\n",
" vectors: List[torch.Tensor]\n",
"\n",
"def load_glove(path):\n",
" with open(path, \"r\") as f:\n",
" lines = [l.strip() for l in f]\n",
" \n",
" stoi = {}\n",
" itos = []\n",
" vectors = []\n",
" for line in lines:\n",
" splits = line.split()\n",
" word = splits[0]\n",
" vec = torch.tensor([float(v) for v in splits[1:]])\n",
" \n",
" # We append the word to the idx2word list\n",
" # so its index will be len(idx2word) - 1\n",
" itos.append(word)\n",
" stoi[word] = len(itos) - 1\n",
"\n",
" vectors.append(vec)\n",
"\n",
" return GloVe(stoi=stoi, itos=itos, vectors=vectors)\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 400000 words\n"
]
}
],
"source": [
"glove = load_glove(glove_file_path) \n",
"print('Loaded {} words'.format(len(glove.itos)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Loaded 400000 words\n",
"The returned GloVe object includes attributes:\n",
"\n",
"- stoi string-to-index returns a dictionary of words to indexes\n",
"- itos index-to-string returns an array of words by index\n",
"- vectors returns the actual vectors. To get a word vector get the index to get the vector:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def get_word(word):\n",
" return glove.vectors[glove.stoi[word]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Finding Closest Vectors\n",
"\n",
"Going from word → vector is easy enough, but to go from vector → word takes more work. Here I'm (naively) calculating the distance for each word in the vocabulary, and sorting based on that distance:\n",
"\n",
"Anyone with a suggestion for optimizing this, please let me know!"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from tqdm.notebook import tqdm\n",
"import numpy as np \n",
"\n",
"def manhattan_distance(vector1, vector2):\n",
" return np.abs(vector1 - vector2).sum()\n",
"\n",
"def cosine_similarity(vector1, vector2):\n",
" return np.dot(vector1, vector2) / (np.linalg.norm(vector1) * np.linalg.norm(vector2))\n",
"\n",
"def closest(vec, n=10, dist='pnorm'):\n",
" \"\"\"\n",
" Find the closest words for a given vector\n",
" \"\"\"\n",
" if dist=='pnorm':\n",
" all_dists = [(w, torch.dist(vec, get_word(w))) for w in tqdm(glove.itos)]\n",
" return sorted(all_dists, key=lambda t: t[1])[:n]\n",
" elif dist =='manhattan':\n",
" all_dists = [(w, manhattan_distance(vec, get_word(w))) for w in tqdm(glove.itos)]\n",
" return sorted(all_dists, key=lambda t: t[1])[:n]\n",
" elif dist =='cosine':\n",
" all_dists = [(w, cosine_similarity(vec, get_word(w))) for w in tqdm(glove.itos)]\n",
" return sorted(all_dists, key=lambda t: t[1], reverse=True)[:n]\n",
" else:\n",
" print('Unknown distance type')\n",
" return None\n",
" \n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will return a list of (word, distance) tuple pairs. Here's a helper function to print that list:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"def print_tuples(tuples):\n",
" for tuple in tuples:\n",
" print('(%.4f) %s' % (tuple[1], tuple[0]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now using a known word vector we can see which other vectors are closest:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "9837c8795f154a268111d258c9182382",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/400000 [00:00\n",
"## Word Analogies with Vector Arithmetic\n",
"The most interesting feature of a well-trained word vector space is that certain semantic relationships (beyond just closeness of words) can be captured with regular vector arithmetic.\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"# In the form w1 : w2 :: w3 : ?\n",
"def analogy(w1, w2, w3, n=10, filter_given=True, dist = 'manhattan'):\n",
" # w2 - w1 + w3 = w4\n",
" result = get_word(w1) - get_word(w2) + get_word(w3)\n",
" closest_words = closest(result, n=n, dist=dist)\n",
" print('\\n[%s - %s + %s = ?]' % (w1, w2, w3))\n",
" # Optionally filter out given words\n",
" if filter_given:\n",
" closest_words = [t for t in closest_words if t[0] not in [w1, w2, w3]]\n",
" \n",
" print('Closest word =',closest_words[0][0]) \n",
" \n",
" vectors = [get_word(w1), get_word(w2), get_word(w3), result, get_word(closest_words[0][0])]\n",
" words = [w1,w2,w3, 'result', closest_words[0][0]]\n",
" return vectors, words\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a0c3985303594297aa94d2227318ea66",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/400000 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"vectors, words = analogy('king', 'man', 'woman')\n",
"plot_embedding_vectors(vectors,words)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "da05f59e31ec4b7ab99f62b1c9f06628",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/400000 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"vectors, words = analogy('man', 'king', 'queen')\n",
"plot_embedding_vectors(vectors,words)\n"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a197c0b5868f4e97a7ac1fadc78b9f3d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/400000 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"vectors, words = analogy('chinese', 'china', 'japan')\n",
"plot_embedding_vectors(vectors, words)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Comment:** One of the applications of word embeddings is for example using them in the embedding layer of your model instead of using randomly initialised input that is being corrected during training. \n",
"\n",
"These pre-trained word embeddings (from Word2Vec, Glove, etc.) can either be kept static or modified during training.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# BERT embeddings"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Glove embeddings have nice properties but cannot distinguish between multiple meanings of a word. \n",
"\n",
"E.g. the word *play* can be used as a verb if it refers to the activity of playing (e.g. between kids) or as a noun if it refers to a theatre play.\n",
"\n",
"Instead, contextualised embeddings learn to encode words depending on their context within a sentence (word sequence). We will see for example BERT embeddings which offer contextualized representations leveraging the Transformer architecture."
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"\n",
"def plot_embedding_vectors_multi_type(vectors, words):\n",
" fig, ax = plt.subplots()\n",
"\n",
" # Plot the points\n",
" ax.scatter([v[0] for v in vectors], [v[1] for v in vectors], c='blue', label='Words', marker='o')\n",
" \n",
" # Label points\n",
" for i in range(len(vectors)):\n",
" ax.text(vectors[i][0], vectors[i][1], words[i], fontsize=12, ha='right')\n",
"\n",
" # Legend\n",
" ax.legend()\n",
"\n",
" plt.show()\n",
" \n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.\n",
"/opt/homebrew/Caskroom/miniforge/base/envs/dl-2023/lib/python3.8/site-packages/transformers/tokenization_utils_base.py:2418: FutureWarning: The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).\n",
" warnings.warn(\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAooAAAGdCAYAAACVT1IyAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAABCzklEQVR4nO3de1xVVf7/8feRuyjHO6IioKViphWYgpoyGmp5ycto4ZimaZamptZo9kuny9C3GZ1qkrxUOFo6poOTU96o0dJQQ5OpGc36Ot4FFVIQL8hl/f7gwfl6ZIuA3MTX8/HYDzvrrL3PZ58dnDd777WOzRhjBAAAAFyjRmUXAAAAgKqJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACy5VnYBuHXl5eXp5MmTql27tmw2W2WXAwAAisEYo/Pnz6tJkyaqUaPoc4YERZTayZMn5e/vX9llAACAUjh27JiaNWtWZB+CIkqtdu3akvL/R/Px8ankagAAQHFkZGTI39/f8TleFIIiSq3gcrOPjw9BEQCAW0xxbhtjMAsAAAAsERQBAABgiaAIAEAlW7p0qWw2m9PSsGFD9ejRQ5999plT32v7Xb2MHj3a0W/u3LlOz7m5ual58+YaN26cUlJSJEk9evQocnsFy9y5c8t8nwMDA53qRdXEPYooV8YY5eTkKDc3t7JLqdZcXFzk6urKNEXALS42NlZt2rSRMUYpKSl699131b9/f61bt079+/d39Bs6dKimT59eaP2GDRsWatu4caPsdrsyMzO1efNmzZs3TwkJCUpKSlJMTIwyMjIcfT///HO99tprjjoK3GhkLKovgiLKzZUrV5ScnKyLFy9Wdim3hZo1a8rPz0/u7u6VXQqAUmrXrp1CQ0Mdj/v06aO6detq5cqVTkHR19dXnTt3LtY2Q0JC1KBBA0lSr169lJqaqtjYWG3fvl0RERFOfX/88UfLOnD7IiiiXOTl5enQoUNycXFRkyZN5O7uztmucmKM0ZUrV3TmzBkdOnRId9555w0nUAVwa/D09JS7u7vc3NzKbJuhoaGKjY3VqVOnymybBUaPHq01a9Zo165dmjx5snbu3KmaNWtq+PDhevPNN1WzZs3rrnv58mXNnj1bX375pePzo3Xr1po5c6YGDhzo6NezZ0+dOHFC+/fvd/pcMcbozjvvVOvWrfX5558Xq94ePXo4gvO0adO0Z88eNW7cWOPHj9cLL7zg9Lv06NGjevHFF7V582alp6erRYsWevLJJ/Xcc885+m3dulURERHasmWLevTo4Vj38OHDCgoKUmxsrONye8F7lZSUpMmTJ+vrr79W3bp1NWzYMP3+97+Xh4dHsfahvBEUUS6uXLmivLw8+fv7F/mLAWXDy8tLbm5uOnLkiK5cuSJPT8/KLglAKeTm5ionJ0fGGJ06dUp/+MMfdOHCBUVFRTn1K7it51ouLi43/KP80KFDkqRWrVqVXeFXyc7O1kMPPaSnnnpKM2fOVEJCgl577TUdOXJE//jHP667XlZWln755RfNmDFDTZs21ZUrV/TFF19o8ODBio2N1eOPPy5JmjJligYOHKgvv/xSvXr1cqy/YcMGHTx4UO+8806J6k1JSdGIESM0ffp0zZkzR2vXrtWsWbPUpEkTx2ueOXNG4eHhunLlil599VUFBgbqs88+04wZM3Tw4EHFxMSU4p3Kf68GDBigsWPHavr06fr666/16quvym636+WXXy7VNsucAUopPT3dSDLp6emFnrt06ZLZt2+fuXTpUiVUdnviPQduXbGxsUZSocXDw8PExMQ49bXqV7AsX77c0W/OnDlGkklJSTHZ2dnm7Nmz5pNPPjHe3t7mscceK7KOxMTEUu3HqFGjjCTz9ttvO7W//vrrRpLZvn27oy0gIMCMGjXqutvKyckx2dnZZuzYsebee+91tOfm5poWLVqYgQMHOvXv27evadmypcnLyyt2vd27dzeSzK5du5za27Zta3r37u14PHPmTMt+Tz/9tLHZbObAgQPGGGO2bNliJJktW7Y49Tt06JCRZGJjYx1tBe/VJ5984tT3oYceMq1bty72PpRGUZ/f1+KMIgAAVcSyZcsUHBwsSUpNTdXatWs1ceJE5ebmatKkSY5+w4YN0/PPP19o/RYtWhRqa9y4sdPjBx54QH/5y1/KuHJnI0aMcHocFRWl2bNna8uWLerSpct111u9erXeeust/etf/9KFCxcc7VdfJalRo4YmTZqk559/XkePHlXz5s118OBBbdy4UX/84x9LfJtT48aNdf/99zu1tW/fXklJSY7H//znP9W2bdtC/UaPHq333ntP//znP0t1htZmsznde1rw2v/85z+Vmytt2yYlJ0t+flK3bpKLS4lf4qZxIxMAAFVEcHCwQkNDFRoaqj59+mjRokWKjIzUCy+8oHPnzjn6NWzY0NHv6qVevXqFtvnFF18oMTFRmzZt0pAhQ/T111/r2WefLbd9cHV1Vf369Z3aCsJqWlraddeLi4vTsGHD1LRpU3300UfasWOHEhMTNWbMGF2+fNmp75gxY+Tl5aWFCxdKkhYsWCAvLy+NGTOmxPVeW6skeXh46NKlS47HaWlp8vPzK9SvSZMmN9yvotSsWbPQrUIeHh66fPmyAgOliAgpKir/38BAKS6uVC9zUwiKQBUSGBiot956q7LLAFCFtG/fXpcuXdJPP/1UqvU7dOig0NBQRUZGavXq1XrwwQe1ePFiJSYmlnGl+XJycgoFp4J5G61CWYGPPvpIQUFBWrVqlR555BF17txZoaGhysrKKtTXbrdr1KhRev/99/XLL78oNjZWUVFRqlOnTpnuS4H69esrOTm5UPvJkyclyTGqvCD0XVtzampqsV9r//78f48fd24/cUIaOrTiwyJBEbjKwoULVbt2baebxDMzM+Xm5qZu3bo59d22bZtsNlupf3kDQHEUXAK1miOxpGw2mxYsWCAXFxe99NJLN7296/n444+dHq9YsUKSnEYCW9V27QwZKSkp+vTTTy37T548WampqRo6dKjOnTvndGm+rPXs2VP79u3Td99959S+bNky2Ww2xzRDgYGBkqTvv//eqd+6deuK9Tq5udKGDdbPGZP/79Sp+f0qCvcoosqryPs0IiIilJmZqd27dzvmKNu2bZsaN26sxMREXbx40TGKe+vWrWrSpEmJ70vJzc2VzWZjChsAhfz73/92/KGalpamuLg4xcfHa9CgQQoKCnL0O3XqlHbu3FlofR8fH7Vt27bI17jzzjs1fvx4xcTEaPv27eratWuZ7oO7u7vmzZunzMxMdezY0THquW/fvkW+Vr9+/RQXF6dnnnlGQ4cO1bFjx/Tqq6/Kz89PP//8c6H+rVq1Up8+fbRhwwZ17dpVHTp0KNP9uNpzzz2nZcuW6eGHH9Yrr7yigIAAff7554qJidHTTz/t+Bxo3LixevXqpejoaNWtW1cBAQH68ssvFVfM04Dbtknnz1//eWOkY8fy+xWRucsUn1So0uLiVKH3abRu3VpNmjTR1q1bHW1bt27VwIED1bJlSyUkJDi1R0RE6OzZs3r88cdVt25d1axZU3379nX6pbZ06VLVqVNHn332mdq2bSsPDw8dOXJEp0+fVv/+/eXl5aWgoKBCf4FL+V/B1bx5c3l4eKhJkyaaPHly+ew4gCrhiSeeUFhYmMLCwjRixAh99913mj9/vlauXOnUb82aNY5+Vy/jx48v1uvMmTNHtWrVKpcpWNzc3PTZZ58pPj5eAwcO1DvvvKNx48Zp9erVRa73xBNP6I033tCGDRv00EMP6X/+5380c+bMQlMDXW348OGSVK5nE6X8s7kJCQn61a9+pVmzZqlfv37atGmT3nzzTf35z3926rt8+XL17NlTv/3tb/XrX/9aJ06cKHT8rsfi6vZN9SsT5Tr+GtVaeU+P87e/GWOzGZP/N9T/LTZb/vK3v91M9dcXFRVlIiMjHY87duxoVq9ebZ5++mnz4osvGmOMycrKMl5eXub99983AwYMMMHBwebrr782SUlJpnfv3uaOO+4wV65cMcbkTzfh5uZmwsPDzTfffGN+/PFHk5mZafr27WvatWtnEhISzO7du014eLjx8vIyf/rTn4wxxqxevdr4+PiY9evXmyNHjphdu3aZxYsXX7dupscBUNlGjRplvL29K+z1Bg8ebJo0aeL4fXur27Kl8Gee1XLN7DslxvQ4uOXl5kpTpvzfPRlXM0ay2fLv0xg4sOwvQ/fo0UPPPfeccnJydOnSJe3du1cPPPCAcnNzHRO57ty5U5cuXVLXrl315JNP6ptvvlF4eLik/Htz/P399fe//12//vWvJeVPqhoTE+O4NPLTTz9pw4YN2rlzpzp16iRJ+uCDDxzTYkj53wJQcBnDzc1NzZs3LzQ1AwDcbrKysvTdd9/p22+/1dq1azV//vwy/eaaytStm9SsWf7AFavPP5st//lrbpkvV1x6RpW0bVvhEV9Xu/o+jbIWERGhCxcuKDExUdu2bVOrVq3UqFEjde/eXYmJibpw4YK2bt2q5s2b68CBA3J1dXWEPSl/dFzr1q21v2DomvLv2Wnfvr3j8f79++Xq6ur0Xapt2rRxGrH361//WpcuXVKLFi00btw4rV271vKbGACgvOXl5SknJ6fIpaIkJycrPDxcL7/8sp566inLqX4KvuHmektuRY4GKQEXF+ntt/P/+9rpIAsev/VWxc6nSFBElVSZ92nccccdatasmbZs2aItW7aoe/fukvJvUg4KCtI333yjLVu26Fe/+pWM1Z98yv96ratH7nl5eRX6TlJJRU4M6+/vrwMHDjjmB3vmmWf0wAMPKDs7uyx2EwCKbcyYMXJzcytykfLvyc7MzCzXWgIDA2WMUXp6ut577z25WKSmnj17Fllry5Yty7XGmzF4sLRmjdS0qXN7s2b57YMHV2w9XHpGlWQxr+lN9SupiIgIbd26VWfPnnX69oPu3btr06ZN2rlzp5544gm1bdtWOTk52rVrl+PSc1pamn766Seny8jXCg4OVk5Ojnbv3u24nHzgwAGnCXWl/IA5YMAADRgwQBMnTlSbNm30ww8/6L777iv7nQaA65g7d265DxgpS4sWLdL5IoYPe3h4VGA1JTd4cP6tVVXhm1kIiqiSKvs+jYiICE2cOFHZ2dmOM4pSflB8+umndfnyZUVERMjf318DBw7UuHHjtGjRItWuXVszZ85U06ZNNXDgwOtuv3Xr1urTp4/GjRunxYsXy9XVVVOnTpWXl5ejz9KlS5Wbm6tOnTqpZs2aWr58uby8vBQQEFA+Ow0A1xEYGOiYI/BW0Lp168ou4aa5uFTcFDhF4dIzqqTKvk8jIiJCly5d0h133CFfX19He/fu3XX+/Hm1bNlS/v7+kqTY2FiFhISoX79+CgsLkzFG69evv+HN1bGxsfL391f37t01ePBgjR8/Xo0aNXI8X6dOHS1ZskRdunRR+/bt9eWXX+of//hHkd9sAABAWbKZ691kBdxARkaG7Ha70tPT5ePj4/Tc5cuXdejQIQUFBRX6HsuSiIvLH/189cAWf//8kFjR92lUdWX1ngMAqreiPr+vxaVnVGlV6T4NAABuNwRFVHlV5T4NAABuN9yjCAAAAEsERQAAAFgiKKJcMVaq4vBeAwDKGkER5aJgapiLFy9WciW3j4L3urp85ykAoPIxmAXlwsXFRXXq1NHp06clSTVr1izy6+pQesYYXbx4UadPn1adOnUsv84KAIDSICii3DRu3FiSHGER5atOnTqO9xwAgLJAUES5sdls8vPzU6NGjZSdnV3Z5VRrbm5unEkEAJQ5giLKnYuLCyEGAIBbEINZAAAAYImgCAAAAEsERQAAAFgiKFYjMTExCgoKkqenp0JCQrRt27br9t2+fbu6dOmi+vXry8vLS23atNGf/vSnCqwWAABUdQxmqSZWrVqlqVOnKiYmRl26dNGiRYvUt29f7du3T82bNy/U39vbW5MmTVL79u3l7e2t7du366mnnpK3t7fGjx9fCXsAAACqGpvhe7+qhU6dOum+++7Te++952gLDg7WI488oujo6GJtY/DgwfL29tby5cuL1T8jI0N2u13p6eny8fEpVd0AAKBileTzm0vP1cCVK1e0Z88eRUZGOrVHRkYqISGhWNvYu3evEhIS1L179+v2ycrKUkZGhtMCAACqL4JiNZCamqrc3Fz5+vo6tfv6+iolJaXIdZs1ayYPDw+FhoZq4sSJevLJJ6/bNzo6Wna73bH4+/uXSf0AAKBqIihWI9d+l7Ix5obfr7xt2zbt3r1bCxcu1FtvvaWVK1det++sWbOUnp7uWI4dO1YmdQMAgKqJwSzVQIMGDeTi4lLo7OHp06cLnWW8VlBQkCTp7rvv1qlTpzR37lw99thjln09PDzk4eFRNkUDAIAqjzOK1YC7u7tCQkIUHx/v1B4fH6/w8PBib8cYo6ysrLIuDwAA3KI4o1hNTJs2TSNHjlRoaKjCwsK0ePFiHT16VBMmTJCUf9n4xIkTWrZsmSRpwYIFat68udq0aSMpf17FP/7xj3r22WcrbR8AAEDVQlCsJoYPH660tDS98sorSk5OVrt27bR+/XoFBARIkpKTk3X06FFH/7y8PM2aNUuHDh2Sq6urWrZsqTfeeENPPfVUZe0CAACoYphHEaXGPIoAANx6mEcRAAAAN42gCAAAAEsERQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUAQAAYImgCAAAAEsERQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUAQAAYImgCAAAAEsERQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUAQAAYImgCAAAAEsERQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUAQAAYImgCAAAAEsERQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUq5GYmBgFBQXJ09NTISEh2rZt23X7xsXF6cEHH1TDhg3l4+OjsLAwbdq0qQKrBQAAVR1BsZpYtWqVpk6dqtmzZ2vv3r3q1q2b+vbtq6NHj1r2//rrr/Xggw9q/fr12rNnjyIiItS/f3/t3bu3gisHAABVlc0YYyq7CNy8Tp066b777tN7773naAsODtYjjzyi6OjoYm3jrrvu0vDhw/Xyyy8Xq39GRobsdrvS09Pl4+NTqroBAEDFKsnnN2cUq4ErV65oz549ioyMdGqPjIxUQkJCsbaRl5en8+fPq169euVRIgAAuAW5VnYBuHmpqanKzc2Vr6+vU7uvr69SUlKKtY158+bpwoULGjZs2HX7ZGVlKSsry/E4IyOjdAUDAIBbAmcUqxGbzeb02BhTqM3KypUrNXfuXK1atUqNGjW6br/o6GjZ7XbH4u/vf9M1AwCAqougWA00aNBALi4uhc4enj59utBZxmutWrVKY8eO1SeffKJevXoV2XfWrFlKT093LMeOHbvp2gEAQNVFUKwG3N3dFRISovj4eKf2+Ph4hYeHX3e9lStXavTo0VqxYoUefvjhG76Oh4eHfHx8nBYAAFB9cY9iNTFt2jSNHDlSoaGhCgsL0+LFi3X06FFNmDBBUv7ZwBMnTmjZsmWS8kPi448/rrfffludO3d2nI308vKS3W6vtP0AAABVB0Gxmhg+fLjS0tL0yiuvKDk5We3atdP69esVEBAgSUpOTnaaU3HRokXKycnRxIkTNXHiREf7qFGjtHTp0oouHwAAVEHMo4hSYx5FAABuPcyjCAAAgJtGUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUKxGYmJiFBQUJE9PT4WEhGjbtm3X7ZucnKyoqCi1bt1aNWrU0NSpUyuuUAAAcEsgKFYTq1at0tSpUzV79mzt3btX3bp1U9++fXX06FHL/llZWWrYsKFmz56tDh06VHC1AADgVmAzxpjKLgI3r1OnTrrvvvv03nvvOdqCg4P1yCOPKDo6ush1e/TooXvuuUdvvfVWiV4zIyNDdrtd6enp8vHxKU3ZAACggpXk85szitXAlStXtGfPHkVGRjq1R0ZGKiEhocxeJysrSxkZGU4LAACovgiK1UBqaqpyc3Pl6+vr1O7r66uUlJQye53o6GjZ7XbH4u/vX2bbBgAAVQ9BsRqx2WxOj40xhdpuxqxZs5Senu5Yjh07VmbbBgAAVY9rZReAm9egQQO5uLgUOnt4+vTpQmcZb4aHh4c8PDzKbHsAAKBq44xiNeDu7q6QkBDFx8c7tcfHxys8PLySqgIAALc6zihWE9OmTdPIkSMVGhqqsLAwLV68WEePHtWECRMk5V82PnHihJYtW+ZYJykpSZKUmZmpM2fOKCkpSe7u7mrbtm1l7AIAAKhiCIrVxPDhw5WWlqZXXnlFycnJateundavX6+AgABJ+RNsXzun4r333uv47z179mjFihUKCAjQ4cOHK7J0AABQRTGPIkqNeRQBALj1MI8iAAAAbhpBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoViMxMTEKCgqSp6enQkJCtG3btiL7f/XVVwoJCZGnp6datGihhQsXVlClAADgVkBQrCZWrVqlqVOnavbs2dq7d6+6deumvn376ujRo5b9Dx06pIceekjdunXT3r179eKLL2ry5Mn629/+VsGVAwCAqspmjDGVXQRuXqdOnXTffffpvffec7QFBwfrkUceUXR0dKH+v/3tb7Vu3Trt37/f0TZhwgT961//0o4dO4r1mhkZGbLb7UpPT5ePj8/N7wQAACh3Jfn85oxiNXDlyhXt2bNHkZGRTu2RkZFKSEiwXGfHjh2F+vfu3Vu7d+9WdnZ2udUKAABuHa6VXQBuXmpqqnJzc+Xr6+vU7uvrq5SUFMt1UlJSLPvn5OQoNTVVfn5+hdbJyspSVlaW43FGRkYZVA8AAKoqzihWIzabzemxMaZQ2436W7UXiI6Olt1udyz+/v43WTEAAKjKCIrVQIMGDeTi4lLo7OHp06cLnTUs0LhxY8v+rq6uql+/vuU6s2bNUnp6umM5duxY2ewAAACokgiK1YC7u7tCQkIUHx/v1B4fH6/w8HDLdcLCwgr137x5s0JDQ+Xm5ma5joeHh3x8fJwWAABQfREUq4lp06bp/fff14cffqj9+/frueee09GjRzVhwgRJ+WcDH3/8cUf/CRMm6MiRI5o2bZr279+vDz/8UB988IFmzJhRWbsAAACqGAazVBPDhw9XWlqaXnnlFSUnJ6tdu3Zav369AgICJEnJyclOcyoGBQVp/fr1eu6557RgwQI1adJE77zzjoYMGVJZuwAAAKoY5lFEqTGPIgAAtx7mUQQAAMBNIygCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIihWA2fPntXIkSNlt9tlt9s1cuRInTt3rsh14uLi1Lt3bzVo0EA2m01JSUkVUisAALh1EBSrgaioKCUlJWnjxo3auHGjkpKSNHLkyCLXuXDhgrp06aI33nijgqoEAAC3GtfKLgA3Z//+/dq4caN27typTp06SZKWLFmisLAwHThwQK1bt7ZcryBIHj58uKJKBQAAtxjOKN7iduzYIbvd7giJktS5c2fZ7XYlJCSU6WtlZWUpIyPDaQEAANUXQfEWl5KSokaNGhVqb9SokVJSUsr0taKjox33Qdrtdvn7+5fp9gEAQNVCUKyi5s6dK5vNVuSye/duSZLNZiu0vjHGsv1mzJo1S+np6Y7l2LFjZbp9AABQtXCPYhU1adIkPfroo0X2CQwM1Pfff69Tp04Veu7MmTPy9fUt05o8PDzk4eFRptsEAABVF0GximrQoIEaNGhww35hYWFKT0/Xt99+q/vvv1+StGvXLqWnpys8PLy8ywQAANUYl55vccHBwerTp4/GjRunnTt3aufOnRo3bpz69evnNOK5TZs2Wrt2rePxL7/8oqSkJO3bt0+SdODAASUlJZX5fY0AAODWRVCsBj7++GPdfffdioyMVGRkpNq3b6/ly5c79Tlw4IDS09Mdj9etW6d7771XDz/8sCTp0Ucf1b333quFCxdWaO0AAKDqshljTGUXgVtTRkaG7Ha70tPT5ePjU9nlAACAYijJ5zdnFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERAAAAlgiKAAAAsERQBAAAgCWCIgAAACwRFAEAAGCJoAgAAABLBEUAAABYIigCAADAEkERQJW3dOlS2Ww2p6Vhw4bq0aOHPvvsM6e+1/a7ehk9erSj39y5c52ec3NzU/PmzTVu3DilpKRIknr06FHk9gqWuXPnlvk+BwYGOtVb3fXo0UPt2rWr7DIAXMO1sgsAgOKKjY1VmzZtZIxRSkqK3n33XfXv31/r1q1T//79Hf2GDh2q6dOnF1q/YcOGhdo2btwou92uzMxMbd68WfPmzVNCQoKSkpIUExOjjIwMR9/PP/9cr732mqOOAs2aNSvjPQWAqoGgCOCW0a5dO4WGhjoe9+nTR3Xr1tXKlSudgqKvr686d+5crG2GhISoQYMGkqRevXopNTVVsbGx2r59uyIiIpz6/vjjj5Z1oPQuXryomjVrVnYZAK6DS88Ablmenp5yd3eXm5tbmW2zIACeOnWqzLZZYPTo0apVq5b+85//qGfPnvL29lbDhg01adIkXbx4sch1L1++rOnTp+uee+6R3W5XvXr1FBYWpk8//dSpX8+ePR1nXa9mjNEdd9yhhx9+uFi1Tp06Vd7e3k5nVAsMHz5cvr6+ys7OdrStWrVKYWFh8vb2Vq1atdS7d2/t3bvXcv9/+OEHRUZGqnbt2urZs6dTn23btqlz587y8vJS06ZN9f/+3/9Tbm5usWoGUPYIitXA2bNnNXLkSNntdtntdo0cOVLnzp27bv/s7Gz99re/1d133y1vb281adJEjz/+uE6ePFlxRQOlkJubq5ycHGVnZ+v48eOaOnWqLly4oKioKKd+xhjl5OQUWq4NT1YOHTokSWrVqlW57EN2drYeeugh9ezZU3//+981adIkLVq0SMOHDy9yvaysLP3yyy+aMWOG/v73v2vlypXq2rWrBg8erGXLljn6TZkyRQcOHNCXX37ptP6GDRt08OBBTZw4sVh1jhkzRhcvXtQnn3zi1H7u3Dl9+umn+s1vfuMI6L///e/12GOPqW3btvrkk0+0fPlynT9/Xt26ddO+ffuc1r9y5YoGDBigX/3qV/r000/1u9/9zvFcSkqKHn30UY0YMUKffvqphg4dqtdee01TpkwpVs0AyoHBLa9Pnz6mXbt2JiEhwSQkJJh27dqZfv36Xbf/uXPnTK9evcyqVavMjz/+aHbs2GE6depkQkJCSvS66enpRpJJT0+/2V0AihQbG2skFVo8PDxMTEyMU1+rfgXL8uXLHf3mzJljJJmUlBSTnZ1tzp49az755BPj7e1tHnvssSLrSExMLNV+jBo1ykgyb7/9tlP766+/biSZ7du3O9oCAgLMqFGjrrutnJwck52dbcaOHWvuvfdeR3tubq5p0aKFGThwoFP/vn37mpYtW5q8vLxi13vfffeZ8PBwp7aYmBgjyfzwww/GGGOOHj1qXF1dzbPPPuvU7/z586Zx48Zm2LBhjraC/f/www8LvVb37t2NJPPpp586tY8bN87UqFHDHDlypNh1AyhaST6/uUfxFrd//35t3LhRO3fuVKdOnSRJS5YsUVhYmA4cOKDWrVsXWsdutys+Pt6p7c9//rPuv/9+HT16VM2bN6+Q2oGSWrZsmYKDgyVJqampWrt2rSZOnKjc3FxNmjTJ0W/YsGF6/vnnC63fokWLQm2NGzd2evzAAw/oL3/5SxlX7mzEiBFOj6OiojR79mxt2bJFXbp0ue56q1ev1ltvvaV//etfunDhgqPd09PT8d81atTQpEmT9Pzzzzt+ng8ePKiNGzfqj3/8o2w2W7HrfOKJJ/Tss886/S6JjY1Vx44dHSOUN23apJycHD3++OPKyclxqql79+7asmVLoe0OGTLE8vVq166tAQMGOLVFRUVpyZIl+vrrr/Wb3/ym2LUDKBtcer7F7dixQ3a73RESJalz586y2+1KSEgo9nbS09Nls9lUp06dcqgSKBvBwcEKDQ1VaGio+vTpo0WLFikyMlIvvPCC0+0WDRs2dPS7eqlXr16hbX7xxRdKTEzUpk2bNGTIEH399dd69tlny20fXF1dVb9+fae2grCalpZ23fXi4uI0bNgwNW3aVB999JF27NihxMREjRkzRpcvX3bqO2bMGHl5eWnhwoWSpAULFsjLy0tjxowpUa0jRoyQh4eHli5dKknat2+fEhMT9cQTTzj6FNzL2bFjR7m5uTktq1atUmpqqtM2a9asKR8fH8vX8/X1LdRW8N7s3JmmlSulrVslblkEKg5nFG9xKSkpatSoUaH2Ro0aOeaCu5HLly9r5syZioqKuu4vcCn/HqmsrCzHY6ub3IGK1r59e23atEk//fST7r///hKv36FDB8eo5wcffFC9e/fW4sWLNXbsWHXs2LGsy1VOTo7S0tKcwmLBz+q1AfJqH330kYKCgrRq1Sqns4JX/0wWsNvtGjVqlN5//33NmDFDsbGxioqKKvEfgnXr1tXAgQO1bNkyx7RAnp6eeuyxxxx9Ct67NWvWKCAg4IbbLOqMptUAok8+yX9vFiyorwUL8tuaNZPeflsaPLgkewOgNDijWEVdOxmw1bJ7925J1r94jTHFusSUnZ2tRx99VHl5eYqJiSmyb3R0tGPAjN1ul7+/f+l2DihDSUlJkqznSCwpm82mBQsWyMXFRS+99NJNb+96Pv74Y6fHK1askJQ/6XRRtbm7uzv9XKekpBQa9Vxg8uTJSk1N1dChQ3Xu3DmnS/Ml8cQTT+jkyZNav369PvroIw0aNMgpcPbu3Vuurq46ePCg5VnckkwjdP78ea1bt87xOC5OmjNnhfI/qh5wtJ84IQ0dmv88gPLFGcUqatKkSXr00UeL7BMYGKjvv//e8q/wM2fOWF7GuVp2draGDRumQ4cO6Z///GeRZxMladasWZo2bZrjcUZGBmERFerf//634z64tLQ0xcXFKT4+XoMGDVJQUJCj36lTp7Rz585C6/v4+Kht27ZFvsadd96p8ePHKyYmRtu3b1fXrl3LdB/c3d01b948ZWZmqmPHjkpISNBrr72mvn37Fvla/fr1U1xcnJ555hkNHTpUx44d06uvvio/Pz/9/PPPhfq3atVKffr00YYNG9S1a1d16NChVPVGRkaqWbNmeuaZZ5SSkuJ02VnK/z30yiuvaPbs2frvf//rmNvy1KlT+vbbb+Xt7e00srko9evX19NPP62jR4+qZctWGjVqvaQlkp6W9H/3Thsj2WzS1KnSwIGSi0updg1AcZT/2BqUp3379hlJZteuXY62nTt3Gknmxx9/vO56V65cMY888oi56667zOnTp0v12ox6RkWxGvVst9vNPffcY+bPn28uX77s6Httv6uXLl26OPoVjHo+c+ZModc7deqUqVWrlomIiLCs42ZGPXt7e5vvv//e9OjRw3h5eZl69eqZp59+2mRmZjr1tRr1/MYbb5jAwEDj4eFhgoODzZIlSxz7YWXp0qVGkvnrX/9aqnoLvPjii0aS8ff3N7m5uZZ9/v73v5uIiAjj4+NjPDw8TEBAgBk6dKj54osvCu2/le7du5u77rrLbN261YSGhho3Nw8j+RnpRSNlm/x4WHjZsuWmdg24LZXk89tmTDEmFkOV1rdvX508eVKLFi2SJI0fP14BAQH6xz/+4ejTpk0bRUdHa9CgQcrJydGQIUP03Xff6bPPPnM681ivXj25u7sX63UzMjJkt9uVnp5+w7ORAPInnF6zZo0yMzMr5PWGDBminTt36vDhw2U6KXlFWLlSumZ6TEsrVkhX3TIJoBhK8vnNpedq4OOPP9bkyZMVGRkpSRowYIDeffddpz4HDhxQenq6JOn48eOO+4Duuecep35btmwp8j4pAFVbVlaWvvvuO3377bdau3at5s+ff8uFREny8yvbfgBKh6BYDdSrV08fffRRkX2uPnEcGBhYrG+oAFA8eXl5ysvLK7KPq2vF/LpNTk5WeHi4fHx89NRTT1lO9ZObm1vk7wCbzSaXSr7xr1u3/NHNJ07kX2S+ls2W/3y3bhVfG3A7YdQzANykMWPGFJpD8NpFkpYuXVrul50L/hBMT0/Xe++9Zxn4evbsWWStLVu2LNcai8PFJX8KHCk/FF6t4PFbbzGQBShv3KOIUuMeRSDf4cOHC00sfa2STBNT3g4cOKDz589f93kPDw/dfffdFVjR9cXFSVOmSMeP/1+bv39+SGQeRaB0SvL5TVBEqREUAVSE3Fxp2zYpOTn/nsRu3TiTCNwMBrMAAKoNFxeJMXZA5eAeRQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUAQAAYImgCAAAAEsERQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUAQAAYImgCAAAAEsERQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUAQAAYImgCAAAAEsERQAAAFgiKAIAAMASQREAAACWCIoAAACwRFAEAACAJYIiAAAALBEUAQAAYKnEQXHp0qWy2WxOS8OGDdWjRw999tlnTn2v7Xf1Mnr0aEe/uXPnOj3n5uam5s2ba9y4cUpJSZEk9ejRo8jtFSxz5869qTfESmBgoFO9KLmtW7fKZrNp69atlV0KAAAoJtfSrhgbG6s2bdrIGKOUlBS9++676t+/v9atW6f+/fs7+g0dOlTTp08vtH7Dhg0LtW3cuFF2u12ZmZnavHmz5s2bp4SEBCUlJSkmJkYZGRmOvp9//rlee+01Rx0FmjVrVtpdQjm67777tGPHDrVt27aySwEAAMVU6qDYrl07hYaGOh736dNHdevW1cqVK52Coq+vrzp37lysbYaEhKhBgwaSpF69eik1NVWxsbHavn27IiIinPr++OOPlnWgasnOzpbNZpOPj0+x/z8AAABVQ5ndo+jp6Sl3d3e5ubmV1SYdAfDUqVNlts0Co0ePVq1atfSf//xHPXv2lLe3txo2bKhJkybp4sWLRa57+fJlTZ8+Xffcc4/sdrvq1aunsLAwffrpp079evbs6TjrejVjjO644w49/PDDxa7XGKPf//73CggIkKenp0JDQxUfH68ePXqoR48eTn0zMjI0Y8YMBQUFyd3dXU2bNtXUqVN14cIFp342m02TJk3S8uXLFRwcrJo1a6pDhw6FbiGQpJ9//llRUVFq1KiRPDw8FBwcrCVLljj1Kbi8vHz5ck2fPl1NmzaVh4eH/vd///e6l5537dql/v37q379+vL09FTLli01derUYr8vAACg/JT6jGJubq5ycnJkjNGpU6f0hz/8QRcuXFBUVJRTP2OMcnJyCq3v4uIim81W5GscOnRIktSqVavSllmk7OxsPfTQQ3rqqac0c+ZMJSQk6LXXXtORI0f0j3/847rrZWVl6ZdfftGMGTPUtGlTXblyRV988YUGDx6s2NhYPf7445KkKVOmaODAgfryyy/Vq1cvx/obNmzQwYMH9c477xS71tmzZys6Olrjx4/X4MGDdezYMT355JPKzs52en8uXryo7t276/jx43rxxRfVvn17/ec//9HLL7+sH374QV988YXT+/75558rMTFRr7zyimrVqqU333xTgwYN0oEDB9SiRQtJ0r59+xQeHq7mzZtr3rx5aty4sTZt2qQXXnjBstZZs2YpLCxMCxcuVI0aNdSoUSPHvaZX27Rpk/r376/g4GDNnz9fzZs31+HDh7V58+Zivy8AAKAcmRKKjY01kgotHh4eJiYmxqmvVb+CZfny5Y5+c+bMMZJMSkqKyc7ONmfPnjWffPKJ8fb2No899liRdSQmJpZ0F4wxxowaNcpIMm+//bZT++uvv24kme3btzvaAgICzKhRo667rZycHJOdnW3Gjh1r7r33Xkd7bm6uadGihRk4cKBT/759+5qWLVuavLy8YtX6yy+/GA8PDzN8+HCn9h07dhhJpnv37o626OhoU6NGjULvy5o1a4wks379ekebJOPr62syMjIcbSkpKaZGjRomOjra0da7d2/TrFkzk56e7rTN8ePHG0nm8OHDxhhjtmzZYiSZBx54oNA+FDy3ZcsWR1vLli1Ny5YtzaVLl4r1PgAAgJuXnp5uJBX6XLdS6kvPy5YtU2JiohITE7VhwwaNGjVKEydO1LvvvuvUb9iwYY5+Vy8PPfRQoW02btxYbm5uqlu3roYNG6aQkBD95S9/KW2JxTJixAinxwVnRLds2VLkeqtXr1aXLl1Uq1Ytubq6ys3NTR988IH279/v6FOjRg1NmjRJn332mY4ePSpJOnjwoDZu3KhnnnnmhmdUC+zcuVNZWVkaNmyYU3vnzp0VGBionJwcjRw5Una7XS+99JJ8fHwc7QVL7969nS79FowOT01NVfPmzdWrVy/t2rVLvr6+atSokY4cOSIp/zL7l19+qUGDBqlmzZpO24yMjJQk7d6926muIUOG3HCffvrpJx08eFBjx46Vp6en03O5udLWrdLKlfn/5uYW620CAABlrNRBMTg4WKGhoQoNDVWfPn20aNEiRUZG6oUXXtC5c+cc/Ro2bOjod/VSr169Qtv84osvlJiYqE2bNmnIkCH6+uuv9eyzz5a2xBtydXVV/fr1ndoaN24sSUpLS7vuenFxcRo2bJiaNm2qjz76SDt27FBiYqLGjBmjy5cvO/UdM2aMvLy8tHDhQknSggUL5OXlpTFjxhS7zoJafH19Cz3n6+urffv2KSkpSRs3bpSfn5/OnTunhg0bys3NzbHUrl1bxhilpqZK+r/L+VFRUdq+fbsCAwMVGRmpM2fOyMPDQ5cuXXK8dk5Ojv785z87bc/NzU1Dhw61fK/8/PxuuE9nzpyRVHiUelycFBgoRURIUVH5/wYG5rcDAICKVep7FK20b99emzZt0k8//aT777+/xOt36NDBMer5wQcfVO/evbV48WKNHTtWHTt2LMtSJUk5OTlKS0tzCosF99JdGyCv9tFHHykoKEirVq1yOiuYlZVVqK/dbteoUaP0/vvva8aMGYqNjVVUVJTq1KlT7DoLarEa1HPs2DGdPXtWGzZsUKdOndSsWTN5eHjo4MGDWr16tQIDA536F7y/UVFRGjFihHx8fHTXXXdp/vz5+uCDD/T999879a9bt65cXFw0cuRITZw40em5zMxMRUREOM4sFijOmdKC6ZGOHz/uaIuLk4YOla4Z+6MTJ/Lb16yRBg++4aYBAEAZKdNvZklKSpJkPUdiSdlsNi1YsEAuLi566aWXbnp71/Pxxx87PV6xYoUkFRpJfG1t7u7uToEoJSWl0KjnApMnT1ZqaqqGDh2qc+fOadKkSSWqsVOnTvLw8NCqVauc2nfu3KmTJ0/KxcVFnTp1kiT169dPycnJql27ts6fP1/oTO61wVGSrly5osWLF8tut6tDhw5Oz9WsWVMRERHau3evWrdurVatWjmWO+64Q5Iszw7fSKtWrdSyZUt9+OGHysrKUm6uNGVK4ZAo/V/b1KlchgYAoCKV+oziv//9b8do5rS0NMXFxSk+Pl6DBg1SUFCQo9+pU6e0c+fOQuv7+PjccPLlO++8U+PHj1dMTIy2b9+url27lrZcS+7u7po3b54yMzPVsWNHx6jnvn37Fvla/fr1U1xcnJ555hkNHTpUx44d06uvvio/Pz/9/PPPhfq3atVKffr00YYNG9S1a9dCYexG6tWrp2nTpik6Olp169bVoEGDdPz4cf3ud79T7dq1nUaVT506VX/729/0ww8/aM2aNfL391deXp6OHj2qzZs3a/r06Y5QKUmLFi1STEyM/Pz8FB8f7zjjeLW3335bXbt2VXBwsE6cOFGi2ouyYMEC9e/fX507d1afPs/p+PHmko5K2iTJOcAbIx07Jm3bJhWR4QEAQFkq6UgZq1HPdrvd3HPPPWb+/Pnm8uXLjr7X9rt66dKli6NfwajnM2fOFHq9U6dOmVq1apmIiAjLOm5m1LO3t7f5/vvvTY8ePYyXl5epV6+eefrpp01mZqZTX6tRz2+88YYJDAw0Hh4eJjg42CxZssSxH1aWLl1qJJm//vWvxaqvYFtFLX/6059M48aNTa1atZzWzczMNHXr1jUNGzY07u7uxm63m7vvvts899xzJiUlxdFPkvnNb35jduzYYcaMGWMCAwPNqVOnLPf30KFDZtSoUaZJkybGzc3NNGjQwISGhjqNmioY2bx69epC+2M16tmY/JHbffv2NTVr2o3kYaSWRnrO5EfDwsuKFcV6+wAAwHWUZNSzzRiri33V3+jRo7VmzRplZmZWyOsNGTJEO3fu1OHDh4s1KXlqaqpj4ElR7r77brm4uBQaRFOnTh396U9/0hNPPFHsGu+8806NGTNGs2bNKlb/jIwM2e12paeny8fHp9ivY2Xr1vyBKzeyZQtnFAEAuBkl+fwu08EscJaVlaXvvvtO3377rdauXav58+cX+5trGjRo4HQZ+F//+pdWrlyp8PBw+fj46MCBA3rzzTdVu3ZtnT17Vt9++61jANGuXbuUnp6u8PDwEtVrjLEckFMRunWTmjXLH7hi9aeLzZb/fLduFV8bAAC3q2oXFPPy8pSXl1dkH1fXitnt5ORkR7B76qmnLKf6yc3NLfQVf1ez2WxycXGRt7e3du/erQ8++EDnzp2T3W5Xjx499Prrr2vq1KkaN26cFi1aJEkaP368+vXrp9atWzu206ZNG0VHR2vQoEG6cOGCXn/9dQ0YMEB+fn5KS0tTTEyMjh8/rl//+tdl/0YUg4uL9Pbb+aObbTbnsFgwZuitt/L7AQCAilHtLj2PHj36hpN0V6Vd7tGjh7766qvrPh8QEKDDhw8XuY1ffvlFkydP1rp16yRJAwYM0Lvvvus0BY/NZlNsbKxGjx6ty5cvKyoqSrt27VJqaqrq16+vjh076qWXXirRNERleem5QFxc/ujnq2bNkb9/fkhkahwAAG5eST6/q11QPHz48A3v7QsNDa2gam7swIEDOn/+/HWf9/Dw0N13312BFRVfeQRFKX8KnG3bpORkyc8v/3IzZxIBACgbt3VQRMUpr6AIAADKT0k+v8t0wm0AAABUHwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAAS66VXQBuXQVf6pORkVHJlQAAgOIq+NwuzpfzERRRagXfUe3v71/JlQAAgJI6f/687HZ7kX34rmeUWl5enk6ePKnatWvLZrNVdjklkpGRIX9/fx07dozvqa5COC5VF8em6uLYVE1V+bgYY3T+/Hk1adJENWoUfRciZxRRajVq1FCzZs0qu4yb4uPjU+V+gMFxqco4NlUXx6ZqqqrH5UZnEgswmAUAAACWCIoAAACwRFDEbcnDw0Nz5syRh4dHZZeCq3Bcqi6OTdXFsamaqstxYTALAAAALHFGEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERRRbcXExCgoKEienp4KCQnRtm3biuz/1VdfKSQkRJ6enmrRooUWLlxYQZXeXkpyXOLi4vTggw+qYcOG8vHxUVhYmDZt2lSB1d5eSvozU+Cbb76Rq6ur7rnnnvIt8DZW0mOTlZWl2bNnKyAgQB4eHmrZsqU+/PDDCqr29lHS4/Lxxx+rQ4cOqlmzpvz8/PTEE08oLS2tgqotJQNUQ3/961+Nm5ubWbJkidm3b5+ZMmWK8fb2NkeOHLHs/9///tfUrFnTTJkyxezbt88sWbLEuLm5mTVr1lRw5dVbSY/LlClTzP/8z/+Yb7/91vz0009m1qxZxs3NzXz33XcVXHn1V9JjU+DcuXOmRYsWJjIy0nTo0KFiir3NlObYDBgwwHTq1MnEx8ebQ4cOmV27dplvvvmmAquu/kp6XLZt22Zq1Khh3n77bfPf//7XbNu2zdx1113mkUceqeDKS4agiGrp/vvvNxMmTHBqa9OmjZk5c6Zl/xdeeMG0adPGqe2pp54ynTt3Lrcab0clPS5W2rZta373u9+VdWm3vdIem+HDh5uXXnrJzJkzh6BYTkp6bDZs2GDsdrtJS0uriPJuWyU9Ln/4wx9MixYtnNreeecd06xZs3KrsSxw6RnVzpUrV7Rnzx5FRkY6tUdGRiohIcFynR07dhTq37t3b+3evVvZ2dnlVuvtpDTH5Vp5eXk6f/686tWrVx4l3rZKe2xiY2N18OBBzZkzp7xLvG2V5tisW7dOoaGhevPNN9W0aVO1atVKM2bM0KVLlyqi5NtCaY5LeHi4jh8/rvXr18sYo1OnTmnNmjV6+OGHK6LkUnOt7AKAspaamqrc3Fz5+vo6tfv6+iolJcVynZSUFMv+OTk5Sk1NlZ+fX7nVe7sozXG51rx583ThwgUNGzasPEq8bZXm2Pz888+aOXOmtm3bJldXPkrKS2mOzX//+19t375dnp6eWrt2rVJTU/XMM8/ol19+4T7FMlKa4xIeHq6PP/5Yw4cP1+XLl5WTk6MBAwboz3/+c0WUXGqcUUS1ZbPZnB4bYwq13ai/VTtuTkmPS4GVK1dq7ty5WrVqlRo1alRe5d3WintscnNzFRUVpd/97ndq1apVRZV3WyvJz01eXp5sNps+/vhj3X///XrooYc0f/58LV26lLOKZawkx2Xfvn2aPHmyXn75Ze3Zs0cbN27UoUOHNGHChIootdT4MxDVToMGDeTi4lLor7rTp08X+uuvQOPGjS37u7q6qn79+uVW6+2kNMelwKpVqzR27FitXr1avXr1Ks8yb0slPTbnz5/X7t27tXfvXk2aNElSfjgxxsjV1VWbN2/Wr371qwqpvborzc+Nn5+fmjZtKrvd7mgLDg6WMUbHjx/XnXfeWa413w5Kc1yio6PVpUsXPf/885Kk9u3by9vbW926ddNrr71WZa9ccUYR1Y67u7tCQkIUHx/v1B4fH6/w8HDLdcLCwgr137x5s0JDQ+Xm5lZutd5OSnNcpPwziaNHj9aKFSuq/L08t6qSHhsfHx/98MMPSkpKciwTJkxQ69atlZSUpE6dOlVU6dVeaX5uunTpopMnTyozM9PR9tNPP6lGjRpq1qxZudZ7uyjNcbl48aJq1HCOXS4uLpL+7wpWlVRZo2iA8lQwbcEHH3xg9u3bZ6ZOnWq8vb3N4cOHjTHGzJw504wcOdLRv2B6nOeee87s27fPfPDBB0yPUw5KelxWrFhhXF1dzYIFC0xycrJjOXfuXGXtQrVV0mNzLUY9l5+SHpvz58+bZs2amaFDh5r//Oc/5quvvjJ33nmnefLJJytrF6qlkh6X2NhY4+rqamJiYszBgwfN9u3bTWhoqLn//vsraxeKhaCIamvBggUmICDAuLu7m/vuu8989dVXjudGjRplunfv7tR/69at5t577zXu7u4mMDDQvPfeexVc8e2hJMele/fuRlKhZdSoURVf+G2gpD8zVyMolq+SHpv9+/ebXr16GS8vL9OsWTMzbdo0c/HixQquuvor6XF55513TNu2bY2Xl5fx8/MzI0aMMMePH6/gqkvGZkxVPt8JAACAysI9igAAALBEUAQAAIAlgiIAAAAsERQBAABgiaAIAAAASwRFAAAAWCIoAgAAwBJBEQAAAJYIigAAALBEUAQAAIAlgiIAAAAsERQBAABg6f8D462DrCW935wAAAAASUVORK5CYII=",
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# OPTIONAL: Enable the logger for tracking information\n",
"import logging\n",
"from transformers import BertTokenizer, BertModel\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"\n",
"tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
"# Load the tokenizer for the pre-trained model\n",
"\n",
"\n",
"\n",
"def tokenize_parse(sentence,model):\n",
" encoded_dict = tokenizer.encode_plus(\n",
" sentence, \n",
" add_special_tokens=True, # Add '[CLS]' and '[SEP]'\n",
" max_length=64, # Adjust sentence length\n",
" pad_to_max_length=True, # Pad/truncate sentences\n",
" return_attention_mask=True,# Generate attention masks\n",
" return_tensors='pt', # Return PyTorch tensors\n",
" )\n",
" \n",
" \n",
" input_ids = encoded_dict['input_ids']\n",
" \n",
" # Construct an attention mask (identifying padding/non-padding).\n",
" attention_masks = (encoded_dict['attention_mask'])\n",
" with torch.no_grad():\n",
" outputs = model(input_ids,attention_masks)\n",
" embeddings = outputs.last_hidden_state[0]\n",
" return input_ids, embeddings\n",
"\n",
"\n",
"model = BertModel.from_pretrained(\"bert-base-uncased\")\n",
"\n",
"custom_text = \"Let the dogs out to play with each other\"\n",
"custom_text2 = \"There is a new play in the national theatre \"\n",
"word_1 = 'play'\n",
"\n",
"input_ids1, output_embeddings1 = tokenize_parse(custom_text,model)\n",
"\n",
"input_ids2, output_embeddings2 = tokenize_parse(custom_text2,model)\n",
"\n",
"input_ids_word1, output_embeddings_word1 = tokenize_parse(word_1,model)\n",
"\n",
"word_id1 = tokenizer.decode(token_ids=input_ids1[0]).split().index('play')\n",
"word_id2 = tokenizer.decode(token_ids=input_ids2[0]).split().index('play')\n",
"word_id_w1 = tokenizer.decode(token_ids=input_ids_word1[0]).split().index('play')\n",
"\n",
"vectors = [output_embeddings1[word_id1],output_embeddings2[word_id2],output_embeddings_word1[word_id_w1]]\n",
"words = ['BERT_play_noun','BERT_play_verb', 'BERT_play_generic']\n",
"\n",
"plot_embedding_vectors_multi_type(vectors,words)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note**: Above we demonstrate the vector operations using just the first two dimensions of the embedding vectors. These may not be very representative. We present below more ways of visualising embeddings in 2D space.\n",
" "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Visualisation of embeddings\n",
"\n",
"In all cases visited above we have embedding vectors with high dimensionlity. We will see below how we can use principal component analysis (PCA) to visualise embeddings in 2D space. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# PCA visualisation\n"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "cf88f936ea334333b7dfa62a6af2b688",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/400000 [00:00"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"from sklearn.decomposition import PCA\n",
"import matplotlib.pyplot as plt\n",
"import random\n",
"\n",
"def display_pca_scatterplot_2D(words=None, topn=5):\n",
"\n",
" word_vectors = ([get_word(w) for w in words])\n",
" neighborhoods = []\n",
" new_words = []\n",
" for word in words:\n",
" neighborhood = []\n",
" neighbors = closest(get_word(word),n=topn)\n",
" for n in neighbors:\n",
" n_vector = get_word(n[0])\n",
" n_word = n[0]\n",
" new_words.append(n_word)\n",
" print(n_word)\n",
" neighborhood.append(n_vector.numpy())\n",
" \n",
" neighborhoods.extend(neighborhood)\n",
" \n",
" word_vectors = np.array(neighborhoods)\n",
" words = new_words\n",
" two_dim = PCA(random_state=0).fit_transform(word_vectors)[:,:2]\n",
" \n",
" principalDf = pd.DataFrame(data = two_dim\n",
" , columns = ['principal component 1', 'principal component 2'])\n",
" \n",
" finalDf = principalDf\n",
" finalDf['word']=words\n",
"\n",
" fig = plt.figure(figsize = (8,8))\n",
" ax = fig.add_subplot(1,1,1) \n",
" ax.set_xlabel('Principal Component 1', fontsize = 15)\n",
" ax.set_ylabel('Principal Component 2', fontsize = 15)\n",
" ax.set_title('Visualisation via 2 component PCA', fontsize = 20)\n",
"\n",
" hexadecimal_alphabets = '0123456789ABCDEF'\n",
" color = [\"#\" + ''.join([random.choice(hexadecimal_alphabets) for j in range(6)]) for i in range(len(words))]\n",
" for i,target in enumerate(words):\n",
" \n",
" indicesToKeep = finalDf['word'] == target\n",
" if i%topn==0:\n",
" si = 150\n",
" else:\n",
" si = 50\n",
" ax.scatter(finalDf.loc[indicesToKeep, 'principal component 1']\n",
" , finalDf.loc[indicesToKeep, 'principal component 2']\n",
" , c = color[i//topn]\n",
" , s = si)\n",
" \n",
" for i, txt in enumerate(words):\n",
" indicesToKeep = finalDf['word'] == txt\n",
" x = finalDf.loc[indicesToKeep, 'principal component 1'].iloc[0]\n",
" y = finalDf.loc[indicesToKeep, 'principal component 2'].iloc[0]\n",
" ax.annotate(txt, (x, y))\n",
" ax.grid()\n",
"\n",
"words=['queen','king','man','woman', 'mother', 'father']\n",
"\n",
"display_pca_scatterplot_2D(words,10)\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.18"
}
},
"nbformat": 4,
"nbformat_minor": 2
}