{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Practical 12 \n",
"\n",
"This practical has two parts (jupyter notebooks), **Part I: Word embeddings** and **Part II: Transformers**.\n",
"\n",
"# Part I: Word embeddings\n",
"\n",
"This Jupyter Notebook consists of the following parts:\n",
"1. [Word Embeddings](#word_embeddings) \n",
" 1. [Word2Vec](#word2vec) \n",
" 2. [Word2Vec Architectures](#word2vec_arch) \n",
" 3. [Word Embeddings Visualisation](#visual)\n",
"2. [Exploring Word Vectors with GloVe](glove)\n",
" 1. [Loading Word Vectors](#loading)\n",
" 2. [Finding Closest Vectors](#finding)\n",
" 3. [Word Analogies with Vector Arithmetic](#analogies)\n",
"3. [Motivation for Part II: Transformers](#transformers)\n",
"\n",
"\n",
"Check out a supplementary jupyter notebook for Part I, if interested in Skip-Gram implementation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"# Word embeddings"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Word2Vec\n",
"\n",
"There are two classes of vector models: count-based (TF-IDF, Bag-of-Words) and neural-based. In this practical we will be focusing on neural word embeddings, i.e. word embeddings learned by a neural network.\n",
"\n",
"**Main idea:** to use neural architectures that are predicting (not counting) the next word or a context of a given word.\n",
"\n",
"One of the most known such models is **Word2Vec**. It is based on a neural network that is predicting the probability of a word given it's context. It was created by Mikolov et al. (2013). Here are the main papers on the topic:\n",
"\n",
"* [Efficient Estimation of Word Representations in Vector Space](https://arxiv.org/pdf/1301.3781.pdf)\n",
"* [Distributed Representations of Words and Phrases and their Compositionality](https://arxiv.org/abs/1310.4546)\n",
"\n",
"These vectors are ususally reffered to as **_distributed representations of words_** or **_word embeddings_**.\n",
"\n",
"_As word embeddings are a key building block of deep learning models for NLP, word2vec is often assumed to belong to the same group. Technically however, word2vec is not be considered to be part of deep learning, as its architecture is neither deep nor uses non-linearities_ \n",
"\n",
"*a quote from [Sebastian Ruder's blog](https://ruder.io/word-embeddings-1/) \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Word2Vec Architectures\n",
"\n",
"There are two Word2Vec architectures: Skip-Gram and CBOW.\n",
"\n",
"**Skip-Gram** predicts context words given the central word. Skip-Gram with negative sampling is the most popular approach.\n",
"\n",
"**CBOW (Continuous Bag-of-Words)** predicts the central word from the sum of context vectors. This simple sum of word vectors is called \"bag of words\", which gives the name for the model.\n",
"\n",
"\n",
"\n",
"\n",
"*the image is taken for [Lena Voita's blog](https://lena-voita.github.io/nlp_course/word_embeddings.html#main_content) \n",
"*if interested, please, check it out, it is very informative and illustrative"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How does it work?\n",
"\n",
"Word2vec takes a large text corpus as input and maps each word to a vector, producing word coordinates as output. It first creates a dictionary by training on the input text data, and then calculates a vector representation of the words. The vector representation is learned on contextual proximity: words that occur in the text next to the same words (and therefore, according to the distributive hypothesis, have a similar meaning) will have close coordinates in the vector representation. \n",
"\n",
"To calculate the proximity of words, usually the cosine or euclidean distances between vectors are used.\n",
"\n",
"Using distributed representations you can build semantic proportions (also known as analogies) and solve examples like:\n",
"\n",
"*king: male = queen: female*\n",
" $\\Rightarrow$\n",
"*king - man + woman = queen*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Word Embeddings Visualization\n",
"\n",
"Go to https://projector.tensorflow.org/ and visualize Word2Vec embeddings. \n",
"\n",
"Original Word2Vec repository: https://code.google.com/archive/p/word2vec/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"\n",
"# Exploring Word Vectors with GloVe:\n",
"\n",
"As we have seen, the word2vec algorithms (such as Skip-Gram) predicts words in a context (e.g. what is the most likely word to appear in \"the cat ? the mouse\"), while GloVe vectors are based on global counts across the corpus — [see How is GloVe different from word2vec?](https://www.quora.com/How-is-GloVe-different-from-word2vec) on Quora for some better explanations.\n",
"\n",
"The best feature of GloVe is that multiple sets of pre-trained vectors are easily available for [download](https://nlp.stanford.edu/projects/glove/), so that's what we'll use here.\n",
"\n",
"Part II of this notebook is taken from [practical-pytorch tutorials](https://github.com/spro/practical-pytorch/blob/master/glove-word-vectors/glove-word-vectors.ipynb)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing torchtext"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"#! pip install torchtext"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Loading Word Vectors\n",
"Torchtext includes functions to download GloVe (and other) embeddings"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"from torchtext.vocab import GloVe"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
".vector_cache/glove.6B.zip: 862MB [02:42, 5.32MB/s] \n",
"100%|█████████▉| 399999/400000 [00:07<00:00, 55651.76it/s]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded 400000 words\n"
]
}
],
"source": [
"glove = GloVe(name='6B', dim=50)\n",
"print('Loaded {} words'.format(len(glove.itos)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Loaded 400000 words\n",
"The returned GloVe object includes attributes:\n",
"\n",
"- stoi string-to-index returns a dictionary of words to indexes\n",
"- itos index-to-string returns an array of words by index\n",
"- vectors returns the actual vectors. To get a word vector get the index to get the vector:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def get_word(word):\n",
" return glove.vectors[glove.stoi[word]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"## Finding Closest Vectors\n",
"\n",
"Going from word → vector is easy enough, but to go from vector → word takes more work. Here I'm (naively) calculating the distance for each word in the vocabulary, and sorting based on that distance:\n",
"\n",
"Anyone with a suggestion for optimizing this, please let me know!"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"from tqdm import tqdm_notebook as tqdm\n",
"def closest(vec, n=10):\n",
" \"\"\"\n",
" Find the closest words for a given vector\n",
" \"\"\"\n",
" all_dists = [(w, torch.dist(vec, get_word(w))) for w in tqdm(glove.itos)]\n",
" return sorted(all_dists, key=lambda t: t[1])[:n]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This will return a list of (word, distance) tuple pairs. Here's a helper function to print that list:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"def print_tuples(tuples):\n",
" for tuple in tuples:\n",
" print('(%.4f) %s' % (tuple[1], tuple[0]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now using a known word vector we can see which other vectors are closest:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
":6: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0\n",
"Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`\n",
" all_dists = [(w, torch.dist(vec, get_word(w))) for w in tqdm(glove.itos)]\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "7b6796e5aca840a88f2e25ade0a5f5b0",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/400000 [00:00\n",
"## Word Analogies with Vector Arithmetic\n",
"The most interesting feature of a well-trained word vector space is that certain semantic relationships (beyond just close-ness of words) can be captured with regular vector arithmetic.\n",
"\n",
"\n",
"\n",
"(image borrowed from a slide from [a slide from Omer Levy and Yoav Goldberg](https://levyomer.wordpress.com/2014/04/25/linguistic-regularities-in-sparse-and-explicit-word-representations/))"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"# In the form w1 : w2 :: w3 : ?\n",
"def analogy(w1, w2, w3, n=10, filter_given=True):\n",
" # w2 - w1 + w3 = w4\n",
" closest_words = closest(get_word(w2) - get_word(w1) + get_word(w3), n=n)\n",
" print('\\n[%s - %s + %s = ?]' % (w1, w2, w3))\n",
" # Optionally filter out given words\n",
" if filter_given:\n",
" closest_words = [t for t in closest_words if t[0] not in [w1, w2, w3]]\n",
" \n",
" print_tuples(closest_words[:n])"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
":6: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0\n",
"Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`\n",
" all_dists = [(w, torch.dist(vec, get_word(w))) for w in tqdm(glove.itos)]\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "98c394174809481f9705bf6edf088ba3",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/400000 [00:00\n",
"# Motivation for Part II: Transformers"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Word embeddings that we discussed above have one major limitation – they are **static**. This means that each word has a vector that does not change given different contexts.\n",
"\n",
"While **contextualized** word embeddings can give words different embeddings based on the meaning they carry in the context of the sentence."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"_\"If we’re using this GloVe representation, then the word “stick” would be represented by this vector no-matter what the context was. “Wait a minute” said a number of NLP researchers (Peters et. al., 2017, McCann et. al., 2017, and yet again Peters et. al., 2018 in the ELMo paper ), “stick”” has multiple meanings depending on where it’s used. Why not give it an embedding based on the context it’s used in – to both capture the word meaning in that context as well as other contextual information?”. And so, contextualized word-embeddings were born...\"_"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"*the image and the quote are taken from [Jay Alammar's blogpost](https://jalammar.github.io/illustrated-bert/) about BERT, ELMo and co."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Contextualized embeddings can be captured with Transformer-based models like BERT that are usually trained to predict randomly masked words. \n",
"See **Part II** of this practical to learn more about transformers!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}