assignment 3 hello vectors

Assignment 3: Hello Vectors

Assignment 3: hello vectors #.

Welcome to this week’s programming assignment of the specialization. In this assignment we will explore word vectors. In natural language processing, we represent each word as a vector consisting of numbers. The vector encodes the meaning of the word. These numbers (or weights) for each word are learned using various machine learning models, which we will explore in more detail later in this specialization. Rather than make you code the machine learning models from scratch, we will show you how to use them. In the real world, you can always load the trained word vectors, and you will almost never have to train them from scratch. In this assignment you will

Predict analogies between words.

Use PCA to reduce the dimensionality of the word embeddings and plot them in two dimensions.

Compare word embeddings by using a similarity measure (the cosine similarity).

Understand how these vector space models work.

1.0 Predict the Countries from Capitals #

During the presentation of the module, we have illustrated the word analogies by finding the capital of a country from the country. In this part of the assignment we have changed the problem a bit. You are asked to predict the countries that corresponde to some capitals . You are playing trivia against some second grader who just took their geography test and knows all the capitals by heart. Thanks to NLP, you will be able to answer the questions properly. In other words, you will write a program that can give you the country by its capital. That way you are pretty sure you will win the trivia game. We will start by exploring the data set.

Important Note on Submission to the AutoGrader #

Before submitting your assignment to the AutoGrader, please make sure you are not doing the following:

You have not added any extra print statement(s) in the assignment.

You have not added any extra code cell(s) in the assignment.

You have not changed any of the function parameters.

You are not using any global variables inside your graded exercises. Unless specifically instructed to do so, please refrain from it and use the local variables instead.

You are not changing the assignment code where it is not required, like creating extra variables.

If you do any of the following, you will get something like, Grader not found (or similarly unexpected) error upon submitting your assignment. Before asking for help/debugging the errors in your assignment, check for these first. If this is the case, and you don’t remember the changes you have made, you can get a fresh copy of the assignment by following these instructions .

1.1 Importing the data #

As usual, you start by importing some essential Python libraries and the load dataset. The dataset will be loaded as a Pandas DataFrame , which is very a common method in data science. Because of the large size of the data, this may take a few minutes.

To Run This Code On Your Own Machine: #

Note that because the original google news word embedding dataset is about 3.64 gigabytes, the workspace is not able to handle the full file set. So we’ve downloaded the full dataset, extracted a sample of the words that we’re going to analyze in this assignment, and saved it in a pickle file called word_embeddings_capitals.p

If you want to download the full dataset on your own and choose your own set of word embeddings, please see the instructions and some helper code.

Download the dataset from this page .

Search in the page for ‘GoogleNews-vectors-negative300.bin.gz’ and click the link to download.

You’ll need to unzip the file.

Copy-paste the code below and run it on your local machine after downloading the dataset to the same directory as the notebook.

Now we will load the word embeddings as a Python dictionary . As stated, these have already been obtained through a machine learning algorithm.

Each of the word embedding is a 300-dimensional vector.

Predict relationships among words #

Now you will write a function that will use the word embeddings to predict relationships among words.

The function will take as input three words.

The first two are related to each other.

It will predict a 4th word which is related to the third word in a similar manner as the two first words are related to each other.

As an example, “Athens is to Greece as Bangkok is to ______”?

You will write a program that is capable of finding the fourth word.

We will give you a hint to show you how to compute this.

A similar analogy would be the following:

You will implement a function that can tell you the capital of a country. You should use the same methodology shown in the figure above. To do this, you’ll first compute the cosine similarity metric or the Euclidean distance.

1.2 Cosine Similarity #

The cosine similarity function is:

\(A\) and \(B\) represent the word vectors and \(A_i\) or \(B_i\) represent index i of that vector. Note that if A and B are identical, you will get \(cos(\theta) = 1\) .

Otherwise, if they are the total opposite, meaning, \(A= -B\) , then you would get \(cos(\theta) = -1\) .

If you get \(cos(\theta) =0\) , that means that they are orthogonal (or perpendicular).

Numbers between 0 and 1 indicate a similarity score.

Numbers between -1 and 0 indicate a dissimilarity score.

Instructions : Implement a function that takes in two word vectors and computes the cosine distance.

• Python's NumPy library adds support for linear algebra operations (e.g., dot product, vector norm ...).
• Use numpy.dot .
• Use numpy.linalg.norm .

Expected Output :

\(\approx\) 0.651095

1.3 Euclidean distance #

You will now implement a function that computes the similarity between two vectors using the Euclidean distance. Euclidean distance is defined as:

\(n\) is the number of elements in the vector

\(A\) and \(B\) are the corresponding word vectors.

The more similar the words, the more likely the Euclidean distance will be close to 0.

Instructions : Write a function that computes the Euclidean distance between two vectors.

Expected Output:

1.4 Finding the country of each capital #

Now, you will use the previous functions to compute similarities between vectors, and use these to find the capital cities of countries. You will write a function that takes in three words, and the embeddings dictionary. Your task is to find the capital cities. For example, given the following words:

1: Athens 2: Greece 3: Baghdad,

your task is to predict the country 4: Iraq.

Instructions :

To predict the capital you might want to look at the King - Man + Woman = Queen example above, and implement that scheme into a mathematical function, using the word embeddings and a similarity function.

Iterate over the embeddings dictionary and compute the cosine similarity score between your vector and the current word embedding.

You should add a check to make sure that the word you return is not any of the words that you fed into your function. Return the one with the highest score.

Expected Output: (Approximately)

(‘Egypt’, 0.7626821)

1.5 Model Accuracy #

Now you will test your new function on the dataset and check the accuracy of the model:

Instructions : Write a program that can compute the accuracy on the dataset provided for you. You have to iterate over every row to get the corresponding words and feed them into you get_country function above.

• Use pandas.DataFrame.iterrows .

NOTE: The cell below takes about 30 SECONDS to run.

\(\approx\) 0.92

3.0 Plotting the vectors using PCA #

Now you will explore the distance between word vectors after reducing their dimension. The technique we will employ is known as principal component analysis (PCA) . As we saw, we are working in a 300-dimensional space in this case. Although from a computational perspective we were able to perform a good job, it is impossible to visualize results in such high dimensional spaces.

You can think of PCA as a method that projects our vectors in a space of reduced dimension, while keeping the maximum information about the original vectors in their reduced counterparts. In this case, by maximum infomation we mean that the Euclidean distance between the original vectors and their projected siblings is minimal. Hence vectors that were originally close in the embeddings dictionary, will produce lower dimensional vectors that are still close to each other.

You will see that when you map out the words, similar words will be clustered next to each other. For example, the words ‘sad’, ‘happy’, ‘joyful’ all describe emotion and are supposed to be near each other when plotted. The words: ‘oil’, ‘gas’, and ‘petroleum’ all describe natural resources. Words like ‘city’, ‘village’, ‘town’ could be seen as synonyms and describe a similar thing.

Before plotting the words, you need to first be able to reduce each word vector with PCA into 2 dimensions and then plot it. The steps to compute PCA are as follows:

Mean normalize the data

Compute the covariance matrix of your data ( \(\Sigma\) ).

Compute the eigenvectors and the eigenvalues of your covariance matrix

Multiply the first K eigenvectors by your normalized data. The transformation should look something as follows:

You will write a program that takes in a data set where each row corresponds to a word vector.

The word vectors are of dimension 300.

Use PCA to change the 300 dimensions to n_components dimensions.

The new matrix should be of dimension m, n_componentns .

First de-mean the data

Get the eigenvalues using linalg.eigh . Use ‘eigh’ rather than ‘eig’ since R is symmetric. The performance gain when using eigh instead of eig is substantial.

Sort the eigenvectors and eigenvalues by decreasing order of the eigenvalues.

Get a subset of the eigenvectors (choose how many principle components you want to use using n_components).

Return the new transformation of the data by multiplying the eigenvectors with the original data.

• Use numpy.mean(a,axis=None) : If you set axis = 0 , you take the mean for each column. If you set axis = 1 , you take the mean for each row. Remember that each row is a word vector, and the number of columns are the number of dimensions in a word vector.
• Use numpy.cov(m, rowvar=True) . This calculates the covariance matrix. By default rowvar is True . From the documentation: "If rowvar is True (default), then each row represents a variable, with observations in the columns." In our case, each row is a word vector observation, and each column is a feature (variable).
• Use numpy.linalg.eigh(a, UPLO='L')
• Use numpy.argsort sorts the values in an array from smallest to largest, then returns the indices from this sort.
• In order to reverse the order of a list, you can use: x[::-1] .
• To apply the sorted indices to eigenvalues, you can use this format x[indices_sorted] .
• When applying the sorted indices to eigen vectors, note that each column represents an eigenvector. In order to preserve the rows but sort on the columns, you can use this format x[:,indices_sorted]
• To transform the data using a subset of the most relevant principle components, take the matrix multiplication of the eigenvectors with the original data.
• The data is of shape (n_observations, n_features) .
• The subset of eigenvectors are in a matrix of shape (n_features, n_components) .
• To multiply these together, take the transposes of both the eigenvectors (n_components, n_features) and the data (n_features, n_observations).
• The product of these two has dimensions (n_components,n_observations) . Take its transpose to get the shape (n_observations, n_components) .

Your original matrix was: (3,10) and it became:

Now you will use your pca function to plot a few words we have chosen for you. You will see that similar words tend to be clustered near each other. Sometimes, even antonyms tend to be clustered near each other. Antonyms describe the same thing but just tend to be on the other end of the scale They are usually found in the same location of a sentence, have the same parts of speech, and thus when learning the word vectors, you end up getting similar weights. In the next week we will go over how you learn them, but for now let’s just enjoy using them.

Instructions: Run the cell below.

What do you notice?

The word vectors for gas, oil and petroleum appear related to each other, because their vectors are close to each other. Similarly, sad, joyful and happy all express emotions, and are also near each other.

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

• Notifications You must be signed in to change notification settings

C1_W3_Assignment.py

Latest commit, file metadata and controls.

Lab: Assignment 3: Hello Vectors

I’ve finished the coding of the solution and there are some errors when grading the solution. The errors is when grading the get_country and get_accuracy functions.

I found the error on my code, it is because it is not general and I’m using the word_embeddings. However, the instructions said use embeddings.

That was it! Thank you!

Related topics

• Python's NumPy library adds support for linear algebra operations (e.g., dot product, vector norm ...).
• Use numpy.dot .
• Use numpy.linalg.norm .
• Use pandas.DataFrame.iterrows .
• Use numpy.mean(a,axis=None) : If you set axis = 0 , you take the mean for each column. If you set axis = 1 , you take the mean for each row. Remember that each row is a word vector, and the number of columns are the number of dimensions in a word vector.
• Use numpy.cov(m, rowvar=True) . This calculates the covariance matrix. By default rowvar is True . From the documentation: \"If rowvar is True (default), then each row represents a variable, with observations in the columns.\" In our case, each row is a word vector observation, and each column is a feature (variable).
• Use numpy.linalg.eigh(a, UPLO='L')
• Use numpy.argsort sorts the values in an array from smallest to largest, then returns the indices from this sort.
• In order to reverse the order of a list, you can use: x[::-1] .
• To apply the sorted indices to eigenvalues, you can use this format x[indices_sorted] .
• When applying the sorted indices to eigen vectors, note that each column represents an eigenvector. In order to preserve the rows but sort on the columns, you can use this format x[:,indices_sorted]
• To transform the data using a subset of the most relevant principle components, take the matrix multiplication of the eigenvectors with the original data.
• The data is of shape (n_observations, n_features) .
• The subset of eigenvectors are in a matrix of shape (n_features, n_components) .
• To multiply these together, take the transposes of both the eigenvectors (n_components, n_features) and the data (n_features, n_observations).
• The product of these two has dimensions (n_components,n_observations) . Take its transpose to get the shape (n_observations, n_components) .