SLAI Natural Language Processing

Lab 1: Tokens and Collocations

In this lab, you will learn how to split a string into a list of tokens. You will also learn how to identify words that frequently appear together in text; such multi-word expressions often have meaning beyond the individual words, and being able to identify them is useful for NLP applications.

Learning goals

Once you complete this lab, you should:

A basic tokenizer

In this section of the lab, you’ll learn about implementing a tokenizer. The tokenizer will use a loop to split a string into tokens.

For example, "Siri, alexa, your gmail spam detector and spell checkers are all examples of NLP deployed in the real world!" should become ['Siri', ',', 'alexa', ',', 'your', 'gmail', 'spam', 'detector', 'and', 'spell', 'checkers', 'are', 'all', 'examples', 'of', 'NLP', 'deployed', 'in', 'the', 'real', 'world', '!'].

The characters ., ,, !, and ? will be adjacent to words, but should be their own tokens.

Your task

Run the code to see how the basic_tokenize function works.

After running the existing version of the function, experiment with the code as follows to understand how it works:

  1. Comment out the two lines labeled for TASK 1. What happens? Why do we need these lines?
  2. Comment out the line labeled for TASK 2. What happens? Why do we need this line?
  3. Comment out out the two lines labeled for TASK 3. What happens? Why do we need these lines?
    • Hint: you’ll need to change the input to make the code break in this case! Try deleting the punctuation.

Put your answers to these questions in answers.txt inside the replit project.

Tokenizing with NLTK

Understanding a custom tokenizer is a useful exercise to learn how to tokenize text, but in your other projects, you will work with a pre-built tokenizer from a library called NLTK. Here’s how you can call their tokenize function:

>>> import nltk
>>> sentence = "Siri, alexa, your gmail spam detector and spell checkers are all examples of NLP deployed in the real world!"
>>> tokens = nltk.word_tokenize(sentence)
>>> print(tokens)
['Siri', ',', 'alexa', ',', 'your', 'gmail', 'spam', 'detector', 'and', 'spell', 'checkers', 'are', 'all', 'examples', 'of', 'NLP', 'deployed', 'in', 'the', 'real', 'world', '!']

You can see that the tokens are the same as what you got from the basic_tokenize function, but the code is super easy to import. However, this tokenizer is actually more sophisticated than ours. Try to find a sentence for which the NLTK tokenizer has different output than basic_tokenize, and write it down in answers.txt If you are struggling to find one, try the sentences from this paragraph (I promise that there is at least one such sentence).

If you have extra time, try to make basic_tokenize have the same output as the NLTK tokenizer!

Word count code

In main.py, I have included code that counts tokens and pairs of adjacent tokens (bigrams) from a list of tokens. See the two functions: count_bigrams and count_tokens. You should familiarize yourself with this code, as you will be asked to write similar functions yourself later on in the week. You will need to call these functions in the next section.

For now, run both functions on a list of tokens and see what happens.

Tuples and looping

You’ll notice that the keys in the dictionary returned by count_bigrams look like lists but with () instead of []. These are called tuples, and you can use indices to access their items, just like you would with a list. Here’s an example of how to index tuples, plus how to loop through a dictionary:

>>> d = {("carleton", "cs"): 10, ("is", "great"): 20}
>>> for k, v in d.items():
...     print(k)
...     print(k[0])
...     print(k[1])
...     print(v)
... 
('carleton', 'cs')
carleton
cs
10
('is', 'great')
is
great
20

Finding collocations with PMI

Collocations are pairs of words like “New York” and “social distancing” that appear frequently together and often carry some special meaning when they are combined. One way of identifying collocations in text is by using a formula called pointwise mutual information (PMI). Words that appear frequently together have a high PMI score.

PMI formula

PMI is calculated as follows for two tokens \(w_1\) and \(w_2\). We need these token probabilities:

The formula for PMI is then: \(log_2(\frac{P(w_1 w_2)}{P(w_1) \cdot P(w_2)})\)

Use the formula to calculate the PMI. I recommend using the log2 function from numpy like this:

  >>> import numpy as np
  >>> print(np.log2(3))
  1.584962500721156

Find the PMI for "spam detector" in the sentence "Siri, alexa, your gmail spam detector and spell checkers are all examples of NLP deployed in the real world!". Does it match your worksheet?

Working with a corpus

A corpus is a collection of written texts. For this lab, we’ll work with a corpus of Jane Austen novels. A function has been provided to load Austen’s six most popular novels in one string:

from util import load_austen
novels = load_austen()

Next, convert all of the text from the novels into tokens, using the function from NLTK.

Finally, loop through all of the bigrams in your corpus and calculate their PMI. Store each PMI score in a dictionary as you go.

The highest PMI scores

To find collocations, you’ll need to sort your dictionary of PMI scores. Here’s an example of how you can sort key-value pairs from a dictionary in python, from highest value to lowest value.

>>> d = {"carleton": 5, "college": 2, "SLAI": 20}
>>> s = sorted(d.items(), key=lambda item: item[1], reverse=True)
>>> print(s)
[('SLAI', 20), ('carleton', 5), ('college', 2)]

Print the 20 bigrams with the highest PMI scores. Use list slicing to find the first 20 items in a list. Here’s an example to get the first three items:

>>> lst = ["carleton", "college", "SLAI", "is", "cool"]
>>> print(lst[:3])
['carleton', 'college', 'SLAI']

Filtering

You may see that the bigrams with the highest PMI scores are those that occur only once, with compound words like maid-servants or stylizations like _Miss Bertram_. Try filtering your results so that you only consider bigrams where each token appears at least 10 times - does that improve your results? Is 5, 25, or 50 better? Discuss with your partner.

Extensions

These extensions may be completed in any order if you finish the main part of the assignment.

Chi-square

Read through section 5.3.3 of this textbook chapter on collocations to learn about the chi-square test, another method for finding collocations in text. Implement this method. Do you think the results are better or worse than your results with PMI? Why or why not?

Numpy and pandas

Another way to store bigram counts is using a numpy array, which is like a list of lists but more efficient. This function creates a numpy array of counts. Run the code with a single sentence and try to figure out how it works.

def count_numpy(tokens):
    # figure out what this does!
    token_to_idx = {token: i for i, token in enumerate(sorted(set(tokens)))}

    # this creates a table of unigram/bigram counts. figure out how it works!
    arr = np.zeros((len(token_to_idx), len(token_to_idx)), dtype=np.int16)
    for i in range(len(tokens) - 1):
        arr[token_to_idx[tokens[i]]][token_to_idx[tokens[i + 1]]] += 1
    return token_to_idx, arr

Then, try replicating the PMI results with this numpy array, to get acclimated with using numpy. You may find that the sum function is useful.

Exploring word counts

Skim through this article on a study done regarding the changing language usage by the famous mystery author Agatha Christie. I’ve provided two books: one of her first (“The Mysterious Affair at Styles”), and her last (“Elephants Can Remember”). Can you reproduce the results of the study? You can count words, but what words are you looking for?

Credits

Many thanks to Dave Musicant, Andy Exley, James Ryan, François Dominic Laramée, and Rada Mihalcea, from whom I’ve used some of their ideas on presenting this material.

Thanks to the maintainers of Primer Spec from EECS 485 at the University of Michigan, which is being used to style this webpage.