Applied Natural Language Processing

Code for the course notes Natural Language Processing: From Corpus Statistics to Grounded LLMs, which you can read in full at https://nlp.jcrlabz.com.

Read the book

Online at https://nlp.jcrlabz.com, or as a PDF: download the latest build. The PDF is republished from the same source every time the site is, so the two never disagree.

What is here

book/worked-examples/ holds one notebook per worked example. Each reproduces a table printed in the book, so you can check any number the text claims rather than take it on trust.

Archive/ holds the demonstration notebooks from earlier runs of the course. They are kept for reference and are not maintained.

Running an example

Click any notebook below and it opens in Colab. Choose Runtime > Run all.

Every notebook is self contained. The first cell defines everything the example needs and runs in the notebook itself, so there is no data file to fetch and nothing to install beyond what the install cell installs.

Colab opens a notebook from GitHub read only. Click Copy to Drive to keep your changes.

Read the chapter first. The notebook is the check, not the explanation.

The worked examples

notebook	what it shows	chapter
bridging_the_gap	One request, three answers	1. What Natural Language Processing Is
edit_distance	Edit distance, every cell	2. Corpora and Preprocessing
empirical_laws	Zipf and Heaps, fitted	3. The Empirical Laws of Text
weighting	tf-idf and PMI, by hand	4. Term Weighting and Similarity
bpe	BPE, five merges	5. Subword Tokenisation
hal	HAL, ramped and asymmetric	7. Count Vectors, PPMI, and SVD
coals	COALS, all three steps	7. Count Vectors, PPMI, and SVD
svd	SVD, and what truncation costs	7. Count Vectors, PPMI, and SVD
svd_rank	Choosing K from the spectrum	7. Count Vectors, PPMI, and SVD
word2vec	word2vec, one step at a time	9. Learned Word Embeddings
factorisation	Three methods, one matrix	9. Learned Word Embeddings
ngram_lm	Smoothing, perplexity and the U-curve	10. n-gram Language Models and Perplexity
neural_lm	A neural LM, counted and trained	11. Neural Language Models
rnn	Unrolling, and the vanishing gradient	12. Recurrent Networks
gated	The gradient highway	13. Gated Recurrence: LSTM and GRU
classification	Accuracy, F1 and a confusion matrix	14. Text Classification and Evaluation
contextual	What one vector per word costs	15. Contextual Embeddings
attention	Self-attention, one query at a time	16. Self-Attention and the Transformer
decoding	Greedy, beam, temperature, top-p	18. Steering LLMs: Decoding and Prompting
bleu	BLEU, ROUGE and their blind spots	21. Evaluating Generated Text
ibm_model1	EM learning an alignment	24. Machine Translation

About this file

This README and the notebooks in book/worked-examples/ are generated from the book source and pushed here on every publish, so edits made to them here are overwritten. Everything else in the repository is edited by hand and is left alone. Report a problem with an example against the book at https://nlp.jcrlabz.com.