Natural Language Processing

Build NLP components and working pipelines in Python and NumPy: tokenization, corpus vectors, fitted TF-IDF search, bigram models, Naive Bayes and logistic classification, spelling correction, HMM tagging, SVD embeddings and attention forward passes. Finish with a mini engine that searches, classifies and completes from fitted state. Implement the algorithms directly, with explicit assumptions and small reproducible data.

11 projects, 275 hands-on levels, run in your browser.

Syllabus

Foundations: code through language: Never written code before? Start here. You will learn the basics of Python, output, variables, types, decisions, loops, and functions, through words, tokens, and text. By the end you are ready for Project 1.
Text Foundations: Turn raw text into explicit tokens, a shared vocabulary, IDs and n-grams, then compose per-document corpus statistics. The project teaches normalization as a lossy task choice and preserves document boundaries and fitted feature ordering.
Counting & Vectors: Once text is tokens, the first model is just counting. This project turns counts into the vectors every classical NLP method uses: word frequencies and the Zipf pattern they follow, bag-of-words vectors over a fixed vocabulary, term-frequency weighting, and similarity between documents. It ends by moving the same operations into numpy, the representation the rest of the track builds on.
TF-IDF & Search: Build document frequency, IDF and TF-IDF, then compare count-cosine retrieval with a fitted smoothed TF-IDF SearchIndex. Queries reuse stored vocabulary and weights; ranking has stable ties and explicit no-match behavior.
N-gram Language Models: Estimate bigram probabilities, smooth over a fixed support, score observed events and compute comparable perplexity. Compose a boundary-aware fitted BigramLM with explicit start/end outcomes and bounded sampling. This teaches a count-based language model, not transformer training.
Text Classification: Sorting text into categories, spam or not, positive or negative, is the workhorse task of applied NLP. This project builds two classic classifiers from scratch: multinomial Naive Bayes, with its class priors and smoothed word likelihoods scored in log space, and logistic regression with the sigmoid and a gradient step. It also builds the metrics that tell you whether a classifier is any good, accuracy, precision, recall, and F1, and wires them into a small sentiment classifier.
Edit Distance & Spelling: How different are two strings, and what did the user probably mean to type? This project builds edit distance with dynamic programming, the longest common subsequence and a similarity ratio, the four edit operations (insert, delete, replace, transpose) that generate spelling candidates, a frequency-based spell corrector in the style of Norvig's, and fuzzy matching that snaps a misspelling to the nearest known word.
Sequence Labeling with HMMs: Many NLP tasks label each word in a sentence: part of speech, named-entity type. The Hidden Markov Model is the classic tool, modeling a hidden chain of tags that each emit a word. This project builds an HMM from counts (transition, emission, and initial distributions), the forward algorithm that sums over all tag sequences to score a sentence, the Viterbi algorithm that finds the single best tag sequence, its application to part-of-speech tagging, and how to evaluate the result, all in numpy.
Word Embeddings: Words that appear in similar contexts have similar meanings, so a word can be represented by the company it keeps. This project builds embeddings from first principles: the co-occurrence matrix counting which words appear near which, the positive pointwise mutual information that turns counts into association strengths, dimensionality reduction with SVD to get dense vectors, cosine similarity between them, and the nearest-neighbor and analogy queries that made embeddings famous.
Attention: Build stable softmax, scaled query/key scores and weighted values with supplied projection matrices. Compose sinusoidal positions, exact optional causal masking, residuals, normalization and feedforward layers into an AttentionEncoder forward pass. Training and text generation are separate outcomes not claimed here.
Capstone: A Mini NLP Engine: The finale assembles the track into one working system. You will build a preprocessing pipeline that cleans raw text into tokens and a vocabulary, a TF-IDF search that retrieves the most relevant document for a query, a Naive Bayes classifier that labels messages as spam or not, a bigram model that autocompletes the next word, and a dispatcher that routes a request to the right component. By the end you have a small but complete natural-language engine, built from scratch.

Key concepts

Accuracy: The fraction of predictions that are correct. Easy to read but misleading on imbalanced data, where always guessing the majority class scores high.
Attention: A weighted combination of value vectors, with weights obtained by softmax over query/key compatibility scores. Masks may restrict the allowed keys. Weights des…
Bag of words: Representing a document as a vector of word counts over a vocabulary, discarding word order. Simple but a strong baseline for classification and search.
Bigram: An n-gram of size two: a pair of adjacent tokens. Bigram counts power the simplest language models.
Co-occurrence matrix: Counts of center/context pairs under a specified window and boundary policy. A two-sided window gives symmetric full-corpus counts, but arbitrary supplied pair…
Confusion matrix: A table of true positives, false positives, false negatives, and true negatives, from which precision, recall, and accuracy are computed.
Corpus: A body of text used to train or test a model, often a large collection of documents. The plural is corpora.
Cosine similarity: Dot product divided by the product of Euclidean norms, measuring directional agreement for nonzero vectors. A zero vector has no direction; helpers in this cou…
Cross-entropy: Mean negative log probability of evaluated events, measured in bits with log2 or nats with natural log. An event assigned zero probability gives infinite loss;…
Distributional hypothesis: The empirical idea that words used in similar contexts often share aspects of meaning. Contextual similarity may also group antonyms or encode corpus biases, s…
Document: One unit of text in a corpus: a sentence, a review, an email, an article. Documents are what classifiers label and search engines retrieve.
Document-term matrix: A matrix whose rows are documents and columns are vocabulary words, each cell a count (or weight). The standard input to classical NLP methods.
Dynamic programming: Solving a problem by filling a table of subproblem answers and reusing them, the technique behind edit distance, LCS, and Viterbi.
Edit distance: The minimum number of single-character edits to turn one string into another. The Levenshtein version allows insertions, deletions, and substitutions.
Emission probability: In an HMM, the probability of a hidden state producing an observed word, P(word | tag). The rows of the emission matrix sum to one.
F1 score: The harmonic mean 2PR/(P+R) of precision and recall for a stated positive class or averaging policy. It excludes true negatives and does not by itself solve cl…
Forward algorithm: An HMM dynamic program that SUMS over all hidden paths to compute the total probability of an observation sequence. Where Viterbi takes a max, forward takes a…
Fuzzy matching: Matching a string to the closest known string by edit distance, with a threshold to reject far-off inputs. Used in name matching and search-as-you-type.
Hapax legomenon: A token type occurring exactly once in the specified corpus. Hapaxes often contribute to vocabulary growth as more text is observed; no fixed corpus has an unb…
Hidden Markov Model: A model of a hidden chain of states (tags) that each emit an observation (a word), defined by transition, emission, and initial probabilities. Decoded with Vit…
Information retrieval: Finding the documents most relevant to a query, classically by encoding both as vectors and ranking by cosine similarity. The basis of search engines.
Inverse document frequency (IDF): A corpus-fitted rarity weight. Unsmoothed IDF is log(N/df) for positive document frequency; a word present in every document has weight zero. Smoothed formulas…
Jaccard similarity: Intersection size divided by union size for two sets, ignoring repeated occurrences. The course returns zero for two empty sets as a declared convention.
Language model: A model that assigns probabilities to sequences of words, or predicts the next word given the previous ones. From n-grams to transformers, the same idea scaled…
Laplace (add-one) smoothing: Add-one smoothing: add one pseudo-count to every outcome in a fixed support before normalization. It gives those supported outcomes positive probability; a sep…
Layer normalization: Normalization of each feature row using its mean and population variance plus positive epsilon. The normalized variance is var/(var+eps), not exactly one; cons…
Lemmatization: Mapping an inflected form to a dictionary lemma, potentially using vocabulary and grammatical context. It can preserve linguistic distinctions better than crud…
Levenshtein distance: Edit distance allowing insertion, deletion, and substitution, computed by filling a dynamic-programming table. The standard string-similarity measure for spell…
Likelihood: How likely a word (or document) is given a class, P(word | class), estimated from per-class counts with smoothing. Multiplied across a document's words.
Logistic regression: A discriminative classifier that runs a weighted sum of features through the sigmoid to get a probability, trained by gradient descent. A strong text-classific…
Longest common subsequence: The longest sequence of characters appearing in both strings in order but not necessarily contiguously, found by dynamic programming. Basis of a similarity rat…
Maximum likelihood estimate: Choosing parameters that maximize observed-data likelihood. For an unsmoothed bigram table, P(b|a) is count(a,b) divided by the total outgoing transitions from…
N-gram: A contiguous window of n tokens. Unigrams are single words, bigrams are pairs, trigrams are triples, the basic unit of local context.
Naive Bayes: A classifier that scores each class as its prior times the product of word likelihoods, picking the highest. 'Naive' because it assumes words are condi…
Named-entity recognition: Finding and labeling spans of text that name entities, people, places, organizations, a key sequence-labeling task in information extraction.
Natural Language Processing: The field of getting computers to work with human language: tokenizing, classifying, translating, generating, and understanding text. NLP combines linguistics,…
Normalization: Applying declared transformations such as lowercasing, punctuation deletion or number masking. These can reduce spelling variation but also remove useful disti…
Out-of-vocabulary (OOV): A token absent from a fitted vocabulary. Declare whether to reject it, ignore its feature contribution, or map it to an explicitly modeled unknown outcome. Ski…
Part-of-speech tagging: Labeling each word with its grammatical category (noun, verb, adjective). A standard sequence-labeling task, often solved with an HMM and Viterbi.
Perplexity: The inverse geometric mean of the probabilities assigned to scored events, equal to 2 raised to their mean negative log2 probability. Lower indicates better pr…
Positional encoding: Position-dependent vectors added to token vectors, such as paired sine/cosine features. Unmasked attention without positional information is permutation-equiva…
Positive PMI: Positive pointwise mutual information: max(0,log2(P(a,b)/(P(a)P(b)))) for positive joint mass. Use separate row and column marginals; zero-joint entries become…
Precision: Of the items the classifier called positive, the fraction that actually are: TP / (TP + FP). High precision means few false alarms.
Prior probability: How likely a class is before seeing the document, estimated as the fraction of training documents with that label. The starting point of a Bayes score.
Recall: Of the actual positives, the fraction the classifier found: TP / (TP + FN). High recall means few misses.
Residual connection: Adding a sub-layer's input back to its output (x + f(x)), which helps gradients flow through deep networks. Used around attention and feed-forward layers.
Sampling: Choosing the next word at random in proportion to the model's probabilities, usually by inverse-transform sampling over the cumulative distribution. Gives…
Self-attention: Attention whose queries, keys and values derive from the same sequence, usually through separate projections. It mixes permitted positions; causal or other mas…
Sentiment analysis: Classifying the emotional polarity of text, positive, negative, or neutral, a common application of text classification.
Sequence labeling: Assigning a label to each token in a sequence, like part-of-speech tags or entity types. Hidden Markov Models are the classic tool.
Sigmoid: The mathematical function 1/(1+exp(-z)), mapping real scores into (0,1). Stable floating-point evaluation branches on the sign of z; extreme finite values can…
Singular value decomposition: Singular value decomposition M=U diag(s) Vt. Truncating the largest singular directions gives a low-rank approximation; squared singular values measure Frobeni…
Smoothing: Redistributing probability mass so unseen outcomes within a declared finite support can receive positive probability. Add-k uses a positive pseudo-count and a…
Softmax: A function that turns a vector of scores into a probability distribution, exp(x) over the sum of exp(x), computed stably by subtracting the max first. It produ…
Spell correction: Selecting a plausible known word using edit candidates and evidence such as corpus frequency. This course uses one-edit candidates, preserves known inputs and…
Stemming: A heuristic that removes word endings to merge related forms, sometimes producing a nonword such as runn. Its speed and accuracy depend on the algorithm, langu…
Stopword: A word chosen for removal under a task-specific policy, often because it is frequent and weak at distinguishing topics. Common words can still carry negation,…
Term frequency (TF): A term occurrence count or a declared transformation of it, such as count divided by document length, log scaling or maximum-count normalization. These variant…
Text classification: Assigning a label to a document, spam or not, positive or negative, the workhorse supervised task of NLP.
Text generation: Running a language model forward to produce text, picking the next word greedily (the most probable) or by sampling from its distribution.
TF-IDF: The product of a declared term-frequency value and corpus-fitted IDF. Use the same vocabulary ordering and IDF on queries and documents; the weight is a retrie…
Token: One unit produced by tokenization, typically a word but sometimes a subword or character. A document becomes a sequence of tokens.
Tokenization: Dividing text into units such as words, subwords or characters. Lowercasing and punctuation removal are separate normalization choices; this course deliberatel…
Transformer: The architecture built from stacked blocks of self-attention and feed-forward layers with residual connections and layer norm. It underlies modern large langua…
Transition probability: In an HMM, the probability of moving from one hidden state to the next, P(tag_t | tag_{t-1}). The rows of the transition matrix sum to one.
Type-token ratio: Distinct token types divided by total token occurrences. It describes lexical variety in a particular sample and depends strongly on sample length; an empty-sa…
Vector analogy: An embedding query formed as b-a+c, then compared with eligible candidate vectors. Famous examples illustrate a possible relationship, not a property guarantee…
Viterbi algorithm: A dynamic program for the most probable joint hidden path and observations in an HMM. Log-space scores avoid product underflow during decoding; predecessor/fin…
Vocabulary: The set of distinct tokens a model knows, usually mapped to integer ids. Words outside it are out-of-vocabulary (OOV).
Word embedding: A vector representation of an item such as a token. This project constructs short dense word vectors from context counts, PPMI and SVD; proximity reflects the…
Zipf's law: The empirical pattern that a word's frequency is roughly inversely proportional to its frequency rank: a few words are very common and a long tail is rare.