NLP foundations — tokens, embeddings, attention — AI for Analysts Module 4

Computers don't read text; they multiply numbers. NLP is mostly about how to turn text into useful numbers and back.

Tokenisation

First, text is broken into tokens — units roughly the size of common sub-words. 'unemployment' might be one token; 'photosynthesise' might be three. Modern tokenisers (Byte-Pair Encoding, SentencePiece) learn the optimal vocabulary from data. A typical English page is ~500-700 tokens.

Embeddings

Each token gets mapped to a vector — a list of, say, 1024 numbers. Tokens with similar meaning end up at nearby positions in vector space. 'King − Man + Woman ≈ Queen' is the famous Word2Vec demo of how semantic relations are encoded as geometric relations. Modern LLMs learn embeddings as part of training; you don't separately compute them.

Attention

The transformer's central innovation. For each token in the sequence, attention asks: how relevant is every other token to this one? It computes a weighted average of all other tokens' representations, weighted by relevance. This lets the model decide, at every layer, which parts of the context matter for the current token.

text

Sentence: "The bank raised rates."
When processing "bank", attention can weight "rates" highly
— enough to interpret "bank" as a financial institution
rather than a river bank.

Multi-head attention

Instead of one attention computation, the transformer runs many in parallel (typically 8-128 'heads'). Each head can specialise in a different kind of relationship — syntactic dependencies, semantic similarity, positional patterns. The outputs are concatenated and projected back to the embedding dimension.

The full transformer block

Multi-head self-attention → add residual → layer norm → feedforward network → add residual → layer norm. Stack 12, 24, 96 of these blocks, train on trillions of tokens, and you have a modern LLM.

If you read one paper, read this one

'Attention Is All You Need' is eight pages. The mathematics is elementary. The clarity of presentation is extraordinary. Read it once carefully; it's the founding document of the entire LLM era.

Exercise

An analyst pastes a 2,000-word central-bank monetary-policy statement into a Claude or ChatGPT API call and asks for a summary. (1) Roughly how many tokens does this consume on input? (2) If the model's context window is 200,000 tokens, what proportion of capacity does this use? (3) Why might the analyst care about tokens beyond cost — what other constraint does token count drive? (4) Briefly: explain in one sentence why we tokenise text at the sub-word level (BPE / SentencePiece) rather than character-by-character or word-by-word.