Neural Nets & Transformers

Tokens

Models do not read letters or words. They read tokens — small pieces with integer ids.

Byte Pair Encoding

BPE starts from characters and glues frequent pairs into new tokens. That merge list is the tokenizer.

Special Tokens

Pad, end, and chat-role markers are extra rows in the table. Agents live and die by them.

The Embedding Table

Token id to vector: a lookup table that is the first layer of every transformer.

Positions

Without a position signal, a transformer is a bag of tokens. Order is how 'pay now' differs from 'now pay'.

Attention

Each token builds a query, matches keys, softmax to weights, then a weighted sum of values.

Multi-Head Attention

Several attention heads in parallel: different subspaces, different relationships.

The Transformer Block

Residual stream, attention, then a feed-forward net. Depth is how many times we mix and think.

Residuals and LayerNorm

Add the delta back. Normalize so depth does not explode. That highway is why 96 layers can train.

Encoder vs Decoder

Decoders generate left to right with a causal mask. Encoders see both sides. Agents almost always call a decoder.

Pretraining

Next-token prediction at web scale. The game is plausible continuation, not a database of truth.

Logits and Unembedding

The last vectors become one score per vocab id. Softmax turns those logits into chances.

Decoding

Turn logits into the next token: greedy, temperature, then repeat until stop.

The KV Cache

Past keys and values do not change. Store them. That is why the first token is slow and the rest are faster.

Context Windows

A hard token budget for one forward pass. Pack it like a suitcase. Silent truncation forgets the spec.

Prefill and Decode

Prefill reads the prompt. Decode writes one token at a time. Agent latency is usually prefill plus a short decode.

Fine-Tuning

Continue training on your data. SFT copies demonstrations. Preference training copies taste. Prompting is cheaper.

LoRA

Freeze the big matrix. Learn a small low-rank patch. Store adapters, not a second full model.

Limitations

Hallucination, knowledge cutoff, and quadratic attention — what transformers will not save you from.

Lost in the Middle

Models use the start and the end of a long prompt more reliably than the middle. Pack for that.

Scaling

Bigger models, more data, more compute usually lower loss. They do not usually add obedience to your schema.

Padding and Masks

Batches need equal length. Pad on the right (or left), then mask so attention does not look at pad.

Why Transformers

They mix any pair of tokens in one layer and train in parallel over the sequence. That beat RNNs for language.

The Stack an Agent Actually Runs

One call is tokens, embeddings, blocks, logits, decode, cache. Your job is the sequence and the stop rules.