Pretraining

Pretraining is the expensive phase: run a transformer over a huge text corpus and train it to predict the next token. That single game, played trillions of times, produces a model that continues almost anything: English, Python, broken JSON, threatening emails.

The loss is cross-entropy on the true next id. Teacher forcing: at train time the model sees the true prefix, not its own samples. That is why it can learn long books without collapsing on its first mistake. If it were trained only on its own rollouts, early errors would poison the rest of the sequence and learning would stall.

Self-supervised means the labels are the text. No human tagged each token. Humans did choose the corpus, the filters, the tokenizer, and the cutoff date. That is product, not a math footnote. Garbage in the crawl becomes fluent garbage in the weights.

This lesson is the objective. Later lessons are how scores become a token, how past keys are cached, and how the window is a suitcase. Do not skip the objective. Hallucination is not a mysterious disease. It is what “plausible next token” looks like when the prefix asks for a fact.

A wrong picture

A wrong picture is: “pretraining stores a database of true sentences.” It stores weights that score continuations. Many true facts are easy to continue. Many false facts are also easy to continue if they are fluent. The loss does not know about the world. It knows about the next id in the corpus.

Another wrong picture is: “bigger pretraining means the model follows my policy.” Policy is later: fine-tuning, preference data, prompts, tools. A bigger pretrained model is a stronger linguistic prior. It will still invent a URL if inventing a URL is the plausible continuation.

A third wrong picture is: “the model saw my private tickets.” It did not, unless they leaked onto the public web or you later fine-tuned on them. It did not see events after its cutoff. It did not agree to your policies.

The game in words

1. Take a sequence of token ids from the corpus.
2. For each position i, hide the future (causal mask).
3. Score every vocab id as a possible next token.
4. Loss is “how surprised were we that the true id was the one that followed?”
5. Nudge weights to be less surprised next time.

Teacher forcing means position i always sees the true tokens < i, even if a sampled model would have already gone off the rails. Generation later does go off the rails; training did not practice recovery unless you add that later.

A base model continues text: it will finish a blog post, a function, a rant. A chat model was further trained to answer in roles (fine-tuning part of this track). If you stuff a base-style completion prompt into a chat model, you are fighting its later training. If you stuff a chat wrap into a base model, you are speaking specials it never learned as roles.

Scale: more parameters, more data, more compute — loss falls until you bottleneck on one of them. “Emergent” abilities are often thresholds on a smooth curve plus a metric that was near zero. For agents, a bigger model on a bad prompt still calls the wrong tool. Scaling is a later lesson. The objective does not change.

A tiny example in words

A character-level bigram table is a transformer with amnesia: context length 1. Count how often b follows a. Predict with the most common follower. Sample a string. You will see loops and local habits, not long-range copy.

Scale context and capacity, and you get long-range copy and “reasoning-shaped” completions. The objective did not change. Only the function class did. That is the whole jump from a bigram table to a deep decoder: same next-piece game, richer conditioning.

A bigram table is next-token with memory 1

Count neighbors. Greedy-continue. Print the table slices and the sample. Characters here stand in for tokens so the lists stay short.

Run to execute this in your browser. Nothing is sent to a server.

You should see a greedy loop: the most common follower of the last character wins every time. sample will look stuck or chanting. That is maximum-likelihood with no long context.

A real pretrained transformer conditions on thousands of previous ids, not one character. It can copy a job id from the start of the prompt into a tool call at the end — when attention and positions cooperate. The loss is still “next id.”

Change the corpus to a palindrome-like string and watch the table change. Pretraining is this counting idea on a giant, rich function, not a spreadsheet of facts.

What the objective will not give you

• Truth: plausible is not true.
• Freshness: weights freeze; the world does not.
• Your schema: unless that schema was common in the crawl or you train later.
• Permission: continuing a threatening email is still next-token.
• Tools: the model does not call Python because it predicted the word search. Your loop does.

Tools, retrieval, abstain rules, and evals exist because pretraining is not a database. Treat the base skill as a linguistic prior, then constrain it. The next track (hosted chat products) is how people wrap that prior. This track is the prior itself.

How agents use this

Pretraining explains fluency and hallucination: the game is “plausible next token,” not “true in the world.” When the prefix asks for a citation, a URL, or a Python API, the plausible continuation is often a well-formed fake. Fluency is not a truthfulness signal.

Give the model a tool that returns a real number. Put the observation in the sequence. Ask it to quote. That is how you use a linguistic prior without pretending it is a ledger.

Do not fine-tune on unreviewed logs to “make it know our PDFs” if the PDFs change weekly. That is retrieval, unless you like stale weights. Fine-tuning is a later lesson. The point here: the pretrained game is continuation.

• Prior: fluent English and code, not a knowledge contract.
• Cutoff: no events after train day, unless tools.
• Private data: not inside the base weights.
• Chat vs base: later training changes the default wrap; the next-token game remains.
• Agents: constrain with tools, schemas, and stop rules.

Note:The model did not see your private tickets unless they leaked onto the public web. It did not see events after its cutoff. It did not agree to your policies.