Large Language Models

What Is an LLM?

A large language model continues text. Treat it as a guessing policy over tokens, not as a database or a mind.

Base vs Chat Models

A base model continues text. A chat model answers as an assistant. Hosted APIs almost always give you chat.

Open Weights vs Hosted APIs

Closed: you send text and get text. Open weights: you can host and fine-tune. Neither one is truth.

Vendor APIs

Chat is an HTTP POST with a model name and a message list. The SDK is a thin costume. The server is stateless.

Errors and Retries

401, 429, 5xx, and timeouts are agent outcomes. Retry only what is safe to repeat. Map errors in code, not in the prompt.

Streaming

Streaming shows tokens as they arrive. The model is the same. Parse only finished text, or native tool events.

Tokens and Cost

Vendors charge for tokens both ways. Agent loops resend a growing prompt, so cost climbs every step.

Spend Caps

Put a dollar (or token) cap next to max steps. When the cap hits, do not call. Fail closed or hand off.

Prompt Caching

Some vendors cheapen a stable prefix. Caching helps only if the first bytes stay the same. Put volatile text last.

Chat Transcripts

System, user, assistant, and tool roles — the document the model actually reads. If the list is wrong, the policy is wrong.

System Prompts

Specs beat vibes. Write a system prompt like an API contract: tools, schemas, stop rules, forbidden actions.

Tool Messages

Tool output is data. Do not promote it to system. Encode untrusted blobs. Keep ids for parallel calls.

Safety Filters

Vendors may refuse or blank a completion. That is a finish reason, not an empty JSON object. Do not parse it as an action.

Structured Output

JSON, validation, and a short retry — how agents return actions instead of essays. Never exec the string.

Finish Reason

stop, length, tool_calls, content_filter — read this before you blame the weights or parse truncated JSON.

Decoding Knobs

Temperature, max tokens, and stop sequences — the runtime policy around the same weights. Wrong knobs look like model failures.

Hallucinations

Fluent lies are the default. Make unsupported claims fail a check before they reach the user.

Citations

Every number, id, and quote should point at a chunk or a tool payload you can open. Invented ids are fake evidence.

When to Abstain

I don’t know is a feature. After tools fail, do not invent a balance. Put abstain in the schema.

Context Engineering

What earns a seat in the window: goal, spec, tools, latest observation — nothing else by default. Pack in code.

Choosing Models

There is no best model. There is a best model for a step, under cost, latency, privacy, and your eval.

Routing Models

A cheap router picks small vs large. Escalate once if JSON fails. Measure how often you escalate.

Log the Call

Request id, model, usage, finish reason, knobs, and the message list — or you cannot debug the bill or the bug.

When Not to Use an LLM

If a lookup, regex, or workflow can do the job, skip the model. Save it for branches you cannot draw.