Streaming

Streaming sends pieces of the completion as they are generated. The UI can type. The user feels faster. The model is the same. You did not get a smarter network. You got a different delivery: many small events instead of one JSON blob at the end.

The Transformers track split prefill (read the whole prompt) from decode (emit one token at a time). Streaming does not skip prefill. The first token still waits until the prompt is processed. A spinner during that wait is honest. A fake “thinking” paragraph you wrote in the client is a costume.

Rules that keep agents from executing garbage:

• Concatenate deltas into one string (a delta is a small piece of text).
• Validate JSON / tool calls only when the stream ends, or when the vendor sends a finished tool-call event.
• Partial JSON is not an action.
• If the stream dies mid-sentence, treat it like a timeout or a truncated completion (finish_reason may be missing; you must still decide).

Why partial JSON is poison

A tool call that looks like {"ac is not {"action": "get_job"}. If you json.loads early, you throw. If you “repair” early, you guess a key. If you execute the guess, you might call the wrong tool or the right tool with a truncated id. Agents fail here in production when someone wanted the UI to feel snappy.

Some vendors stream native tool_calls objects with an index and a finished flag. Prefer those over regex on prose. Still validate arguments against your schema. Vendors are not your type checker. A finished native call with job_id: -1 is perfect JSON and a bad business object.

Run to execute this in your browser. Nothing is sent to a server.

After one chunk the text is '{"ac' and parsed is None. After all chunks you get a dict with get_job and 17. If you had executed the partial, you would have executed garbage. The lesson is mechanical: assemble, then parse.

Try printing parse_action(assemble(deltas[:2])). You still should not get a valid object. Two thirds of a JSON string is not an action either.

UX without lying

Use streaming for user-visible prose: a draft email, a status explanation, a “here is what I found.” For tools, wait for a complete object (or the native event). You can still show “Calling get_job…” from your code once the call is validated — that is your UI, not a partial model string.

Show a spinner during prefill. First token is not “the model started thinking in the UI.” It is “prefill finished.” If first token is slow, the prompt is fat, the vendor is loaded, or the region is wrong. That is a packing and capacity problem (later lessons), not a reason to parse early.

If you stream to a user, decide what happens when they hit stop. Cancel the HTTP stream. Do not execute a half tool call. Do not store an incomplete assistant message as if it were a finished action. You may store it as a truncated draft with finish_reason=cancelled.

Many vendors use server-sent events: many small JSON rows, each with a text delta or a tool-call delta, and a final row with usage. Assemble by index when they stream parallel tool arguments. Buffer bytes until you have a complete event; do not json.loads a half event line. If the connection dies, you may have usage missing — still treat the text as truncated.

Backpressure: if the UI cannot paint as fast as tokens arrive, drop paint frames, not tokens. Your assembler must keep every delta. The user-visible typewriter is optional. The string you parse is not.

Logging and secrets

Logging every delta can leak partial secrets into debug stores and can 10x your log volume. Log the assembled message, redacted. If you need to debug stream assembly, do it in a locked debug bucket for a sample of traces, not in the product warehouse.

A mid-stream disconnect is an error. Apply the previous lesson: if no tool ran, you may retry the LLM call; if you already displayed a partial refund sentence, do not also execute a guessed refund.

Usage often arrives only on the last event. If you never see a final event, you have no usage and no finish reason — treat as timeout/truncated, not as stop. Do not parse. Do not bill-guess from character counts as if they were vendor usage; you may estimate for the cap, then reconcile when a later retry succeeds.

Streaming a tool call to a customer UI is usually the wrong UX. Stream the explanation after the tool succeeded. The typewriter on arguments is how a job id appears one digit at a time and someone screenshots a half id.

What goes wrong

• json.loads on every delta “for snappiness.”
• Regex for the first curly brace and executing whatever follows.
• Treating stream UX as a different model and changing temperature only on streamed calls, accidentally.
• Not handling cancel. The user clicked stop; your executor still POSTed the payment API.
• Storing raw deltas with API keys the user pasted mid-sentence.

How agents use this

Two paths in code: complete() (one blob) for tools and tests; stream() for prose. Both return the same finished type: message + usage + finish_reason. Tests should use complete(). Do not require a stream assembler to unit-test the parser.

When the vendor supports native tool events, parse those. When it only streams text, assemble then json.loads then validate. Cap max_tokens so a stream cannot ramble for a novel while you wait to parse.

Tests: one chunk fails parse; all chunks pass; cancel means no tool. If you cannot write those three tests, you are not ready to stream tools.

Watch out:Partial JSON is not an action. Assemble, then parse. Logging every delta is how secrets and noise enter the warehouse.