Vendor APIs

You will talk to almost every commercial LLM through a chat API: an HTTP POST with a JSON body. HTTP is the web’s request/response protocol. POST means “here is a body, do this.” If you can write the JSON, you can debug the SDK. The SDK is a library that builds that JSON and parses the reply. When the SDK is confusing, print the JSON.

A typical request includes:

• model — which weights + tokenizer + alignment (a name like a product SKU)
• messages — a list of role + content (later: tool calls as structured fields)
• temperature, max_tokens — decoding knobs (own lesson)
• optional: tools, response_format, seed, stop sequences, whether to stream

A typical response includes:

• output messages — usually one assistant turn
• usage — prompt tokens vs completion tokens (and sometimes cached tokens)
• finish reason — stop, length, tool_calls, content filter
• id — a request id you log next to your trace step

Agents that ignore finish_reason and usage are flying without instruments. Truncated JSON looks like a “dumb model.” A silent 2x bill looks like “the model was thinking.” Both are readable from the response if you store it.

The server does not remember you

You append the assistant and tool turns. The server is stateless. Stateless means: this POST does not know about the last POST unless you resend the history. If you forget to send the tool result back, the model never saw it. “The API has memory” is a myth. Your transcript is the memory.

Some vendor products offer “threads” or stored conversations. That is their database wrapped as a convenience. For an agent you operate, keep the transcript in your store so you can redact, trim, test, and replay. Convenience threads are fine for a chat UI toy. They are a problem when you need to prove what the model saw.

Joeven’s Try it boxes cannot hit a network. The client below records calls, checks the schema, and returns a scripted assistant turn. Production looks the same until complete() becomes an HTTP POST. That is the point of the fake: your loop should not care.

Run to execute this in your browser. Nothing is sent to a server.

The first print is a JSON object with an assistant message, usage, and finish_reason. You then append that message and a tool result, and call again. The second answer can use the observation. calls billed is 2. That is an agent loop: two POSTs, growing messages, two lines on the invoice. If you skipped msgs.append for the tool, the second call would still only know the question.

The fake counts prompt tokens as words. Real usage comes from the vendor tokenizer. Use their usage for money. Use a local estimate only for packing. Mixing the two is how you “stay under budget” in a notebook and overflow in production.

Vendor differences are real but shallow

OpenAI-style, Anthropic-style, Gemini-style: roles, where the system prompt sits, the JSON shape of a tool call, whether there is a developer role. Wrap one complete(messages) so the agent loop does not care. Inside the wrapper, map to the vendor. Do not sprinkle vendor field names through the planner.

Auth: a bearer token or a vendor-specific key header. The key lives in the environment, not in the message list, not in the system prompt, not in a git repo. If the model ever sees sk-live-..., you have a logging and prompt-injection problem. Rotate the key.

Timeouts: set a client timeout shorter than the user will wait, and shorter than your spend-cap patience. A hung POST is the errors lesson. Headers: request id you generate, plus the vendor’s id you store from the response. Content-type JSON. Do not gzip the body unless you know the vendor accepts it.

SDKs hide retries. Turn off the SDK’s silent retry or you will double-call next to your own retry policy. Print the JSON once in a staging log (redacted) when you integrate a new vendor; after that, trust your wrapper tests.

Idempotency and retries are the next lesson. Streaming is the lesson after that. Structured output is a later part. This lesson is the POST and the transcript.

A walkthrough: job 17

The user asks for job 17. Your code builds messages: system spec, user text. POST. Assistant says it needs get_job. You run get_job(17) in your process. You append a tool message. POST again. Assistant quotes status=failed. You parse, you stop. Four objects in the list. Two billed calls. One tool execution that never went to the GPU. That split is the whole craft.

What goes wrong

• Treating the SDK session as memory and sending only the new user line.
• Dropping usage so finance cannot see which feature burned the budget.
• Putting API keys in messages “so the model can call the vendor.” The model should not hold the key. Your tool should.
• Parsing only choices[0].message.content and ignoring tool_calls and finish_reason.
• Retrying the whole loop when one POST 500s, duplicating a refund tool. Next lesson.

How agents use this

Log every request id, model, usage, and finish reason next to the trace step. When finance asks why the bill jumped, you will answer with a call count and a token sum, not with a story about thinking.

Unit-test the loop with FakeChatClient. Script the assistant turns. Assert which tool ran. You do not need a paid key to prove the transcript is appended correctly. You need a paid key to prove the vendor’s live schema still matches your wrapper — that is a smaller, rarer test.

Never put API keys in messages. Never log raw keys. Redact in the logger, not as a comment.

The fake counts words as tokens. Production uses usage. Tests assert call count and message roles. That is enough to catch “forgot to append tool” without spending a dollar.

Watch out:Streaming is UX, not a different model. Parse the final tool call after you assemble the text (or use native tool-call events). Next lessons: when the POST fails, then streaming.