Curriculum/Large Language Models
Open Weights vs Hosted APIs
Closed: you send text and get text. Open weights: you can host and fine-tune. Neither one is truth.
Closed / hosted means a vendor keeps the weights. You send text (and maybe files) and get text. You cannot inspect layers. You can still eval the inputs and outputs. Keys, rate limits, a data policy, and a region come with the bill. You are renting a completion.
Open weights means you can download (or license) the parameters, host them, and sometimes fine-tune. You get more control over privacy and unit cost at high volume. You also get ops: GPUs, queues, tokenizer files, CUDA drivers, and security patches. You are running a factory.
Neither kind is a database. Open does not mean honest. Closed does not mean safe. The wrap around the model still does the grounding: tools, schemas, allowlists, evals. A local model that emits a fake invoice id is still a fake invoice id. A hosted model that cites a tool you actually ran is still only as true as that tool.
What you actually buy
| Need | Hosted API | Open weights you host |
|---|---|---|
| Time to first demo | Hours | Days to weeks if you have no GPU team |
| Privacy of prompts | Vendor policy + DPA | Your VPC, if you configured it |
| Fine-tune | Vendor product, or not at all | Full or adapter-tune, plus data ops |
| Inspect weights | No | Yes, as tensors, not as “explanations” |
| Patch a jailbreak in the net | Wait for the vendor | Retrain or swap — still not a substitute for code |
| Bill shape | Tokens and seats | Hardware, power, idle time, people |
A DPA is a data processing agreement: the legal text about whether the vendor trains on your prompts. Read it. “We do not train on API data” is a product sentence. The contract is the thing that matters when a customer asks.
“Open weights” is not always “do anything.” Licenses differ. Some forbid commercial hosting. Some require you to name the model. Some restrict military use. Read the license before you ship a customer-facing agent on them. Open is a distribution model, not a moral halo.
A decision you can write down
Start with a hosted API if you have no GPU team and no hard privacy wall. Move to open weights when logs cannot leave, or when token bills beat hardware on your measured mix, not on a tweet. Fine-tune last. The ML and Transformers tracks already said why: you need data, evals, and a reason a prompt plus retrieval lost. Fine-tuning is not how you fix a missing tool.
Vendor fine-tune products sit in the middle: you still do not hold the base weights, but you can adapt. They have the same eval duty as anything else. A fine-tune that memorizes last quarter’s CEOs is a stale database with extra steps.
Open weights add control, not honesty. Truth still needs tools and evals.
Pick a house for the weightsRun to execute this in your browser. Nothing is sent to a server.
The first case is a privacy wall: host. The second is a team without GPUs: hosted API. The third is a genuine fine-tune need: open weights or a vendor product. Most student projects are case two. Most regulated customers start as case one even if they later keep a hosted model for a non-sensitive slice.
The function is a sketch, not a complete policy. Real choices add region, latency p95, JSON reliability, and “does this vendor even offer tool-calling.” Put those on a spreadsheet next to your eval, not next to a generic “arena” rank.
Privacy is a catalog constraint
If traces cannot leave the VPC, the catalog shrinks to what you host — that constraint beats a leaderboard. If only some fields are sensitive, split: run the classifier on a small hosted model with redacted text, run the tool that touches account numbers inside the VPC. Splitting is architecture. Pretending one frontier chat is the whole company is a bill.
Consumer chatbot UIs are not APIs. Sending customer tickets to a web UI is a leak: you do not control retention, training, or who screenshots the thread. Integration means a key in the environment, HTTPS from your server, and a written data policy.
Tokenizers and evals when you swap
Keep complete(messages) behind your function. Swap hosted vs local as a config change. Re-run the eval. Tokenizers differ, so prompts that depend on exact token counts, or on a stop sequence that is one token in model A and three in model B, need a second look. Chat templates differ. A system prompt that was “short” on a hosted tokenizer can blow the window on a local one.
You still do not get truth from open weights. You get control. Truth still needs tools, retrieval, and evals.
What goes wrong
- Choosing open because it feels virtuous, then discovering you have no one to page when the GPU node dies.
- Choosing hosted because it is easy, then pasting secrets into a consumer UI “just to try.”
- Fine-tuning first to “make it know our docs.” Retrieval is cheaper to iterate. Fine-tune when retrieval lost on a measured set.
- Ignoring licenses until legal reviews the week before launch.
- Assuming evals are optional because you can “see the weights.” Seeing tensors does not score tickets.
How agents use this
Config: provider = hosted | local, model = name, base_url = .... The agent loop stays the same. Tests use a fake client. Production uses the config. When privacy says local, the router’s catalog is the local list — including “we do not have a 70B, so this step is extract-only on a 7B with a schema.”
Log the provider and the model id on every span. A week of traces will tell you whether “we must self-host” was a real constraint or a slogan. If you self-host, log GPU queue time too. Latency is part of which model you chose.
Watch out:Sending customer tickets to a consumer chatbot UI is not an API integration. It is a leak.
Check your understanding