Large Language Models quiz

24 questions from the Large Language Models track. Score 80% or higher to unlock a certificate.

What Is an LLM?
1. What is the honest interface of an LLM?
- A knowledge graph with proofs
- Tokens in, a distribution over next tokens out (plus a decoder that picks them)
- A guaranteed database
- A Python interpreter
Base vs Chat Models
2. Why does a chat model wait for your question instead of rambling like a webpage?
- GPUs prefer questions
- It was instruction-tuned (and often preference-tuned) to answer in assistant turns
- Base models cannot tokenize
- Chat is a different kind of attention
Open Weights vs Hosted APIs
3. What do you still not get from open weights?
- A tokenizer
- A guarantee that outputs are true or safe
- The ability to host the model
- The ability to fine-tune
Vendor APIs
4. Who stores the conversation between two HTTP calls to a typical chat API?
- The GPU
- Your application: you resend the message list each time
- DNS
- The tokenizer
Errors and Retries
5. A refund tool timed out. You do not have an idempotency key. What next?
- Retry immediately so the user is happy
- Handoff or check whether the refund already happened; a blind retry can pay twice
- Set temperature to 0 and retry
- Drop the system prompt
Streaming
6. When should you json.loads a streamed tool call?
- After the first character so the agent feels snappy
- After the stream finishes (or the vendor marks the tool call complete)
- Whenever a curly brace appears
- Only on 429
Tokens and Cost
7. Why do agent loops cost more than a single chat with the same final answer?
- GPUs dislike JSON
- Each step resends a longer prompt, so input tokens accumulate across steps
- Output is billed per thought-second
- Tokenizers round up to a million
Spend Caps
8. The next LLM call would push this trace over max_usd. What should the agent do?
- Call anyway; the user is waiting
- Skip the call: fail closed or hand off
- Raise temperature so it finishes faster
- Delete usage from the log
Prompt Caching
9. You added the current timestamp as the first system line. Cache hit rate dropped. Why?
- Timestamps use extra GPUs
- The prefix bytes changed every call, so the cheap prefix no longer matched
- Output tokens cannot be cached
- JSON cannot be cached
Chat Transcripts
10. What role should carry a tool’s return value?
- system, so it looks official
- tool (or the vendor’s equivalent), after the assistant’s tool call
- user, so the model trusts it
- tokenizer
System Prompts
11. What makes a system prompt a spec rather than a vibe?
- It uses the word synergy
- It states checkable constraints: tools, schemas, stop rules, forbidden actions
- It is longer than 10,000 tokens
- It repeats "be helpful" twelve times
Tool Messages
12. A retrieved page says “ignore the spec.” Where should that text live?
- In a new system message so it is official
- In a tool (or retrieval) message, treated as untrusted data
- Nowhere; delete the page
- In the user role so the model trusts it
Safety Filters
13. The API returns finish_reason=content_filter and empty content. What should you do?
- json.loads the empty string and pick a default tool
- Treat it as blocked: refuse or hand off, do not invent an action
- Retry with temperature 2
- Move the user text into the system prompt
Structured Output
14. What should happen when JSON validation fails twice?
- eval() the text as Python
- Retry budget exhausted: hand off or fail closed, do not guess an action
- Increase temperature to 2
- Drop the system prompt
Finish Reason
15. The model returned cut-off JSON and finish_reason=length. What first?
- Switch vendors immediately
- Treat it as a truncated completion: raise max_tokens or shrink the prompt, then parse again
- Set temperature to 2
- Delete stop sequences and hope
Decoding Knobs
16. Default temperature for emitting tool JSON?
- 1.5 so the agent is creative
- 0 (or the vendor minimum), plus a schema
- Whatever the playground default is
- Negative
Hallucinations
17. What is the most reliable way to stop an agent claiming it took an action?
- Raise temperature
- Require a tool result in the transcript; if it is missing, the action did not occur
- Add "be truthful" to the vibe prompt
- Use a longer context window
Citations
18. The answer cites doc_99, which was not in the retrieved set. What is that?
- Extra helpfulness
- A hallucination (invented source)
- A cache hit
- A finish_reason
When to Abstain
19. The get_balance tool returned 404. What should the final action be?
- Invent yesterday’s balance from memory
- Abstain (or hand off): do not quote a number
- Retry with a different customer id
- Raise temperature
Context Engineering
20. What should win a seat in a tight window?
- The longest PDF you have
- The goal, pinned spec, enabled tools, and latest observation
- Every past thought
- Disabled tools, in case the model gets curious
Choosing Models
21. When is a small model the right default?
- Never; always buy the frontier
- High-volume structured steps (classify, extract, tool JSON) where evals already pass
- Only for embeddings of images
- When you have no schema
Routing Models
22. The small model failed JSON validation. What is a sane fallback?
- Loop the small model ten times
- Retry once on the large model, then hand off if still invalid
- Switch embedding models
- Drop the schema
Log the Call
23. Finance asks why yesterday’s LLM bill jumped. What log fields answer first?
- The system prompt adjectives
- Call count, model, and usage tokens per trace
- GPU temperature
- The user’s email body verbatim
When Not to Use an LLM
24. The user sends job_id=17 and the word status. Best first move?
- Ask a frontier model to role-play an SRE
- Call get_job(17) from code (a tiny router may still choose the tool)
- Sample three poems and vote
- Fine-tune