Quizzes

Production Agents quiz

24 questions from the Production Agents track. Score 80% or higher to unlock a certificate.

1. Production Architecture

   1. Why should the HTTP gateway usually enqueue a job instead of running the full agent loop?

   • Queues are fashionable
   • Loops wait on tools, humans, and retries; request timeouts and deploys will kill in-process state
   • Models cannot be called from workers
   • Gateways cannot authenticate users

2. Jobs, Not Requests

   2. A user closes the tab during a 40-step loop. What should still be true?

   • The loop is gone, because HTTP died
   • The job is in the store and can checkpoint, pause, or finish without that tab
   • The model keeps the state in its weights
   • You must restart from the first token

3. Isolation and Confused Deputy

   3. Who is allowed to pick the tenant id for get_invoice?

   • The model, because it read the ticket
   • The gateway / signed job. The tool ignores a mismatched model argument
   • The system prompt’s company name
   • Whoever is in the scratchpad

4. Version Everything the Policy Needs

   4. Why record the worker git SHA next to the prompt version?

   • SHAs look professional in dashboards
   • Code and policy ship together; a mismatch is an untested pair
   • Models ignore SHAs
   • Queues cannot store strings

5. The Smallest Prod-Shaped Stack

   5. What is the smallest production-shaped stack?

   • A Kubernetes cluster and a vector database
   • A queue, a job store, a trace store, and a kill switch — even if they are a list and two dicts
   • One FastAPI function with the loop inside the request
   • A longer system prompt

6. Tracing and Observability

   6. Which attribute helps you explain a cost spike?

   • Only CPU temperature
   • Per-span token counts, model id, and usd_est rolled up by tenant and prompt version
   • The agent’s favorite color in the system prompt
   • Screenshot PNG hashes

7. Redact Before You Export

   7. When should secret redaction run?

   • After the vendor has indexed the trace, so search still works
   • Before any exporter, and before untrusted text is stored as a prompt observation
   • Only in the yearly audit PDF
   • Never — the model needs the real key to debug

8. Metrics That Page a Human

   8. Which signal should wake a human at 2 a.m.?

   • Every model timeout
   • Forbidden-tool count above zero, or cost / queue / HITL past a cap you chose
   • Token count going up during a product launch
   • A slightly longer system prompt

9. Cost and Latency

   9. Why do agent loops cost more than a single chat completion?

   • Python is billed per loop
   • Each step resends growing context, so input tokens scale with steps (and retries)
   • Observations are free
   • Stop conditions increase temperature

10. Budgets Are Stop Conditions

    10. The job hits max_usd mid-loop. What should happen?

    • Switch to a cheaper model and keep going, quietly
    • Stop with a structured error, record it on the job, offer a human if the product needs it
    • Delete the trace so finance is not scared
    • Raise temperature to finish faster

11. Queues and Retries

    11. A worker retries a slice that already refunded the customer. What prevents a second payout?

    • Hoping the model remembers
    • An idempotency key so the write tool is a no-op the second time
    • Deleting the queue
    • Raising temperature

12. Idempotency Keys

    12. What belongs in an idempotency key?

    • The model’s latest thought, so keys stay unique
    • Stable ids: job, step, and tool — not the prose of the thought
    • A random uuid on every retry
    • The tenant’s display name

13. Dead Letters, Poison, Fairness

    13. A 50 MB observation lands on the queue and crashes every worker. What should have happened?

    • Retry until the fleet is empty
    • Gateway rejects it as poison / DLQ at enqueue, with a reason
    • Raise the worker memory and hope
    • Ask the model to compress it after the crash

14. Secrets and Security

    14. The model asks get_invoice with another tenant’s id. What should the tool use?

    • The model’s tenant argument, to be helpful
    • The authenticated job/user tenant from the gateway; deny mismatches
    • A random tenant
    • The system prompt’s company name

15. Tenant From the Gateway

    15. Where does the tenant filter belong?

    • In the system prompt: “only talk about this customer”
    • In every store query and tool, using the signed job tenant
    • In the user’s browser cookies
    • Only on the final sentence checker

16. Deploy

    16. A prompt edit should be treated as:

    • A harmless chat with the model
    • A versioned release that canaries and can roll back
    • Something only designers sign off on
    • An emergency kubectl edit on Friday

17. Canaries and Feature Flags

    17. What do you watch on a canary worker before you promote it?

    • Only that it started
    • Golden-tag success, forbidden tools, cost, HITL rejects, latency — then promote or roll back
    • Whether the prompt “feels nicer”
    • CPU on the laptop that deployed

18. Kill Switches You Have Practiced

    18. A single tenant’s swarm is melting the model bill. First lever?

    • Turn off the whole product
    • Pause that tenant’s queue / flag, leave everyone else running
    • Delete their traces
    • Raise max_steps so jobs finish

19. Evals in CI

    19. Which tests should run on every pull request without needing a paid model API?

    • Only LLM-as-judge on 10,000 live tickets
    • Tool/parser unit tests and golden agent cases with fake models and fixtures
    • Manual clicking in prod
    • A tweet to the vendor

20. What Blocks Merge

    20. A PR fails because a live web search golden flaked. What should you do?

    • Keep it red forever so nobody ships
    • Quarantine that row; do not block merge on live-web flakes. Do block on forbidden tools
    • Skip the whole gate this week
    • Replace CI with a thumbs-up in chat

21. Incident Playbooks

    21. What is the first move in a refund storm?

    • Rewrite the persona to be more careful
    • Disable the refund tool / pause the queue (contain), then inspect traces
    • Scale workers to 200 so jobs finish faster
    • Delete the ledger

22. Contain, Then See

    22. You see get_invoice rows for two tenants on one job. What class is that?

    • A creative model
    • A tool / tenancy bug — disable retrieval if needed, then patch authz
    • A reason to add more personas
    • Normal, because JSON is helpful

23. Every Incident Ends in a Golden

    23. The outage is “over” and nobody wrote a test. What is true?

    • You are done; customers moved on
    • You do not understand the incident yet — write the golden that would have failed before deploy
    • Tests are only for libraries
    • The model will remember not to do it

24. Boring Ops Is the Goal

    24. What makes an agent “in production” more than a public URL?

    • A longer constitution
    • A queue, traces, caps, tenancy, CI that can say no, and a practiced kill switch
    • Calling a larger model
    • Turning temperature to 0