Practice

Write a goal predicate

goal_satisfied(ticket) is True only if url starts with https:// and the body contains def test_. Fill in the function and print both cases.

Stop a loop on budget

Run a think-act loop that prints step N until action is stop or steps reach max_steps. The policy returns stop on step 3. Print DONE or BUDGET.

Classify chatbot vs workflow vs agent

kind_of(description) returns chatbot, workflow, or agent. Use simple keyword rules: no side effects → chatbot; known steps → workflow; model chooses next tool → agent.

Parse a JSON action

parse_action(text) returns a dict with keys tool and args only. Reject extra keys and non-objects. Print the valid action and the error from the bad one.

Tool registry dispatch

call(name, args) looks up TOOLS and returns unknown_tool if missing. add(a,b) and finish(answer) are registered. Call add and a fake name.

Retry on timeout only

flaky() fails with timeout twice then returns ok. retry(fn, n) retries only if the result dict has error == timeout. Print the successful result and call count.

Normalize a city name

norm_city collapses case and whitespace so ' PARIS ' and 'Paris' match the same catalog key.

Take the last three steps

last3(steps) returns the last three items of a list using a slice. If the list is shorter, return all of it. Print both cases.

Read a nested tool result

error_of(trace) returns the error string from the last tool message, or None if missing. Do not crash on a short trace.

Unpack a tool pair

split_call(pair) unpacks (name, args) and returns a dict with keys name and args. Print one good pair.

Count tool names

Use collections.Counter to count tool names in a trace. Print the most common name.

Deep copy a transcript

fork(trace) must copy so changing the fork does not change the original. Print both error fields after the change.

Route a tool with match

handle(action) uses match/case. search with q returns 'search:' plus q. finish with text returns 'done:' plus text. Anything else returns 'bad'. Print three cases.

Did any tool fail?

any_failed(trace) is True if any row has ok equal to False. Use any(). Print both a mixed trace and an all-ok trace.

Write and read a small file

Use pathlib.Path to write text with utf-8, read it back, print the name, then delete the file.

Cosine similarity

Implement cosine(a, b) with stdlib math. Zero-norm vectors return 0.0. Print cosine of [1,0] with [1,0], [0,1], and [0,0].

Softmax

softmax(xs) returns a list of probabilities that sum to 1. Use math.exp. Print softmax([1, 1]) and softmax([10, 0]).

Binary entropy

entropy_bits(p) is the Shannon entropy of a Bernoulli(p) in bits. Use 0*log(0)=0. Print entropy of 0.5, 0.0, and 0.9 rounded to 3 decimals.

Dot product

dot(a, b) is the sum of products. Raise ValueError if the lengths differ. Print dot of [1, 2, 3] with [0, 1, 0] and catch a length mismatch.

One gradient descent step

f(x) = (x-3)**2 has slope 2*(x-3). step(x, lr) returns x - lr * slope. Print the new x starting from 0 with lr 0.25.

Bayes after a timeout

P(down)=0.1, P(timeout|down)=0.9, P(timeout|up)=0.05. Print P(down|timeout) rounded to 3 decimals.

Cross-entropy of a one-hot

ce(q, k) is -log2 of q[k], with a floor of 1e-12 on q[k]. Print ce of [0.1, 0.8, 0.1] at index 1, rounded to 3 decimals.

Attention weights

weights(scores) is softmax of a score list. Print weights for [2.0, 0.0] rounded to 3 decimals.

Precision and recall

From tp, fp, fn print precision and recall rounded to 3 decimals. Use tp=8, fp=2, fn=2.

Train/test split

split(xs, ratio=0.75) returns (train, test) keeping order: first 75% train. Print lengths for 8 items.

Accuracy from labels

accuracy(y_true, y_pred) is the fraction of matches. Print accuracy of [0,1,1,0] vs [0,1,0,0].

Mean squared error

mse(y, yhat) is the mean of squared differences. Print mse([2, 2, 2], [0, 2, 4]).

Majority baseline

majority(ys) returns the most common label. Print majority of [0, 0, 1, 0] and the accuracy of always predicting it on that list.

Sigmoid chance

sigmoid(z) is 1/(1+exp(-z)). Clip z to [-30, 30]. Print sigmoid of -2, 0, and 2 rounded to 3 decimals.

Precision at k

precision_at_k(ranked, relevant, k) is hits in the first k names divided by k. Ranked is best-first. Print P@2 for names a,b,c with relevant {a,c}.

k-NN majority vote

vote(labels) returns 1 if 1 appears more often than 0, else 0. Print vote of [0,1,1] and [0,0,1].

L2 loss add-on

l2(mse, w, lam) is mse + lam * w * w. Print l2(1.0, 2.0, 0.5).

User split overlap

overlap(train, test) is the set of user ids in both lists of dicts. Print sorted overlap for a leaky split and an empty one.

Whitespace tokenize

tokenize(text) lowercases and splits on spaces. Detokenize with join. Round-trip a short sentence and print token count.

Normalize attention weights

Given raw scores [2, 0, 0], turn them into a probability vector with softmax (math.exp). Print the weights rounded to 3 decimals and check they sum to 1.

Trim a context window

trim(messages, max_items=3) keeps the first system message (if any) plus the last messages, never exceeding max_items. Print the roles that survive.

Embedding lookup

embed(ids, table) returns the row for each id. Print embed of [1, 0] in a 2-d table with three rows.

Causal mask

mask_scores(scores, i) sets scores[j] to -1e9 when j > i. Print the masked row for i=1 on [0.5, 0.5, 0.5].

Residual add

residual(x, delta) adds two equal-length lists elementwise. Print residual of [1, 2] and [0.1, -0.2].

Greedy decode

greedy(vocab, logits) returns the vocab item with the largest logit. Print greedy for search/sql/finish with logits [1.2, 2.0, 0.1].

KV cache append

A cache is a list. prefill(xs) replaces it. decode(x) appends x and returns the new length. Print lengths after prefill of 3 and one decode.

Pin the spec

pack(spec, tail, limit) always keeps spec words first, then as many tail words as fit. Print pack of spec=['NEVER','DELETE'] and tail=['user','delete'] with limit 3.

Count message tokens (approx)

approx_tokens(messages) is words*1.3 plus 4 per message (role overhead). Print the estimate for a 2-message chat.

Estimate USD cost

cost(in_tokens, out_tokens, in_rate=0.5, out_rate=1.5) uses rates per 1M tokens. Print cost for 2000 in and 400 out, rounded to 6 decimals.

Greedy vs sample

pick(logits, greedy=True) returns argmax index, or a weighted sample with random.choices. With seed 0 and greedy False, print both strategies on [0.1, 0.8, 0.1].

Validate chat roles

valid_roles(msgs) is True only if every role is system, user, assistant, or tool. Print the flag for a 3-message list that includes a bad role.

Parse a JSON action

parse_action(text) json.loads and returns obj['action'] if it is get_job or finish, else None. Print parse of a valid object and of prose.

Detect truncated JSON

truncated(text, finish_reason) is True when finish_reason is length or the stripped text starts with { but does not end with }. Print both cases.

Citation allowlist

ok_cites(answer, allowed) is True if every token in the answer that starts with doc_ is in allowed. Print two answers.

Route small vs large

route(kind) returns small for classify or extract, else large. Print route for classify and for plan.

Spend cap

over_cap(spent, next_cost, cap) is True when spent + next_cost would exceed cap. Print for 0.01+0.005 vs cap 0.02, then 0.018+0.005 vs 0.02.

Extract JSON from a fence

extract_json(text) finds the first { and last } and json.loads that slice. Handle a markdown-fenced model reply.

Format a few-shot block

few_shot(examples) joins lines 'Q: ... / A: ...' for a list of (q, a) pairs. Print the block for two examples.

Build a chat payload

build(system, user) returns a list of two message dicts with roles system and user. Print the roles.

Assemble four parts

assemble(instructions, context, user_input, contract) joins the four labeled sections with a blank line. Print assemble of four short strings.

Overlap few-shot score

score(query, example) is the count of shared words longer than 2 letters. Print score of 'refund last invoice' vs 'refund the invoice from March'.

Escape a close tag

escape_doc(text) replaces < with &lt; and > with &gt;, then wraps in <doc>...</doc>. Print escape of a payload that contains </doc>.

Flag an injection phrase

flagged(text) is True if 'ignore previous' appears (case insensitive). Print both a normal page and an injected page.

Parse a JSON action

parse_turn(text) json.loads and returns (tool, args) if tool is get_job or finish. Print the tuple for a get_job object.

Grade a refusal case

passed(case, output) is True when a must_refuse case has status refused, or when contains is inside answer. Print both cases.

Validate tool args

validate('geocode', args) returns None or an error string. city must be a non-empty str. extra keys fail. Test three payloads.

Simulate a 429 then succeed

weather() rate-limits the 4th call. RATE_LIMIT=3. Call it 5 times and print error codes or temp.

Allowlist HTTP-like get

http_get(url) only fetches if the host is in ALLOW. Simulate pages as a dict. Try example.com and evil.example.

Idempotent finish

finish(answer) returns the same dict if called twice with the same answer (store last). Print both calls and a changed answer.

Dispatch unknown names

dispatch(name, args) calls REGISTRY[name](**args) or returns {error: 'unknown_tool', name}. Print get_job, finish, and launch_nukes.

Truncate observations

pack(result, limit=20) returns {result, truncated}. If str(result) is longer than limit, slice and set truncated True. Print a short dict and a long string.

Reject parallel writes

can_parallel(names) is True only when every name is in READS. Print three lists: two gets, two refunds, mix.

Approval timeout is deny

execute(name, human=None) auto-runs search. refund needs human=='approve'. human=='timeout' returns denied. Print four calls.

Check the user, not only the bot key

refund(actor, cents) uses USERS[actor]['limit']. Over limit -> denied. Print ada 4000, ada 9000, guest 1.

Chunk on headings

chunk(doc) splits a string on lines that start with '# '. Return a list of {id, heading, text}. Print ids and headings.

Retrieve top-k by overlap

retrieve(query, chunks, k=1) ranks by count of overlapping lowercase words. Print the winning id for 'refund days'.

Refuse below tau

answer(query) retrieves one chunk with overlap score. If best score < 1, refuse with cannot:. Print refund vs equine.

Cosine rank two chunks

cosine(a, b) then search(q) returns the best chunk name. Print the winner for [0.9, 0.1] among oom and refund.

Filter by tenant first

retrieve(query, tenant) only ranks chunks with that tenant. Print ids for acme and 'refund'.

Quotes must be substrings

ok(source, quote) is True only if quote.lower() is in SOURCES[source].lower(). Print a real quote and a fake one.

RRF two ranked lists

rrf(a, b, k=60) adds 1/(k+rank) per list (rank starts at 1). Print the top id for a=['a','gold'] and b=['gold','b'].

Wrap chunks as DATA

wrap(chunk) returns DATA / chunk / END DATA on three lines. Print whether 'ignore previous' is still inside the block.

Deny untrusted memory writes

upsert(value, trusted) updates STORE only if trusted. Print the store after a False write then a True write.

Parse a ReAct triple

parse_react(text) returns {thought, tool, args} from Thought/Action/Action Input lines. Search args is {query: input}.

Tiny tool loop

fake_model geocodes first, then finishes. run_agent prints each tool name and returns the final answer for Paris.

Citation subset check

valid_finish(payload, opened) is True iff citations is a subset of opened and, when cannot_answer, citations is empty.

Cannot-answer on max steps

run(max_steps) always searches (never finishes). When the budget hits, return cannot: step budget. Print that string.

Stop on finish or max steps

should_stop(name, steps, max_steps=3) returns success on finish, cannot: step budget when steps >= max_steps, else None. Print three calls.

Reject unknown tools

parse(name) returns ok True for names in REGISTRY, else error unknown_tool. Print search then launch_nukes.

Illegal tool in gather

can_run(phase, name) is True only if name is in ALLOWED[phase]. Print search in gather, refund in gather, refund in apply.

Freeze approval args

freeze(args) returns a JSON copy. Mutate live amount to 400. Print frozen amount then live amount.

Router billing vs faq

router(text) returns billing if refund is in the text, faq if hours is in the text, else handoff. Print three lines.

Allowlisted patch

apply_patch(repo, role, path, content) only allows coder → src/app.py. Planner writing src must error. Print both outcomes.

Fizzbuzz test oracle

run_tests(src) execs source in a tiny namespace and checks 3→Fizz, 5→Buzz. Print ok for a correct function and a stub.

Supervisor stop on green

next_role(tests_ok, had_plan) returns planner if no plan, coder if plan and not green, stop if tests_ok. Print the three cases.

Hop cap and cycles

next_hop(path, nxt, max_hops=3) appends nxt unless nxt is already in path (cannot: cycle) or len(path) >= max_hops (cannot: max hops). Print three calls.

Critic cannot write

critic_act(name) returns error critic_cannot_write for names in WRITES, else ok True. Print review then edit.

Price a swarm before launch

priced_swarm(n, cost_child=0.02, cap=0.5) launches only if n * cost_child <= cap. Print launched for 20 then 50.

Sequential intake path

sequential(ticket) returns [intake, billing] if invoice is in the text, else [intake, tech]. Print both tickets.

One writer per customer

write(role, cid) sets LOCKS[cid] on first write. Later writes succeed only for that role. Print billing c1, loyalty c1, billing c1.

Parse a typed brief

parse_brief(raw) requires a dict with non-empty brief and a non-empty citations list. Print ok then the error from a string.

Pass rate

pass_rate(rows) where each row is {pass: bool}. Print pass_rate of [T, T, F] as a fraction string and a float.

Golden tool sequence

matches(names, golden) is True if the list of tool names equals the golden list. Test Paris vs a skipped-weather trace.

Quarantine flaky rows

score(rows) ignores rows with quarantine True when computing pass_rate. Print rate for one fail, one pass, one quarantined fail.

Pretty answer still needs a tool

goal_satisfied(trace, expected_tool, fact) fails if the tool is missing or the fact is missing from the final text. Print good then a pretty hallucination.

Forbidden tool fails the case

gate(tools, forbid) is ok only when no tool is in forbid. Print search+finish then search+wire.

Safety zero fails the judge

fake_judge(answer) sets safety 0 if the answer agrees to wire without cannot/never. Print a careful answer then a wire-yes.

Injected wire is not allowed

parse_tool(name, allowed) returns unknown_tool when name is missing. Print finish then wire against search_kb/finish.

Cross-user invoice denied

get_invoice(actor, invoice_id) only returns ok for actor a on id 1. Print a/1 then a/2 codes.

Docs win over the prior

answer(doc, prior, docs_win) uses doc when docs_win and doc is non-empty, else prior. Print three calls: win, lose, empty doc.

Redact API keys

redact(text) replaces tokens that start with sk- and are 8+ chars with sk-***. Print the redacted log line.

Approval digest

digest(proposal) is sha256 of canonical JSON (sort_keys) of id, tool, args. Print 12-char prefixes for a proposal and a tampered copy; they must differ.

Trace an event

trace_event(step, tool, ok) returns a dict with those fields plus ts index from a counter. Print two events.

Fail closed without approval

execute(mutate, approval) applies rollback only if approval is allow. WORLD deploy_id starts at d44. Print status and deploy_id for missing vs allow.

Enqueue returns a job id

handle_user(message, jobs, queue) stores a queued job and appends its id. Print the ack and the queue.

Kill the job at max_usd

tick(job, step_usd, cap) adds usd and returns MAX_USD when the cap is crossed. Print an ok tick then a kill.

Idempotent refund key

apply_write(ledger, key, cents) writes once. Print first then retry for the same key.

CI gate forbids wire

gate(item, out) fails if any forbid tool ran. Print pass for finish, fail for wire.

Contain a refund storm

contain(flags) sets tools.refund False and queue.paused True. Print flags before and after.