Rewards and Policies

Reinforcement learning (RL) is learning from rewards, not from a correct label on every step.

A state is what you can see (the ticket, the last tool result). An action is what you do (call search, call sql, finish). A policy maps state → action, or state → chances over actions. A reward is a number after (or during) the episode: +1 if the ticket closed, −1 if you refunded the wrong order, 0 otherwise.

Supervised learning says “the label was sql.” RL says “you called search, then sql, then finish, and the user was happy.” The credit is delayed. That is harder. It is also how real agents get scored: the user does not label each tool call.

You do not need PPO to use the idea. You need a reward you can compute and a policy you can compare. Keyword policies, always-search, and “ask a human if p < 0.6” are policies. Average reward is the metric. Gradient methods are ways to raise that average when you cannot write the keywords. They still need a reward that matches the product.

Episodes, return, and delayed credit

An episode is one ticket, one conversation, one run until finish or budget death. The return is the sum of rewards (sometimes discounted, which means later rewards count less). A router that is 99% on step 1 and then loops until the budget dies has a terrible return. Per-step accuracy would lie.

Delayed credit is the pain: which action caused the +1? Maybe the second search, maybe the schema retry, maybe luck. Supervised routing on the first tool, when you can label it, is easier. Use RL-shaped scoring when the right action depends on later outcomes you cannot label step by step.

A reward that is “user clicked thumbs up” will teach click-bait. A reward that is “schema valid” will teach valid JSON that is still wrong. A reward that is “refunded so the chat stopped” will teach refunds. No reward, no RL — and a bad reward is worse than a keyword baseline.

Run to execute this in your browser. Nothing is sent to a server.

What printed: three average rewards over 200 sampled states (seed 0, so repeatable). Always-search is about 0.5 here because half the gold map is search. Random is about 0.5 with more noise. Keyword policy is 1.0 — it matches gold every time. Always-search is a policy. Random is a policy. Keywords are a policy. The average reward is the metric. You would not start PPO on this table. You would ship the keywords.

The reward here is dense (every episode, first action, +1 or 0). Real tickets are sparser. The print is the habit: name the policy, sample episodes, average the return, compare to dummies.

RLHF in one paragraph

RLHF / preference training is: humans (or a judge model) pick the better of two answers; a reward model fits that; a policy is pushed toward high reward. Same loop, bigger machinery. The eval track will treat judges as metrics. Here, remember the failure mode: if the judge likes polite wrong answers, the policy will be polite and wrong. The judge is a loss. Write it down.

You will rarely implement REINFORCE in this course. You will often implement return logging: per ticket, a number. That is the RL lens without the optimizer.

Exploration

If the policy never tries a tool, it cannot learn that the tool is good. Dead tools are dead ReLUs in policy space. Eval can force a tool. Production might allow a small chance of a safe exploration, or not — safety may forbid exploring shell. Exploration is a product choice, not only an algorithm.

Reward hacking and a three-step ticket

Reward hacking is getting a high number without doing the job. If you reward “user stopped messaging,” refunds win. If you reward “short traces,” the agent finishes too early. If you reward “JSON valid,” you get valid nonsense. If you reward a judge model that loves polite tone, you get apologies and wrong totals. The optimizer is innocent. The contract was sloppy.

Write rewards like losses: one sentence, computable, frozen. Example: +1 if goal_satisfied, −1 if a forbidden tool ran, −0.2 if schema failed, 0 otherwise. Average return on a freeze. Compare always-search. That is already RL as evaluation. Climbing that average with PPO is optional and late.

Credit assignment in a three-step ticket: search (useful), sql (wrong schema), sql retry (correct), finish (user happy). Who gets the +1? Supervised learning on first-tool labels would only score the search. Full-episode return scores the path and cannot say which step mattered. If you can label the first tool, do that. If the bug is “loops until budget dies,” the return catches it and per-step accuracy may not.

Do not call “we added a critic prompt” RLHF. A judge is a metric. RLHF is a training loop that uses a fitted reward model to push a policy. You can use a judge every day without that loop. Most teams should.

Common mistakes

• Per-step accuracy instead of episode return.
• Rewarding chat closure.
• Calling a prompt tweak “RLHF.”
• Training RL on a reward you cannot recompute on a freeze.
• Skipping the keyword policy comparison.

How agents use this

Every think-act loop is a policy. max_steps, “stop on goal_satisfied,” and “ask a human if p < 0.6” are parts of that policy. Log the episode return (sum of rewards) per ticket, not only per-step accuracy.

If you can label the correct first tool, start with supervised routing. Use RL-shaped metrics when the path matters. Use RL optimizers last, after a dummy, a rule, and a small supervised model lose on honest data — and after the reward matches the product.

Tip:If you can label the correct first tool, start with supervised routing. Use RL when the right action depends on later outcomes you cannot label step by step.