Metrics That Page a Human

If you only plot “tokens used,” finance will find you before you find the bug. If you page on every model timeout, on-call will learn to ignore the pager — correctly. Metrics that matter for agents are symptoms that need a human, plus a few product health numbers you review without a siren.

This is operations, not eval theory. Golden-tag success in prod is sampled from real jobs and from CI; you treat it as a rate you can chart, not as a lecture on how to write goldens. Forbidden-tool count is a page. A slightly longer system prompt is not a metric.

A pager that never sleeps trains people to ignore it. A dashboard that never pages trains people to skip it. You want both, with a bright line.

How the box actually works

Split signals into page, ticket, and dashboard.

Owners: runtime emits the counters (from the same spans as the last lesson). On-call owns the page/no-page line and reviews it after every incident. Product owns “time to first event.” Finance owns the cost cap number, not the query.

Exemplars: every page includes a job_id. Averages without exemplars are how you argue in the channel instead of opening a trace.

Do not page on every LLM timeout. Those belong on a dashboard with a circuit breaker. The breaker is the control. The dashboard is how you see it working. The page is for when the breaker is stuck open too long or when a forbidden tool slipped.

Write the page/no-page line in the same repo as should_page. Thresholds that live only in a SaaS UI will drift from staging. Review the function after every incident: demote pages that never change action, add pages that you learned about from a customer tweet.

A pager-storm ticket

The first week of “production,” the team paged on llm_timeout > 0. A regional blip in the model vendor fired 200 pages. People muted the service. Two days later a refund tool leaked on a canary (forbidden-tool = 3). Nobody came. The mute had no expiry.

The fix was should_page: forbidden tools, cost, queue, HITL SLA. Timeouts alone returned timeouts_on_dashboard. They added a circuit breaker and a slower “vendor unhealthy for 15 minutes” ticket, not a page per job. The next refund leak paged. Someone actually answered.

Run to execute this in your browser. Nothing is sent to a server.

storm pages because forbidden_tool is 3 (cost is under the default cap of 1 dollar per minute). timeout only does not page: twelve model timeouts go to the dashboard. queue pages because depth 500 lasted at least 10 minutes. A refund-tool leak wakes a human. A handful of model timeouts does not. That is the bright line in code.

What goes wrong

Paging on symptoms you cannot act on at 2 a.m. (“vibe is off”). Paging on tokens during a launch you planned. Never paging on forbidden tools because “it might be a false positive” — then you have no signal. Thresholds copied from a web app (error rate 1%) that hide a single wire call.

Queue depth without a duration pages during a 30-second deploy. Require “high for N minutes.” Cost without a tenant split pages the whole company for one runaway customer — you will want per-tenant pause (later).

Dashboards with 40 charts and no saved “failed jobs” query. On-call opens Grafana, not the trace.

How to test it

Unit-test should_page with the four cases in the tryit plus: cost over cap, HITL over SLA, timeouts and forbidden (must still page). Snapshot pager payloads: must include job ids / trace links.

Game day: fire a fake forbidden-tool metric in staging, time to human ack. If nobody knows which channel, the metric is decorative.

Review pages weekly. If a page never leads to action, demote it. If an incident had no page, add one.

How agents use this

Keep a saved query: last 20 failed jobs with tool names and dollars. That query is more useful than a heatmap of tokens.

When you add a dangerous tool, add a forbidden-counter and a page in the same PR. When you add HITL, add the SLA clock. When you add a queue, add depth-over-time.

Circuit-break a vendor instead of paging each timeout. Caps and breakers are controls. Pages are for controls that failed or for blast radius that is already happening.

Write the page/no-page table in the runbook next to the kill switches. At 2 a.m. people will not invent a philosophy of alerting.

Mute with an expiry. A mute that lasts forever is how the next forbidden-tool leak is silent. If a launch will raise tokens, put that on the dashboard in advance rather than paging yourselves for a plan you already had. If a tenant is paused, page on “pause lasted more than SLA” so someone remembers to unpause.