Features

A feature is one number (or a short list of numbers) you extract from a raw example. The model never sees the ticket. It sees the features you chose.

That sentence is easy to skip and expensive to skip. If you pick the wrong features, no amount of training will save you. If you pick a feature that is the answer, the model will look perfect and fail in production. That second bug is leakage. We will meet it again on the splits page. Here, the job is to see features as a contract: “at decision time, the model is allowed to know exactly these numbers.”

Raw input is messy: a chat, a PDF, a stack trace, a user tier. A model wants a list of floats. Feature engineering is the work of turning messy into that list without cheating.

From text to numbers

A tiny, honest start is a bag of words: a 0 or 1 for each word you care about.

[has_down, has_refund, has_please]

The sentence “the site is down” becomes [1, 0, 0]. “please refund” becomes [0, 1, 1]. The model only sees that list. Order disappeared. “down the site is” would look the same. That is a limitation, not a mystery. Bags of words are still useful for routers: a handful of domain words often beat a vague embedding of chat fluff.

Counts (how many times), lengths (how many tokens), and flags (is the user VIP?) are also features. One-hot features are a list of 0/1 flags for a category: plan is free, pro, or enterprise becomes three slots, one of them 1. Do not feed a tool id as a single integer 0, 1, 2, 3 if those numbers are not ordered. The model will treat 3 as “more” than 0.

Embeddings (later) are features too — a long list from another model. You did not hand-write those slots. You still chose the encoder, the text you fed it, and whether you mixed that list with flags. Choice of input is still feature work.

Missing values need a policy. “No last error” is not the number 0 unless 0 already means something. A common pattern is a flag has_last_error plus a code. Silent zeros invent fake structure.

Run to execute this in your browser. Nothing is sent to a server.

What printed: the word list, then four feature lists. [1, 0, 0] for the outage sentence — only “down” fired. [0, 1, 1] for the refund ask. [0, 0, 0] for the greeting: the model sees nothing domain-like. [1, 1, 1] for the last line, which has all three flags on. That last list is a richer input than the greeting. Same idea as a real featurizer, only smaller.

Split on whitespace is crude. “down.” with a period would miss the flag. Real tokenizers and a small word list you maintain beat cleverness. The lesson is the shape: raw text in, a list of numbers out, and the model is blind to whatever you did not encode.

Scale, units, and dominance

If one feature is latency in milliseconds (800) and another is a 0/1 flag, the big number will dominate any distance or any unscaled linear model. Divide each column by a typical size, or keep them in separate heads. Mixing units without scaling is a common silent bug.

Standardizing (subtract a typical center, divide by a typical spread) is one habit. Min-max scaling to 0–1 is another. Either is chosen on train statistics and frozen. Computing the scale on train-plus-test is leakage: the test rows nudged the scale.

Counts of rare words can be huge on one ticket and zero on others. Logs or caps (“token count, but at most 4000”) keep one explosion from owning the score. You are allowed to use human sense. Features are not more scientific because you refused to clip them.

Leakage: features that are the answer

A feature that exists only after the action must not be in the input at decision time.

Examples that have shipped:

• status=resolved while you predict “resolved”
• The final assistant message, while you predict which tool to call first
• The close reason, the refund amount, the human’s later tag
• Retrieval that indexed the eval question itself
• A timestamp of “when the ticket closed” used to predict “will it close”

If a human sitting at decision time could not have known the number, the model may not use it. Write features as of a cutoff: the last user message, tools already called, last error type, token count so far, cosine to each tool doc.

The opposite bug is starvation: you hid the only signal that existed. If the router cannot see that SQL was already called twice and failed, it will call SQL a third time. Decision-time state is a feature.

What to put in the list for an agent

Before you fine-tune anything, write the features a router would see. If a human cannot guess the label from that list, a tiny model cannot either.

Useful families:

• Text flags and counts from the latest user turn (and maybe a short window, not the whole year of chat)
• Tool history — names already called, how many times, last error class
• Retrieval scores — top cosine, gap between first and second neighbor
• Budgets — steps left, tokens so far
• Identity that is allowed — plan tier, language, region, if policy may use them

Useless or dangerous families:

• Gold labels sitting in the table
• Future messages
• Global “this user is difficult” scores computed from the whole ticket including the end
• IDs that encode the answer (ticket prefixes that mean “VIP refund queue”)

Common mistakes

• Feeding raw Unicode and hoping the linear model “reads.” It sees numbers you made, or it sees nothing.
• Concatenating an embedding and a millisecond latency without scaling.
• Using the same feature list for “which tool?” and “was the final answer good?” Those are different times, different x.
• A 500-word bag with 12 labeled rows. You will memorize which rare word appeared in the one urgent ticket.
• Treating cosine 0.82 as a feature named “82% true.” It is an angle. Later: calibration.

How agents use this

A production router is often this list: bag of a few domain words, or an embedding of the utterance, plus tool-history flags, plus a budget. A retrieval step is features too: each chunk’s vector is the feature; the query’s vector is the other feature; the score is geometry.

If the list includes the future, you cheated. If the list is only “the whole chat as one blob” and the decision needed the last error, you starved the model. Draw the line at decision time, write the numbers, and then pick a function. Features first. Model second.

Watch out:A feature that exists only after the action (the final assistant message, the close reason) must not be in the input at decision time.