Unsupervised Learning

Unsupervised learning is what you do when nobody labeled y. The questions change: which points clump? Which traces are unlike the rest? Can we compress this conversation?

You still evaluate — just not with “accuracy against gold” unless you later obtain gold. You look at cluster sizes, stability, and whether a human can name the cluster. Unnamed clusters are not a product. A picture of blobs is a draft taxonomy.

Do not call this a classifier. Do not report “accuracy” against names you invented after looking at the blobs unless you held out a labeling pass. Promotion into supervised learning is a later, honest step.

K-means in one paragraph

1. Place k centroids (means) in the space
2. Assign each point to the nearest centroid
3. Move each centroid to the mean of its points
4. Repeat until assignments stop changing (or you hit a step budget)

It minimizes within-cluster squared error. k is a choice. Init matters. Clusters are blob-shaped. Weird rings will be sliced badly. For a first map of embedding space it is still the right hammer.

Scale your dimensions: a job id in the thousands will dominate a 0–1 flag. Embed first, then cluster the vectors. Running k-means on raw chat tokens is hopeless.

Run to execute this in your browser. Nothing is sent to a server.

What printed: five steps of assignments and centroids. The assignment list should snap to two groups: zeros for the origin blob, ones for the (5, 5) blob (or the reverse, depending on which seed is which). Centroids move toward the mean of each blob and then sit still. That is k-means doing its only trick: nearest mean, then update the mean.

Init here was honest (one seed from each blob). Random init can land both centroids in one blob and stall. Run twice. If points jump clusters, the structure is weak or k is wrong.

Choosing k

The “elbow” of error vs k is a heuristic: plot within-cluster error, look for a bend. For product work, pick k you can label in a meeting. Five named failure modes beat twenty anonymous ones.

Anomaly detection is the cousin: a point far from every centroid is a candidate for a new tool or a new eval case. Do not auto-delete outliers. They are often the incident.

Other unsupervised tools exist (compression, topic sketches). The agent-relevant output is still a nameable group or a flagged oddball. If you cannot name it, you cannot write a rubric, and you cannot supervise later.

From clusters to labels

Then promote a cluster into a supervised label (“this cluster is SQL-timeout; add a retry”). Unsupervised finds the taxonomy; supervised ships it. Humans name a sample from each blob. Disagreement means the blob is mixed: split, or drop.

Clustering failed runs by embedding the last tool error is the usual win. You will find “timeout,” “schema,” and “user cancelled” as separate weather systems. Clustering user goals may show that 40% of volume is “reset password,” which should be a workflow, not an agent.

How to inspect a blob without lying

After k-means, read twenty traces from each cluster. Write a name in a meeting or refuse the cluster. “Cluster 3” is not a finding. “SQL timeouts after the billing API 500s” is a finding. If those twenty traces disagree, the blob is mixed: raise k, change features, or split by a flag you already have (HTTP status).

Stability is a check: run twice with different seeds (or a different pair of starting points). If many points jump, do not ship a taxonomy. Either k is wrong or the space has no blobs, only a smear. A smear still has outliers — far from the mean of everything — which are often new incident types. Sample those. They are eval cases.

k-means will slice a ring or a banana into pie pieces. Embeddings of language are not always round blobs. If inspection shows a gradient (easy to hard tickets) instead of types, stop clustering and build a supervised score instead.

Never report accuracy against names you invented from the same points you clustered. That is circular. The honest path is: cluster on a large unlabeled pile, name from a sample, then label a held-out sample with those names and train or evaluate a classifier. Unsupervised found the menu. Supervised takes the order.

Empty clusters (a centroid with no points) happen if you init badly. Re-seed. Do not interpret an empty cluster as “a rare failure mode we discovered.”

Compressing a conversation into a short list of numbers (an embedding) is unsupervised too: you keep geometry, you drop words. That list is only useful if neighbors still mean the same job. Check with a handful of known pairs (two password tickets close, a password ticket far from an invoice). If those pairs fail, clustering on the lists will fail. Fix the encoder and the text you embed (last error, not the whole fluffy chat) before you pick a new k.

Common mistakes

• Accuracy on names you invented from the same points.
• k-means on unscaled raw features.
• k = 20 because the plot looked busy.
• Treating the cluster id as a gold tool name.
• Never looking at example traces from each blob.

How agents use this

Use unsupervised work to see. Use supervised work to act. A weekly cluster of failures is an ops habit with ML shape: embed, k-means or even just nearest-centroid to last week’s named means, sample, name, promote.

When a new blob appears, that is drift’s cousin: the world grew a kind of ticket you did not label. Do not silently absorb it into the nearest old centroid. Make a new eval case.

Note:Running k-means on raw chat tokens is hopeless. Embed first, then cluster the vectors.