Ingest and Freshness

The rest of the pipeline is the product: ingest, clean, chunk, embed, index, retrieve, rerank, generate, cite. If you only tune the prompt, you are polishing the last 10%. Users blame “the AI.” The page they needed was a draft, a secret, or last indexed in 2024.

Ingest is the job that takes raw sources (files, wiki pages, tickets you actually meant to index) and writes chunks plus metadata into the store the retriever reads. Freshness is how close that store is to the source of truth right now. The index is only as true as the last successful ingest. A failed job with a green chat UI is a liar.

An indexing job is part of the agent’s SLO (the reliability target you quote). If runbooks update at 10:00 and the index lags until tomorrow, the agent will quote yesterday. That is not a model bug. That is an ops bug with a citation.

Filter before you split

Do not embed secrets, draft Confluence pages, or “ignore previous instructions” junk. Chunking is a great way to permanently vectorize a password someone committed. Once it is in the index, every similar query can retrieve it, and every trace can log it.

Filter before you split into chunks:

• Status must be published (not draft, not archived, not personal sandbox)
• Drop pages that look like injection (“ignore previous instructions”)
• Drop obvious secrets (password, api key patterns you actually scan for)
• Drop boilerplate nav and cookie banners so they do not become “policy”
• Honor ACL at ingest: if the page is not readable by the tenant who will query, do not put it in that tenant’s index

Cleaning is part of ingest. HTML to text, strip scripts, keep headings. A chunk that is 80% sidebar links will retrieve for “home” and “login” forever.

Metadata you keep on every chunk

Keep on each chunk, or you cannot operate:

• source path or URL (what the user will open)
• version or checksum (so you know which bytes you embedded)
• indexed_at time (so you can say “this quote is from Tuesday’s index”)
• status or ACL fields you will filter on later (tenant_id, lang, product)

If you cannot say when a chunk was indexed, you cannot trust it in ops. “According to our docs” with no indexed_at is a vibe.

Deleted sources need a policy: tombstone (mark gone and drop from retrieve) or rebuild. An index that still quotes a deleted refund exception is a product incident. Incremental ingest that only inserts and never deletes is how zombies live.

Incremental vs full rebuild

A full rebuild re-reads every source and replaces the index. It is slow and easy to reason about. An incremental job re-embeds only checksums that changed. Incremental is how you stay fresh. It is also how you forget to delete. Pair every upsert with “if checksum missing from source, drop those chunk ids.”

Re-embed when the embedding model changes, not only when the file changes. Mixing two models in one index is random retrieval. Store the model name next to every vector. That lesson returns in embeddings. Ingest is where you record the name.

Run to execute this in your browser. Nothing is sent to a server.

The draft never enters the index. The injection sentence never becomes a vector. The 2024 reboot page is too old for this classroom’s max-age rule. Production max-age is a product choice: some legal policies must stay for years; some runbooks must die in a week. The code is the same idea: a rule at ingest, not a hope at generate.

Freshness is observable

Monitor ingest like you monitor the model API: success rate, lag from source updated to indexed_at, skip reasons, count of chunks. Alert when lag exceeds the SLO. Expose indexed_at in traces so a human can see “quoted Tuesday’s index on Wednesday after an edit.”

A source that fails to parse should fail the job (or quarantine that source), not silently skip while the old chunks remain. Silent skip plus old chunks is stale-with-a-smile.

Deletes, checksums, and two clocks

Sources have an updated time. Chunks have indexed_at. Freshness is the gap between them, plus whether a delete in the source became a delete in the index. If a legal page is removed and the index still quotes it for a month, you did not have incremental ingest. You had insert-only ingest.

A checksum (hash of the bytes you actually embed, after cleaning) tells you whether to re-chunk. Filename + “looks the same in the UI” is not a checksum. Two wiki titles can collide. A move that keeps the title and changes the path should still re-id chunks if your ids include the path.

Legal policies and runbooks want different max-age rules. A terms-of-service page from 2024 may still be binding. A “reboot the rack” runbook from 2024 may be dangerous. Ingest rules are per collection: max_age_days is not global. The classroom used 365 as a demo, not as a law.

Boilerplate is a freshness bug in disguise. If every page starts with the same 500-character nav, every vector shares a direction and retrieve becomes “the wiki, generally.” Cleaning at ingest is how chunks stay about this heading. Measure: sample 20 chunks; if 10 start with “Skip to content,” your cleaner failed.

Secrets scanners will never be complete. Still run them. A password that enters the index will be retrieved by “how do I log in” forever, and traces will log it. Once indexed, deletion must be as loud as ingest success: drop those ids, then confirm retrieve no longer returns them with a test query.

Common mistakes

• Embedding drafts “so retrieval is complete.” Completeness includes poison.
• No checksum, so you re-embed nothing or everything.
• No delete path. The index is an attic.
• Measuring “pages crawled” instead of “chunks searchable and fresh.”
• Putting secrets in traces because ingest did not filter them.

How agents use this

The retrieve tool can only search what ingest wrote. If a field engineer updated the OOM runbook at 10:00 and the bot quotes 2024 at 10:05, the fix is the indexer, not a sterner system prompt.

When you later attach retrieve to an agent, pass indexed_at and source path through to the trace. On-call should see skip counts next to model latency. A green chat UI with a dead indexer is a liar. Treat ingest failures as product failures.

Do not wait for the Agents track to start logging ingest. The library is already a production system.