Documents and Cleanup

Overview

trigger/documents.ts owns document ingestion, embedding, and corpus maintenance for the Library/knowledge base. trigger/document-purge.ts owns the two-stage, SABLE-only document deletion workflow (tombstone, then a retryable Storage delete). trigger/feedback.ts owns retention cleanup of expired in-app feedback attachments. All three are grouped here because they are "keep the Library and its supporting tables correct and tidy" workloads, distinct from the external-connector syncs (connectors.md) and the board/report pipeline (reports.md).

How it works

Indexing. indexDocument is the fast path: triggered when a document is uploaded, it calls indexUploadedDocument (loaded lazily from web/api/_document_ingest.js via createRequire, since it's a shared CommonJS module) to chunk and embed the document, reporting progress through Trigger's metadata. backfillDocumentCorpus is the batch path: given a targetModel (voyage-4 or voyage-context-4), it scans active documents whose active_embedding_model doesn't match, skips structured spreadsheets (handled by a separate pipeline), and re-indexes a bounded batch at a time, cursor-paginated. It defaults to dryRun: true and requires the literal string "REINDEX_ACTIVE_CORPUS" as confirmation before it will actually write.

Reconciliation. reconcileDocumentIngestJobs calls the reconcile_stale_ingest_jobs RPC, then separately cleans up superseded rows left behind by re-indexing: it deletes from knowledge_chunks and spreadsheet_rows in bounded batches (STAGED_CLEANUP_BATCH = 2,000 rows) inside a 40-second wall-clock budget per run, rather than one unbounded DELETE. This batching exists specifically because an unbounded delete over 582,200 superseded rows (left by a 2026-09-08 spreadsheet re-index) blew PostgREST's 8-second statement_timeout on every one of 235 attempts over 20 hours, scanning the table and deleting nothing each time.

Purge. documentStoragePurge is the direct-dispatch fast path for a purge request: it claims a document_storage_deletion_jobs row (via the claim_document_storage_deletion_job RPC), validates the target is a fixed documents-bucket path scoped to the job's organization (validDocumentStorageTarget), deletes the Storage object, and finalizes the catalog purge (complete_document_purge RPC). documentStoragePurgeSweep is the durability backstop: on its own cron, it calls documentStoragePurge directly (triggerAndWait) to catch a job whose direct dispatch failed or whose worker died mid-claim.

Feedback retention. feedbackAttachmentRetention deletes feedback_attachments rows whose retention_expires_at has passed, removing the Storage object first and then flipping status to deleted — batched at 100 rows per run, throwing (and retrying) if any single deletion fails so a persistent Storage error surfaces instead of silently under-reporting.

Tasks

Task	Trigger / cron	What it does
indexDocument	event — triggered on document upload	Chunks and embeds one uploaded document; records a trigger_runs row; runs on connectorWriteQueue.
backfillDocumentCorpus	event — manual batch task, no cron	Re-embeds active documents onto a target embedding model in cursor-paginated batches (max 20 per batch); dryRun: true by default, requires confirmation: "REINDEX_ACTIVE_CORPUS" to write live.
reconcileDocumentIngestJobs	schedule, cron */15 * * * *	Reconciles stale ingest jobs via RPC, then batch-deletes superseded knowledge_chunks / spreadsheet_rows rows within a 40-second budget, reporting stale_rows_remaining when the cadence — not the batch size — is falling behind.
documentStoragePurge	event — direct dispatch (fast path)	Claims and processes one document_storage_deletion_jobs row: deletes the Storage object, then finalizes the catalog purge. Never calls Google Drive, Resend, or any other source provider.
documentStoragePurgeSweep	schedule, cron */10 * * * *	Durability backstop: re-invokes documentStoragePurge (triggerAndWait) to catch anything the direct-dispatch fast path missed.
feedbackAttachmentRetention	schedule, cron 15 3 * * *	Deletes expired feedback_attachments past retention_expires_at, batched at 100 rows per run.

What can go wrong / how to investigate

• Stale ingest rows or superseded chunks keep piling up. Check stale_rows_remaining on the most recent reconcileDocumentIngestJobs run — true means the 40-second budget ran out before the sweep finished, and the cadence (not the batch size) needs attention. This is exactly the shape of the 2026-09-08 incident: a single unbounded DELETE would time out against PostgREST's statement_timeout and delete nothing, forever, until it was replaced with a bounded, budgeted loop.
• A document purge request seems stuck. Look at document_storage_deletion_jobs.status — "blocked" means either claim_document_storage_deletion_job or complete_document_purge found a new dependency or a final check that hadn't cleared; it is not a failure, it is a deliberate hold. Check document_purge_receipts for the event trail (purge_requested, storage_delete_started, storage_deleted, catalog_deleted, purge_failed, reimport_allowed).
• A corpus backfill did nothing even though it "ran." Confirm dryRun was actually set to false and confirmation was exactly "REINDEX_ACTIVE_CORPUS" — anything else is treated as a preview and never writes.
• Feedback attachments aren't being cleaned up. The task throws (and Trigger retries, up to 5 attempts) if any single attachment fails to delete, so a run that keeps failing usually means a persistent Storage error on one attachment, not a systemic retention-policy problem — check the failed count and the logged attachmentId in the run output.
• A purge deleted a document SABLE didn't mean to touch. This should be structurally impossible: validDocumentStorageTarget refuses to purge anything outside a fixed documents-bucket path scoped to the job's own organization. If a purge target looks wrong, treat it as a data-integrity bug in the tombstone/job row, not a task-logic bug.

Where the code lives

• sable-agents-demo/trigger/documents.ts — indexing, corpus backfill, stale ingest-job reconciliation.
• sable-agents-demo/trigger/document-purge.ts — the purge workflow and its sweep.
• sable-agents-demo/trigger/feedback.ts — feedback attachment retention.
• sable-agents-demo/trigger/lib/document-runtime-deps.ts, sable-agents-demo/trigger/lib/document-storage-target.ts — shared runtime dependencies and the purge-target validator.
• sable-agents-demo/web/api/_document_ingest.js — indexUploadedDocument, the shared ingestion implementation both indexDocument and backfillDocumentCorpus call into.
• Supabase: documents, knowledge_chunks, spreadsheet_rows, ingest_jobs hold the Library's live state; document_purge_tombstones, document_purge_receipts, and document_storage_deletion_jobs (migration 20260805190955_document_purge_workflow.sql) back the purge workflow; feedback_attachments (migration 20260722173831_feedback_system.sql) backs attachment retention.