Skip to main content

Documents

The Documents surface is backed by Supabase Storage, the documents catalog table, the knowledge_chunks retrieval index, and spreadsheet_rows for workbook row/range extraction.

Documents support three workflows:

  • In-app document viewing/downloading through short-lived signed URLs.
  • Server-issued upload sessions that allocate a document row before the browser uploads.
  • Ask SABLE retrieval over scoped, chunk-level evidence.

Retrieval Route​

GET /api/doc?id=<documentId>

The route:

  1. Verifies the Supabase user JWT.
  2. Loads the document row.
  3. Blocks restricted documents for non-admin/non-lead users.
  4. Checks project membership for non-leads.
  5. Mints a 10-minute signed URL from the documents bucket.

Upload + Index Flow​

POST /api/doc-upload-session
POST /api/ingest-doc

The browser first asks for an upload session. The server validates project access, creates the document row, chooses a Storage path under documents/<org>/<project>/..., and returns a signed upload token.

For PDFs, DOCX, PPTX, images, ZIP bundles, Outlook messages, and text-like files, /api/ingest-doc extracts text, chunks the document, embeds chunks with Voyage voyage-4, and writes knowledge_chunks. Scanned PDFs and images use Claude vision OCR when text extraction cannot read a real text layer.

For XLSX/XLSM/CSV/TSV files, ingestion treats the file as semi-structured data. It extracts workbook, sheet, table/range, row, column, formula, and display-value metadata into spreadsheet_rows, then creates row/table/sheet/workbook discovery chunks in knowledge_chunks. If a workbook cannot be loaded by the structured parser but still has readable OOXML strings, SABLE indexes that text fallback instead of marking the file unsupported.

Large spreadsheets​

Product catalogs and rate tables can run to tens of thousands of rows (the ceiling is SABLE_SPREADSHEET_MAX_ROWS, default 100,000). Since 19 Oct 2026:

  • Every row is still kept. spreadsheet_rows holds one record per row whatever the size, so an exact lookup (a SKU, a rate code) finds its own row.
  • Big tables are searched in groups. A table with more than 2,000 data rows (SABLE_SPREADSHEET_GROUP_THRESHOLD, optional) gets one search chunk per ~25 rows (SABLE_SPREADSHEET_GROUP_ROWS) instead of one per row. Each group repeats the column headers and names its sheet, table and row range, so citations read "Catalog rows 2-26". Smaller tables are unchanged: one chunk per row.
  • Database writes retry. A batch that hits the database statement timeout waits, halves and tries again before the index fails; activation has its own longer timeout (migration 20261019120000_large_spreadsheet_activation.sql).
  • Progress shows. While a sheet of 1,000+ rows indexes, the Library and the Drafts inventory read documents.metadata.index_progress and say "Indexing 12,000 of 40,000 rows · step 1 of 3". If it fails, the error names the step: extract, embed, write rows, write chunks or activate.
  • Every spreadsheet goes through the background task. Uploads, reindexes and email attachments all index in sable.document.index (Trigger.dev, five retries). Email attachments are queued by the email function and picked up within five minutes by sable.document.index-queued.

Restricted Documents​

Restricted documents, such as contracts and SOWs, are admin/lead only. The route enforces this before issuing a signed URL.

Contract Extraction​

When an uploaded or Drive-synced contracts_sow document is indexed, SABLE extracts structured contract_obligations and SOW budget_lines from the text when possible. These stay admin/lead-only and feed Draft Workbench guardrails, update generation, scope-change review, and outcome estimates.

Admin/lead users can review extraction health through /api/document-inventory.

RAG​

Ask SABLE uses Voyage embeddings and the match_knowledge_chunks_scoped RPC when VOYAGE_API_KEY is configured. Retrieval combines scoped vector and keyword candidates, reranks with Voyage rerank-2.5, balances by source, expands adjacent chunks, and cites chunk-level evidence.

See Search and Indexing for the full extraction and retrieval contract.