Search and indexing

SABLE search is built around owned source records, scoped chunks, and explicit retrieval events. The browser never receives raw document bodies, embeddings, storage paths, service-role keys, or provider secrets.

Indexed Sources

Source	Ingestion path	Indexed representation
Manual upload	/api/doc-upload-session then /api/ingest-doc	documents, Supabase Storage, knowledge_chunks, optional spreadsheet_rows
Google Drive	drive-sync Edge Function	owned Storage blob, documents, knowledge_chunks, optional spreadsheet_rows
Fathom meetings	fathom-sync / fathom-webhook	sources.body, meeting spine rows, transcript knowledge_chunks
Contract/SOW files	upload or Drive	restricted documents, chunks, contract_obligations, budget_lines

Supported Extraction

The current extractor handles:

• PDF text layers, with Claude vision OCR fallback for scanned PDFs.
• DOCX via raw Word text extraction.
• PPTX via OOXML slide text extraction.
• XLSX, XLSM, CSV, and TSV through structured row/table extraction, with OOXML text fallback for workbooks the structured parser cannot load.
• PNG, JPG/JPEG, GIF, and WebP through Claude vision OCR.
• ZIP bundles with bounded recursive extraction for supported document, spreadsheet, deck, image, archive, and text entries.
• Outlook .msg messages as searchable email transcripts.
• Text-like files: TXT, Markdown, HTML, JSON, CSV, and TSV.

Legacy .xls, .xlsb, encrypted files, oversized files, videos, audio, and arbitrary binaries are marked unsupported rather than silently indexed.

Retrieval Flow

1. Resolve the verified viewer from the Supabase JWT.
2. Build the allowed project list server-side.
3. Embed the query with Voyage voyage-4.
4. Call match_knowledge_chunks_scoped for hybrid vector and keyword candidates.
5. Rerank with Voyage rerank-2.5 when available.
6. Balance top results so one source cannot monopolize the answer.
7. Expand each match with adjacent chunks for local context.
8. Return citation-ready evidence to /api/chat or raw search results to /api/search.

Keyword candidates come in two tiers. Chunks containing every query word rank first, exactly as websearch_to_tsquery matches them. After them come chunks containing the query's distinctive words, scored by summed rarity (knowledge_rare_term_matches). A word is distinctive when at most 0.5% of the chunks in the viewer's scope contain it, never more than 100 and never fewer than 3, so the bar is relative to how much the viewer can see. This keeps an acronym such as "EDL" retrievable when the query also names a client the evidence never mentions. Queries made only of common words, single-word queries, queries with more than 16 distinct words, and queries with an exclusion (-word) get no second tier.

/api/chat degrades to structured-spine answers when embeddings are unavailable. /api/search fails closed because it is a search endpoint and requires embeddings.

Access Rules

• can_see_project(project_id) gates document, chunk, and spreadsheet-row reads.
• Restricted documents and chunks are admin/lead only.
• Non-leads cannot use as=all or impersonate another person.
• /api/doc enforces the same project and restricted gates before minting a signed URL.

Operational Signals

Use these fields and tables to diagnose search quality:

• documents.indexed_status, indexed_at, and index_error.
• documents.checksum_sha256, byte_size, source_modified_at, and metadata.
• knowledge_chunks.embedding_model, chunk_index, content_hash, indexed_at, and restricted.
• spreadsheet_rows.sheet_name, table_name, cell_range, display_columns, and formulas.
• retrieval_events.query, filters, candidate_count, reranked_count, latency_ms, and result_chunk_ids.
• rag_eval_questions and rag_eval_runs for lead/admin recall, citation, leakage, and latency checks.

Quality Bar

Before treating a customer workspace as search-ready:

1. Upload or sync at least one representative file per important source type.
2. Confirm indexed_status = indexed and non-zero knowledge_chunks for each project.
3. Ask one known-answer question per project through /api/search or Ask SABLE.
4. Confirm citations name the expected file, page/time anchor, or workbook row context.
5. Confirm a non-member cannot retrieve the same document or search result.
6. Review retrieval_events for candidate counts, rerank counts, and latency outliers.