Incident Response

SABLE has one automated alerting path today — the report watchdog — plus a set of manual checks this page collects so an incident does not start from a blank page.

The watchdog: what it actually catches

sable.report.watchdog.morning (cron 45 6 * * 1-5, America/New_York) and sable.report.watchdog.eod (cron 15 17 * * 1-5) each ask one question: was a morning/eod board delivered for today, per the board_runs table? Both run after the corresponding report's own scheduled time, so a healthy report has already had its chance to land.

The watchdog exists specifically because of the July 15–21 outage: for six days, boards silently failed to deliver and nobody noticed until someone asked why the board hadn't arrived. Both alert messages reference that outage by name so a responder immediately knows this is the failure class the watchdog was built to prevent.

Before touching the database, the watchdog self-tests its own alert channel: if RESEND_API_KEY or SABLE_OPS_ALERT_EMAIL is not set in the Trigger environment, it throws immediately and fails the scheduled run red — deliberately, so a broken alert channel is loud on day one instead of silently "passing" every day until the one day it needs to fire.

If the board_runs check itself throws (for example, Supabase is unreachable), the watchdog still tries to send an alert saying delivery status is unknown, not confirmed-missing — this is the exact shape of a Supabase/connectivity failure, so it is called out explicitly in the alert text as possibly the same class of failure as July 15–21. The original error is always rethrown afterward so the Trigger run fails red regardless of whether the alert itself succeeded.

Reading a watchdog alert

Alert	Meaning	First checks
"watchdog could not verify {kind} board"	The board_runs lookup itself failed before a verdict was reached.	Supabase reachability; SABLE_ORG_SLUG correctness if more than one org exists.
"{kind} board NOT delivered for {date}"	A board_runs row check ran cleanly and found no delivered row for today.	Trigger.dev runs for sable.report.{kind}; the trigger_runs table; REPORT_REQUIRED_CONNECTORS.

The verdict logic (watchdogVerdict) distinguishes "board row exists but not delivered" (a specific delivery_status is reported) from "no board row exists at all for today" — the alert text carries whichever is true, so don't assume a report never ran just because it wasn't delivered.

Common failure classes and where to look

Symptom	Likely cause	Where to look
Board not delivered	Required connector evidence stale or missing	REPORT_REQUIRED_CONNECTORS value on both Trigger and the Supabase spine-mcp Edge environment — a mismatch blocks the workflow before generation even starts. See Board Reliability And Cutover.
No email received at all	Sender/domain misconfiguration	All four send paths read RESEND_FROM/SABLE_ALERT_FROM rather than a hardcoded sender; confirm the configured sender is a verified Resend domain.
Connector data looks stale	Scheduler mismatch or drained-out sync	Check cron.job_run_details (legacy pg_cron jobs) and Trigger.dev run history for the connector's task; confirm only one scheduler is primary for that connector.
A whole account or Resend/HubSpot integration goes dark	Third-party account access lapsed, not a code bug	Confirm with the account owner before debugging code — this has happened before (Resend account access blocked every SABLE email for a stretch with DNS entirely fine).
Fathom meetings look under-attributed	Fathom is not returning attendees for a meeting	This is a known upstream gap, not a SABLE bug; attribution falls back to label/domain matching, which is why signal and evidence rules matter.

Escalation

• Code, connector, or database questions — the SABLE operator or the on-call developer; start with npm run doctor and the Supabase MCP server for read-only inspection before changing anything.
• Third-party account access (Resend, HubSpot, Google Workspace, Granola) — the responsible account owner, tracked outside this site in the repo's indigo-cc-setup skill; these are not code fixes.
• Prompt behavior — SABLE_PROMPT_OWNER (Lindsey Cutts and Kirk Orrick); prompt changes go through authoring-agents, not an ad hoc edit.
• Anything outbound or client-facing — stop and get the responsible application owner's sign-off before sending or publishing anything, incident or not.

After the incident

1. Confirm the watchdog itself reports green before considering the incident closed — a manually fixed board does not retroactively clear an unresolved alert condition unless the underlying board_runs row now shows delivered = true.
2. Write down what actually happened and why, the way docs/2026-09-08-where-we-are.md records verified facts "so they don't have to be re-derived" — a Supabase/GitNexus MCP question answered the slow way once is a lesson only if it's written down.
3. If the root cause is a gap the watchdog didn't catch, that is itself a finding: the watchdog only checks board delivery today, not connector health, chat availability, or approval execution.

Where the code lives

• sable-agents-demo/trigger/watchdog.ts — both scheduled checks, the alert self-test, and watchdogVerdict.
• sable-agents-demo/docs-site/docs/operations/board-reliability.md — the report pipeline the watchdog checks the output of.
• sable-agents-demo/docs-site/docs/operations/troubleshooting.md — symptom-specific fixes for connectors, chat citations, and the docs build.
• sable-agents-demo/docs/2026-09-08-where-we-are.md — verified incident history and open items, written for whoever picks the work up next.