Skip to main content

Watchdogs

Overview​

trigger/watchdog.ts is SABLE's independent backstop for board delivery: it does not trust the report pipeline's own success signal (reports.md), it verifies delivery directly against the board_runs table. trigger/smoke.ts's sableSmoke is grouped here as the closest fit among the four pages, even though — as noted below — it is not actually a scheduled health check.

How it works​

checkKind(kind) ("morning" or "eod") runs in three steps:

  1. Self-test the alert channel first, before any database work. If RESEND_API_KEY or SABLE_OPS_ALERT_EMAIL isn't set, the run throws immediately. This is deliberate: a watchdog whose alert path is broken otherwise "passes" every single day right up until the one day it actually needs to alert — and only then discovers the alert channel was never configured.
  2. Resolve the org and query board_runs for rows where as_of is today (America/New_York) and org_id matches, wrapped in withAlertOnCheckFailure: if the query itself throws (e.g. Supabase is unreachable), the watchdog still fires a best-effort ops alert saying delivery status is unknown, not confirmed-missing, and then always rethrows so the Trigger run fails red. The code comments call out that this is exactly the failure mode behind a July 15–21 outage, where the check itself failing went silent before this guard existed.
  3. Apply watchdogVerdict. A delivered: true row for today and the requested kind is ok. A row that exists but isn't delivered reports the delivery_status. No row at all reports "no delivered <kind> board for <date>". Any non-ok verdict triggers a Resend alert naming the reason and pointing at Trigger.dev runs, trigger_runs, and REPORT_REQUIRED_CONNECTORS as starting points.

sendOpsAlert posts directly to the Resend API (POST https://api.resend.com/emails) — it is a separate call path from the best-effort alerting reports.md's report pipeline uses, and that separation is intentional: the watchdog's own ability to alert must not depend on anything in the report pipeline having worked correctly.

sableSmoke is unrelated to any of this. It opens a trigger_runs row, records which two env vars it checked (SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY), and marks itself complete. It has no cron, no alerting, and no verdict logic — it exists to be triggered by hand as a basic "is this environment wired up at all" connectivity check.

Tasks​

TaskTrigger / cronWhat it does
morningWatchdogschedule, cron pattern 45 6 * * 1-5 (weekdays only), timezone America/New_YorkVerifies a delivered "morning" board_runs row exists for today; alerts via Resend if not. Runs 45 minutes after morningBoard's own 0 6 * * 1-5 cron.
eodWatchdogschedule, cron pattern 15 17 * * 1-5 (weekdays only), timezone America/New_YorkSame check for "eod". Runs 45 minutes after eodPacket's own 30 16 * * 1-5 cron.
sableSmoke (smoke.ts)event — manual/on-demand, no cronRecords a trigger_runs row and reports which env vars it checked (SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY) — a manual connectivity smoke test, not a scheduled or alerting health check. See the note below on why it's placed on this page.

Note on smoke.ts​

sableSmoke doesn't cleanly belong on any of the four pages: it has no cron, doesn't sync a connector, doesn't touch a document or a board, and carries no alerting. It's placed here because, conceptually, it's the closest thing to a health check in the codebase — a task you run to confirm the Trigger environment itself is wired up — even though it shares none of the actual watchdog machinery (no board_runs check, no Resend alert, no verdict). Treat it as a manual connectivity probe, not part of the automated watchdog schedule.

What can go wrong / how to investigate​

  • A watchdog run fails immediately with "watchdog alert channel not configured." RESEND_API_KEY or SABLE_OPS_ALERT_EMAIL is missing from the Trigger environment. This is a deliberate fail-red self-test, not a bug — fix the environment, don't silence the check.
  • A watchdog run fails with "watchdog could not verify <kind> board." The board_runs query itself threw (likely a Supabase connectivity issue). withAlertOnCheckFailure should have already sent a best-effort "delivery status is UNKNOWN" alert before rethrowing — if no alert arrived, check whether the alert channel is also down, since a broken alert path there is swallowed on purpose so it never masks the original error.
  • A watchdog fires "board NOT delivered" but you believe it was sent. Check board_runs directly for as_of = <today> and the relevant kind: either no row exists (the report pipeline never got far enough to persist one, or the wrong org_id was resolved) or a row exists with delivered = false (persisted but the delivery task itself failed — see reports.md's notes on deliverPersistedReport and taskRunErrorMessage).
  • sableSmoke "succeeds" but something is actually broken. It only confirms SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY are present and that a trigger_runs row can be written — it does not call any Supabase edge function, external connector, or the report/board pipeline. A green sableSmoke run is not evidence any of those are healthy.

Where the code lives​

  • trigger/watchdog.ts — morningWatchdog, eodWatchdog, checkKind, watchdogVerdict, withAlertOnCheckFailure, sendOpsAlert, resolveWatchdogOrgId.
  • trigger/smoke.ts — sableSmoke.
  • trigger/lib/supabase.ts — serviceClient, createTriggerRun / completeTriggerRun / failTriggerRun.
  • Supabase: board_runs (migration 0027_board_runs.sql) is the table the watchdog checks directly; trigger_runs (migration 0041_trigger_runs.sql) is where sableSmoke records its run; orgs is used by resolveWatchdogOrgId to scope the board_runs query.