# Monitoring runbook: - Owner of this runbook: . Last reviewed: - Run ledger location: ## Alerts One block per metric. Thresholds are the skill defaults until `decisions.md` records an override. - Freshness - Definition: age of the oldest unprocessed signal. Source: - Threshold: older than 2 schedule intervals. Owner: - First action: check the source connection state and the last run. Escalation: - Stuck pending enrichment - Definition: requests pending for more than 24 hours. Source: run ledger, Apollo waterfall log - First action: look for the missing webhook; never re-request blindly. Owner: - Unknown outcomes - Definition: create, enrollment or write with no confirmed result after 1 hour. Source: run ledger - First action: query the destination by key; retry only what is confirmed missing. Owner: - Duplicates created - Definition: records created by the integration identity that match an existing record by email or domain. Threshold: 1 - First action: stop control, then merge by the CRM's own tool after approval. Owner: - Never-write changes - Definition: changes by the integration identity to any never-write property in the contract. Threshold: 1. Source: property or field history - First action: stop control; restore from the before-values log. Owner: - Fallback queue - Definition: items past deadline, or fallback share of routed items. Threshold: any past deadline, or above 10% - First action: assign, then fix the missing rule or territory. Owner: - Response target - Definition: share of eligible requests with a first human attempt inside the target, unanswered counted as misses. Owner: - Cost per accepted record - Definition: blended variable cost per accepted record per run. Threshold: above 1.25 times the approved model for 2 consecutive runs - First action: run enrichment_cost.py on the run ledger with --billed. Owner: - Acceptance rate - Definition: accepted tasks / generated tasks per run. Threshold: down 20% against the median of the last 4 runs - First action: read rejection reasons for the run. Owner: - Unsubscribe processing failures. Threshold: above 0. First action: stop control for enrollments. Owner: - Deliverability - Definition: Postmaster Tools spam rate per sending domain (personal Gmail recipients only; record "no data" as not checked); hard bounces per batch; opt-outs plus negative replies over 7 days; placement tests to controlled Workspace and Microsoft 365 mailboxes - Threshold: spam rate 0.1% on any day; hard bounces above 2% of attempted sends; opt-outs plus negative replies above 1% of delivered where Postmaster has no data or most recipients are business mailboxes; any placement test in spam - First action: pause new enrollments for that domain; find the cause. Owner: - Mailbox load - Definition: scheduled sends per mailbox per day, new enrollments plus follow-ups. Source: sequencer schedule export. Threshold: above the mailbox's daily limit - First action: hold new enrollments for that mailbox and reschedule later cohorts. Owner: - Webhook state - Definition: whether each sequencer or CRM webhook the workflow depends on is enabled. Threshold: any disabled - First action: fix the receiver, re-enable the webhook, then reconcile missed events from the API. Owner: ## Stop control - What it pauses: new enrollments, CRM writes, paid enrichment - How, per tool: Clay ; sequencer ; HubSpot ; Apollo - Who can trigger it: - What it does not undo: messages already sent, credits already spent, records already created or changed - Last test: ## Resume - The cause is fixed and verified on test records. - Every unknown outcome is reconciled. - No whole-batch replay: only unresolved stages are re-run. - First batch of 10 records, read back and compared with the change list, before the normal schedule resumes.