# Monitoring, the stop control and the weekly review

Depth for procedure step 10. Template: [../assets/monitoring-runbook-template.md](../assets/monitoring-runbook-template.md). Every metric has a source, a threshold, an owner and a first action; a metric without an owner is decoration.

## Default alerts

Most map to a failure mode in SKILL.md. Defaults are starting values; propose overrides for `decisions.md`.

- Freshness: the oldest unprocessed signal is older than 2 schedule intervals. Usually a disabled source connection or a stuck run.
- Stuck pending: an enrichment request pending for more than 24 hours. Usually a lost webhook.
- Unknown outcomes: a create, enrollment or write with no confirmed result after 1 hour. Reconcile by key before any retry.
- Duplicates: any record created by the workflow that duplicates an existing one by email or domain. Threshold 1.
- Never-write changes: any change to a never-write property by the integration identity. Threshold 1.
- Fallback: any item past its deadline, or a fallback share above 10% of routed items.
- Response target: hit rate on the eligible cohort, with unanswered requests counted as misses, never dropped from the denominator.
- Cost: cost per accepted record above 1.25 times the approved model for 2 consecutive runs.
- Acceptance: acceptance rate down 20% relative to the median of the last 4 runs.
- Unsubscribe processing failures: above 0.
- Deliverability: the step 7 pause triggers (spam rate 0.1% on any day, batch hard bounces above 2%, a placement test in spam). Where Postmaster has no data or most recipients are business mailboxes, also opt-outs plus negative replies above 1% of delivered messages over 7 days.
- Mailbox load: any mailbox scheduled above its daily limit on any day, counting new enrollments and follow-ups.
- Webhook state: any sequencer or CRM webhook the workflow depends on is disabled. Threshold 1. Reconcile missed events from the API after re-enabling it.

## Where each number comes from

- HubSpot: property history filtered by source (integration, workflow, import), workflow history, owner change history, the duplicate management tool.
- Salesforce (checked 2026-11-06): field history tracking on the fields the workflow writes, up to 20 fields per object and kept up to 18 months, or 24 through the API, with Field Audit Trail off. It has no entries for changes made before tracking was on. Setup Audit Trail for configuration changes: the Setup page shows the 20 most recent and the download covers the past 180 days.
- Clay: the usage page Run view and per-table error columns; the connection state of each signal source.
- Apollo: the waterfall data request log; sequence and contact activity.
- Sequencers: campaign analytics and lead status counts per campaign and client; opt-outs, negative replies and hard bounces per sending domain and mailbox; the schedule per mailbox; webhook state.
- Google Postmaster Tools: spam rate per sending domain, for personal Gmail recipients only. "No data" is not a pass; record it as not checked.
- The workflow's own run ledger: one row per attempt and action, with run ID, key, outcome and destination ID. If the workflow has no ledger, build one before scaling; nothing else gives you unknown outcomes.

## Pairs that must reconcile

- Approved enrollments and destination leads created (same keys, same count).
- Confirmed sends and campaign events.
- Human replies and CRM tasks or reply records.
- Accepted enrichment results and CRM fields written.
- Each gap is either explained in the review or opened as an incident.

## Weekly review

- Agenda: rejection reasons and their counts, task acceptance, cost per accepted record against the model, failures and incidents, fallback items, the stop-control state.
- Rejection reasons, a small vocabulary reps can pick fast: wrong company, stale evidence, irrelevant change, existing conversation, offer unavailable, insufficient information.
- Rule after four weekly reviews:
  - Expand only with zero safety FAILs, task acceptance at or above the owner's target (default 50% of generated tasks) and cost within the model.
  - Revise when acceptance is below target and one rejection reason dominates; fix that reason first.
  - Stop when safety FAILs repeat, or acceptance stays below half the target.

## The stop control

One documented switch pauses new enrollments, CRM writes and paid enrichment. It is tested once before launch with a test record, and the test is dated in the runbook.

- Clay: turn table auto-run off or pause the workflow; set the MCP identity's credit limit to 0, which hard-blocks further actions. On Enterprise, lowering a workbook's spend limit below what it has spent stops its credit-consuming processes at once.
- Sequencer: pause the campaign. Pausing stops scheduled steps; it does not recall messages already sent.
- HubSpot: turn the workflow off. Records already enrolled behave according to the workflow's settings; check them.
- Apollo: pause the sequence and stop the API job or scheduled export that feeds it.
- What no switch undoes: messages already sent, credits already spent, records already created or changed.
- Resume never replays whole batches. Reconcile unknown outcomes first, then run a first batch of 10 records with read-back.

## Incidents

1. Contain: use the stop control.
2. Identify the run IDs and the records each touched, from the ledger.
3. Restore only changes attributable to those runs, from the before-values log, after checking no person changed the record since.
4. Compensate downstream effects: remove wrongly enrolled leads, suppress where needed, tell the owners of affected accounts.
5. Tell the reporting owner which numbers moved and for which dates.
6. Add a `decisions.md` entry: cause, fix, the rule or test added so it cannot recur.

## Optional evaluation

- A randomized account-level holdout can show whether a proactive play changes outcomes. Never hold out direct requests.
- With fewer than 30 outcomes per arm, call the result inconclusive and keep running.