# Runbook: <flow_id>

- Flow: <one-sentence decision from the spec>, tier <1 | 2 | 3>
- Owners: business <role>, technical <role>, on-call <rota and how to page it>
- Spec: <path>. Field map: <path>. Last reviewed: <YYYY-MM-DD>

## Signals

- Oldest unprocessed item older than <minutes>: the worker, queue or destination is stuck. Start at Contain.
- Exceptions above <threshold>: identity, validation or permission failures. Start at Scope.
- Volume under 50% of the median of the same weekday over the previous 4 weeks: the subscription may be disabled, banned or deleted, or the source stopped. Check the sender's delivery log first.
- Volume over 200%: a backfill, a loop or a retry storm. Check attempts per operation and writes per record per hour.
- Quota above 80% of the daily allowance: find the consumer before other integrations starve.
- Credential expiring within 14 days or revoked: rotate through the secret store; never paste a secret into a ticket or chat.
- Read-back drift above <threshold>: wrong values reached the destination. Start at Contain.

## Contain

- Pause the writer that causes damage: <how, for example scale the worker to zero, turn off the scenario, disable the sync>.
- Keep receiving and storing inbound events while the writer is paused. Do not disable the sender's subscription unless it is the cause.
- Hold downstream automations that act on the affected fields: <workflows, sequences, journeys>.
- Record the time of each step.

## Scope

- Window: from the last trustworthy checkpoint <how to find it> to now.
- Affected records: <query or export that lists keys in the window, IDs and the flow's fields only>.
- Affected actions: messages sent, tasks created, enrolments, conversions uploaded in the window <where to look>.

## Reconcile

```bash
python3 -I "$SKILL_DIR/scripts/reconcile_readback.py" --source expected.csv --dest destination.csv \
  --key <source_key>[:<dest_key>] --map field-map.csv --flow <flow_id> \
  --baseline before.csv --dest-id <record_id_column>
```

- Run it from the incident's working folder, with `SKILL_DIR` set to the martech-integration-engineering skill folder. Exports hold stable IDs and the flow's fields; add `--hide-values` when a baseline column holds personal data.
- Pass: exit code 0 with every owned field listed under Owned fields compared, or every FAIL explained and accepted by the business owner; drift under the tier threshold.

## Repair

- Wrong match (DUPLICATE, KEY_MANGLED): fix the key or crosswalk first; never merge records automatically.
- Stale value (MISMATCH with an older value): check the version rule and per-account serialisation.
- Blank value (NULL_OVERWRITE): find the failed join or lookup; restore from the source, not from memory.
- Field changed by another writer (NON_OWNED_CHANGED): find the writer in field history and apply the ownership rule.

## Replay

- Who may replay: <role>. Approval from: <role> for tier 1.
- Recovery source and window: <for example webhooks journal 3 days, resend 15 or 30 days, search windows under 9,000 results>.
- Size first:

```bash
python3 -I "$SKILL_DIR/scripts/size_sync.py" --records <n> --batch <size> --preset <name> --remaining <from headers> --other-calls <n>
```

- Holds on downstream automations stay on until the read-back passes.
- Batch size <n>; stable keys only; reconcile after each batch.
- Stop when: errors above 5% of a batch, any wrong-customer write, quota above 80%, or a removal above 10% of the population or 1,000 records.

## Verify

- Read-back on a sample of at least 50 records (or all if fewer) passes.
- Downstream checks: <no duplicate tasks or messages, enrolment counts normal>.
- Release the holds, then watch one full cycle.

## Close

- Incident note: cause, window, records affected, actions repeated or suppressed, replay batches and their results, follow-up change requests, owner sign-off.

## Never do

- Return 2xx to silence a sender's retries or avoid a ban; failed work becomes invisible and unrecoverable.
- Run a full resync without knowing what it re-fires (inserts, events, enrolments).
- Clear and fill a shared list or audience without an inventory of every tool that writes members to it.
- Re-enable a disabled subscription before the receiver is fixed.
- Replay without re-checking eligibility (consent, suppression, still a customer) at the action.
- Paste credentials, tokens or customer records into tickets, chat or alerts.