# Operations: severity, tiers, alerts, runbooks, recovery and credentials

## Severity and evidence

- FAIL: the design or configuration can reach the wrong customer, cause an unintended action, lose or duplicate a business effect, write a field without authority, or leave no recovery path. Downgrading a FAIL requires citing a documented exception in `martech-context/decisions.md` with its owner.
- WARN: a plausible defect that needs a decision or a runtime check before it can be rated.
- PASS: checked and correct, stated with the reason or evidence.
- INFO: context that changes how a finding is read, for example "workflow is active in production".
- NOT CHECKED: not assessed, with the reason (no export, no access, out of scope).

Label every finding's evidence:

- config: read from an export, screenshot or settings page.
- runtime: observed in a sandbox or an approved test, with the record IDs.
- docs: stated by the vendor's live documentation, with URL and date.

## Tiers and their defaults

- Tier 1: the flow can contact a customer, change consent or suppression, move money or bidding values, grant or revoke access, or create sales work with a response target.
- Tier 2: the flow changes fields used in sales, success or marketing decisions, or in reports people act on.
- Tier 3: informational copies nothing automated reads.

Practitioner defaults to tune per team (record changes and reasons in the spec):

- Reconciliation cadence: tier 1 daily, tier 2 weekly, tier 3 monthly.
- Read-back drift that alerts: tier 1 above 0.5% of records or any money or consent mismatch; tier 2 above 2%; tier 3 above 5%.
- Exception alerts: tier 1 on the first exception; tier 2 when exceptions exceed 1% of a run or 50 a day; tier 3 in a weekly digest.
- On-call and runbook: both required for tier 1; a runbook required for tier 2.
- Pilot before backfill: tier 1 and 2 always; tier 3 when more than 10,000 records.

## Alert set (OP06)

- Oldest unprocessed item older than the freshness target.
- Exceptions by tier (above).
- Volume under 50% or over 200% of the median of the same weekday over the previous 4 weeks, only when that median is 20 or more per period. Low volumes alert on noise otherwise.
- Quota above 80% of the daily allowance.
- Webhook subscription disabled, banned or deleted. Check daily: Pipedrive deletes a webhook after 3 days without a successful delivery, Klaviyo disables a system webhook after 48 hours in error, and Make answers 410 Gone for webhooks unattached for more than 5 days.
- Credentials expiring within 14 days.
- Read-back drift above the tier threshold.
- 429s above 1% of calls in an hour (RL06).

## Alert content (OP07)

Each alert names:

- The flow and the business consequence ("renewal tasks may be missing for past-due accounts").
- The affected count and the first-seen time.
- The runbook step to start from and the permitted next action.
- Nothing else: no payloads, no personal data, no tokens. Route it to the named owner, not a broad channel.

## Exception queue

Fields:

- `key`, `flow_id`, `reason_code`, `first_failed_at`, `attempts`
- `last_error` (class and code only, no payload)
- `next_redrive_at`, `owner`

Review cadence: tier 1 daily, tier 2 weekly. Re-drive on a bounded schedule (15 minutes, 1 hour, 6 hours), then hand to the owner.

## Runbook steps

Write one per flow from [../assets/runbook-template.md](../assets/runbook-template.md):

- Contain: pause the writer causing damage; keep receiving and storing inbound events; hold downstream automations.
- Scope: last trustworthy checkpoint, failure window, affected subscriptions, objects and actions.
- Reconcile: `scripts/reconcile_readback.py` against the source of truth.
- Repair: fix the cause (key, version rule, failed join, rogue writer) before restoring values.
- Bounded replay: sized with `scripts/size_sync.py`, in batches with stable keys, with holds on and eligibility re-checked at the action.
- Verify: read-back passes, downstream counts normal, holds released, one full cycle watched.
- Close: incident note with cause, window, records affected, actions repeated or suppressed, and follow-up change requests.

Never do:

- Return 2xx to silence a sender.
- Run a full resync without knowing what it re-fires. Hightouch full resync processes current rows as added, so inserts and events can duplicate.
- Clear and fill a shared list or audience without an inventory of its writers. Hightouch clear and fill empties the list, including memberships other tools wrote.
- Re-enable a subscription before the receiver is fixed.

## Recovery sources and windows

- HubSpot: the webhooks journal returns subscription-related changes for the past three days (webhooks journal guide, checked 2026-11-02); beyond that, CRM search with overlapping windows under 9,000 results each, then a read-back (search excludes archived records, can lag, and returns a 400 past 10,000 results).
- Stripe: Dashboard resend up to 15 days after the event, CLI resend up to 30 days. A resend does not cancel automatic retries, so expect both. For current state, read the objects through the API instead of replaying history.
- Pipedrive: Recents with `since_timestamp` and deal history for changes missed during a ban or outage.
- Klaviyo: Get Events filtered by metric and time; re-enabling a disabled system webhook sends only new events.
- Segment: Replay from archived logs on the Business tier; destination retries last four hours.
- Hightouch: full resync (current rows processed as added), reset CDC (captures a baseline and sends nothing), clear and fill (empties the destination list). Choose by what each re-fires.
- Salesforce Bulk API 2.0: re-run only the failed and unprocessed result rows; results beyond the daily batch allocation are not recorded, so re-check those keys.
- Make: incomplete executions (only if storage was on; errors on the first module create none without a Retry handler).
- Zapier: replay within 60 days of the trigger; history replay skips Filter and Paths steps.
- n8n: failed executions can be retried from the executions list. Check in the n8n docs which node the retry starts from, and confirm every write it re-runs is idempotent.

## Triage by symptom

- No delivery attempt in the sender's log: subscription disabled, deleted or banned, wrong event types, wrong account. Check the subscription status page first.
- Authentication failures: rotated or expired secret, signature computed on parsed JSON, clock skew beyond the tolerance, wrong signature version.
- Sender keeps retrying although events are stored: acknowledgement after processing or a slow commit. Measure acknowledgement p99 against half the sender's timeout.
- Delivery fine but destination stale: worker stuck, exceptions parked, version rule dropping valid writes, quota pause. Check the oldest unprocessed item.
- Duplicates in the destination: delivery key used as the business key, read-then-create race, unknown outcome retried blindly, two Stripe Event objects processed as two effects. Check the ledger for the episode key.
- Reconciliation gaps: lookups failing silently into creates, a mangled key, a removal never sent, a second writer. Run the read-back with `--baseline`.

## Credentials and versions (OP08)

- Credentials belong to a service identity in a business-owned account, with least privilege and a rotation date. Never a person's login or a personal iPaaS account: it breaks when they leave, and its writes look like a human's in field history.
- Offboarding check: list every integration that authenticates as a departing person.
- Stripe MCP from December 2026 rejects full-access secret keys and restricted keys without the Agent tag; migrate agent access by 30 November 2026. This concerns agent access, not the runtime's own restricted key.
- Pin API versions: HubSpot dated paths (for example 2026-09), the Stripe account API version, the Salesforce API version.
- Keep a deprecation calendar. Current entries:
  - Google Ads UploadClickConversions has failed since 15 June 2026 for developer tokens that sent no offline conversion uploads between 17 December 2025 and 15 June 2026 (Data Manager API recommended). Google says developer tokens were sunset on 9 September 2026 in favour of Google Cloud projects and that this restriction stays in effect (checked 2026-11-06).
  - Google Customer Match: as of 1 April 2026 most new integrations must use the Data Manager API, and existing eligible integrations can stay on the Google Ads API (checked 2026-11-06).
  - The Salesforce local DX MCP server reached end of life on 2 November 2026: no further releases or support, repositories archived, still installable; Salesforce points developer tooling to forcedotcom/sf-skills (retirement notice, checked 2026-11-06).
  - Salesforce hosted MCP servers: agent registration will be required within three months of Salesforce's notice (article 005360285, read 2026-11-06).

## Rules checked by the linter

- OP01: `exception_queue` not yes is FAIL on tier 1, WARN on tier 2.
- OP02: tier 1 or 2 with `alert_oldest_age_minutes` blank is FAIL; above twice `freshness_target_minutes` is WARN.
- OP03 FAIL: `credential_owner` person.
- OP04 WARN: `api_version` blank for method custom or ipaas.
- OP05: tier 1 with `reconciliation_cadence` none or monthly is FAIL; tier 2 with none is WARN.
- CT03: tier 1 without `on_call` or `runbook` is FAIL; tier 2 without `runbook` is WARN.