Data Incident Runbook: 8 Steps to Recover Marketing and Product Data

A data incident runbook turns an alert into a controlled repair: confirm the affected decision, assign a lead, contain downstream actions, locate the first failing layer, repair the cause, reconcile history, notify consumers and close the prevention work. Restore collection and restore trustworthy reporting as separate milestones. A healthy pipeline does not prove that earlier reports or external destinations have been corrected.

Jafrin Akter AloLast updated 25 October 202614 min read

On this page

A missing purchase tag, a broken CRM association and a duplicate audience sync can all appear as “bad data.” Their consequences differ. The first may distort budget decisions; the second can send work to the wrong owner; the third may contact customers twice. The response should follow those consequences.

Use this runbook with the marketing data quality operating model. It is designed for a small team where the same engineer may investigate a failure and help the business decide when to trust the output again.

1. When should a data problem become an incident?

Declare an incident when the problem threatens a customer action, external destination or consequential decision, or when resolving it requires coordinated work. A critical report can be wrong even while every system remains available.

Google SRE recommends declaring early when another team is needed, customers can see the outage or concentrated investigation has not resolved it after an hour. Those are useful escalation prompts, not reasons to wait an hour before stopping harmful activity. Google incident guidance.

First establish a known discrepancy. Record an expected order, customer or account, where it should appear, where it actually appears and the relevant cutoff. Confirm that the report uses the intended filter, timezone and data version. A valid difference between gross bookings and collected revenue should become a definition issue rather than a fictional pipeline outage.

Use severity definitions your team accepts in advance. This is an illustrative four-level template, not a vendor service-level commitment:

If the impact is uncertain, initially use the more serious plausible severity and investigate. PagerDuty applies this higher-severity rule when classification is unclear. Severity guidance. Reduce severity when evidence supports it, recording why.

A possible privacy or security exposure needs the organization's corresponding response process as well. A data-quality ticket should not become the only record of a suspected disclosure.

2. Who should lead, repair and communicate?

Name one incident lead, one technical repair owner and one communication owner. A small team can combine roles, but each responsibility still needs an explicit name and backup.

Google's incident model separates coordination, operational repair and communication, with a living incident document and an explicit handoff. Incident roles and handoff. Adapt that separation to your actual staffing: the engineer tracing SQL should not also have to answer a new status request every five minutes.

Keep the decision record at the top of one shared incident document:

• What is affected and what remains usable?
• Which periods, customers, accounts or destinations are involved?
• What is paused, and who may restart it?
• What evidence is confirmed, and what remains a hypothesis?
• Who owns the next action and the next update?

Choose an update interval that fits the impact and staffing. It is an internal operating promise, not an invented completion estimate. If the cause is still unknown, say what was checked and when the next evidence will be available.

At Martechdevs, we include monitoring, repair ownership and handover in the warehouse scope. The person taking over should be able to locate a failure and operate the pause point without asking the original builder to remember the architecture.

3. How do you contain the damage before fixing history?

Pause the affected consumer or action at the narrowest effective boundary, while preserving the evidence needed to diagnose it. A blanket full refresh can overwrite useful history, consume resources and resend actions before the cause is understood.

Pausing a sync prevents a future write only if that pause controls the actual sender. It does not recall a message or undo a field already written. Check queues, retries and parallel paths. The reverse ETL guide and Hightouch failure checks cover the destination-specific repair boundary.

Hightouch's Unexpected sync volume control applies to journey sync tiles in Trigger mode. At STOP, it evaluates the planned volume before sending and stops the run; muting that trigger permits subsequent runs and silences its notification. Treat that action as an operational decision, rather than a harmless alert cleanup. Hightouch sync alerts.

For conversion data used in advertising, involve the account owner before changing a live feed. The right containment depends on whether the defect is wrong value, duplicate occurrence, missing coverage or late delivery. Record the chosen action and its effect on bidding inputs. Do not manufacture a zero-conversion period merely to make an upload process look clean.

4. How do you find the first failing layer?

Trace the same known business occurrence through source, collection, ingestion, transformation and destination. Stop at the first point where the expected identity, value or state changes.

Check changes near the first known failure: application releases, tag-manager versions, consent settings, CRM property edits, connector upgrades and API sunsets. Temporal proximity is a lead to investigate, not proof of cause.

Fivetran distinguishes errors that prevent syncing from warnings that allow syncing to continue. A warning therefore needs its own impact assessment; continued movement does not establish complete or correct data. Fivetran alerts.

Reproduce the smallest failing case in an approved environment. Keep source identity and time fixed while testing one change at a time. For an advertising destination, use the Google Ads conversion diagnosis to distinguish setup, acceptance and reporting conditions.

Avoid changing a report filter merely to restore yesterday's total. The goal is to explain the discrepancy, not force agreement between different definitions.

5. How do you repair the cause and backfill safely?

Repair and verify the producing path before replaying history. Define the affected keys and interval, preserve a before-state snapshot and separate rebuilding analytical history from executing operational actions.

Use a bounded dbt incremental backfill when the source records still exist. Run it against the same business rules that will serve future data, or document a versioned exception. Compare the candidate result before replacing the accepted report. For PostgreSQL CDC, establish slot continuity and a supported replacement baseline before assuming a replay can recover every missing change. When the bad change is in Snowflake, use historical evidence and a guarded current-state repair instead of replacing the table with an old snapshot.

A replay should carry a stable occurrence or operation key and a destination-specific policy. Rebuilding a historical customer state must not automatically enroll that customer in an expired onboarding message. Check current eligibility before reissuing an action, and keep repair-only transformations separate from normal activation.

Which recovery windows change the plan?

Identify each destination's actual recovery rules before promising to restore the original report. These limits are not interchangeable:

For GA4 events older than 72 hours, default or RELAXED validation can replace the timestamp with the seventy-two-hour boundary; ENFORCE_RECOMMENDATIONS rejects the event. Neither recreates its original historical position. Measurement Protocol timing. Follow the Measurement Protocol implementation guide for the required identity and context.

Google Ads documents the ninety-day and sixty-three-day limits relative to the associated last click. It deduplicates the same combination of unique identifier, conversion name, date and time. Preserve that identity when investigating a retry; changing the time to bypass a rejection can create a different event. Offline import rules.

Use the offline-conversion setup guide for the eligible path. If the underlying business outcome exists but cannot be restored in the platform with its original context, rebuild the business metric in the warehouse and explain that reporting boundary. Never invent browsing events, consent or attribution identifiers that were not captured.

Keep the late-arriving data policy with the incident so an allowed restatement is handled consistently across tools.

6. What must pass before reopening reports and syncs?

Reopen a consumer only when its relevant acceptance checks pass against the repaired data version and cutoff. A successful job is insufficient if the old error already reached another system.

Reconcile record identities, counts, amounts, currencies and important categories with the authoritative source. Inspect rejected and unresolved records separately. Then verify the destination's actual state, including records that should be removed or corrected.

During dbt build, failed upstream error tests can skip selected downstream resources. That behavior does not automatically pause a separate scheduler or activation tool. dbt build behavior. Connect the relevant dbt tests to the orchestration gate and require the expected successful run before releasing the consumer.

Use a small controlled cohort when resuming an external action. Compare intended operations with accepted operations and read back a sample. Check repeated delivery, a missing optional field and a previously processed record. Expand only when the differences are understood.

Record collection restored, historical reporting repaired and external state repaired as separate milestones. If one remains incomplete, the resolution note should make that explicit instead of declaring the entire incident finished.

7. What should you tell people who used the bad data?

Tell each affected consumer what is unreliable, what action to take now and when the next update will arrive. After repair, state the corrected period, magnitude and decisions worth revisiting.

Use these original message templates as fields to complete, not claims about a real incident:

First notice
Affected output: [report, model, audience or conversion action]
Affected scope: [accounts, events and time interval]
Confirmed problem: [observed discrepancy]
Use now: [known-good cutoff or approved workaround]
Paused action: [specific sender or decision]
Owner and next update: [name, time and timezone]

Progress update
Confirmed since the previous update: [evidence]
Current containment: [what remains paused]
Next action: [owner and test]
Expected next evidence: [time or dependency]
Unresolved: [specific uncertainty]

Resolution note
Corrected range and data version: [scope]
Before and after: [counts and amounts with units]
Source reconciliation: [result and exceptions]
Destination repair: [completed or remaining work]
Decisions to revisit: [budget, audience, revenue or forecast]
Follow-up control and owner: [test, person and due date]

Avoid a speculative repair estimate when the dependency is outside your control. Promise the next update you can deliver. Keep sensitive customer data in the approved evidence location and link to it with appropriate access, rather than copying payloads into every discussion.

8. How should the postmortem prevent a repeat?

Document the cause, why the failure escaped detection and the change that will prevent or expose it earlier. Assign an owner and acceptance test to each action so the postmortem does not become a list of intentions.

Google SRE includes data loss and monitoring failures among its postmortem triggers and emphasizes contributing conditions rather than individual blame. Postmortem guidance. Translate “someone forgot to test” into the missing mechanism: no required fixture, an unowned release gate or a notification nobody receives.

Record occurrence start, detection, containment, trustworthy-data restoration and closure separately. The interval from detection to a green connector is not necessarily time to restore the report. Keep reopened incidents visible rather than resetting their clock to make resolution time look shorter. Carry those incident clocks into a monthly data quality scorecard, keeping unknown start times and open incidents visible.

Add the failing record pattern to the acceptance suite, then rehearse it. If a category was silently mapped to other, add a schema and value-drift check. If no event arrived, compare an expected business population with delivered events. Another generic uptime alert would not catch either case.

How would this runbook handle a checkout tracking failure?

A useful repair restores the correct business history and describes any platform history that cannot be reconstructed. Consider a fictional ninety-six-hour incident affecting one checkout path, with all amounts in USD and no refunds in this simplified example.

The billing system records 400 completed orders worth $40,000. The GA4 report contains 250 orders worth $25,000. The 150-order, $15,000 gap represents 37.5% of the billing population. This comparison is a diagnostic starting point; normal consent exclusions and reporting definitions must be separated from the actual defect.

A representative order trace confirms that the checkout release removed the purchase event from one permitted path. The team records the following response:

1. The growth owner holds automated return-on-ad-spend decisions for the affected period and marks the report incomplete.
2. The engineer restores the producer, then checks a normal purchase, a retry, consent states and the destination receipt.
3. The warehouse report is rebuilt from authoritative order records, with identity and amount reconciliation against all 400 orders.
4. Of the 150 missing platform events, 100 are older than seventy-two hours and 50 are more recent. The older group cannot be restored to its original GA4 timestamp through Measurement Protocol. The recent group still needs valid identity, consent and context before any upload is considered.
5. Sixty missing orders have retained click evidence for a possible Google Ads import. The team checks the action's eligibility, applicable window and duplicate history before uploading any of them.
6. The resolution note separates the restored warehouse total from the remaining gap in GA4 history and identifies the affected budget decisions.

The sixty Ads candidates and fifty recent GA4 candidates may overlap; they are route-specific sets, not totals to add. A data repair should preserve these identities rather than infer eligibility from an aggregate count.

A detection control should compare eligible backend orders with received purchases and use the correct latency allowance. A traffic-based anomaly can help trigger investigation, but a count tied to the business outcome gives the engineer a more precise acceptance target.

Read diagram as text

• Confirm discrepancy and affected decision to Set severity and assign owners.
• Set severity and assign owners to Contain the affected consumer or sender.
• Contain the affected consumer or sender to Trace and repair the first failing layer.
• Trace and repair the first failing layer to Rebuild eligible history with replay controls.
• Rebuild eligible history with replay controls to Source and destination acceptance passes.
• Source and destination acceptance passes to Trace and repair the first failing layer: No.
• Source and destination acceptance passes to Reopen consumers and communicate correction: Yes.
• Reopen consumers and communicate correction to Close the postmortem and prevention test

Open full diagram

What should the incident record and closing checklist contain?

Keep one record linking the trigger, affected outputs, repair and acceptance. Reuse connector incident evidence where relevant, while retaining the business consequence and downstream repair here.

incident:
  reference: "example-checkout-purchase-gap"
  severity: high
  lead: "named incident lead"
  repair_owner: "named measurement engineer"
  consumer_owner: "named growth owner"
  affected_output: "acquisition revenue report"
  source_of_truth: "approved billing order ledger"
  occurrence_boundary: "record exact start, end and timezone"
  first_failing_layer: "checkout event producer"
  containment: "hold affected automated budget decisions"
  repair_scope: "identified order IDs and reporting interval"
  acceptance:
    - "new collection path passes journey fixtures"
    - "warehouse counts and amounts reconcile"
    - "each external replay has an eligibility decision"
    - "unrecoverable platform history is identified"
  reopen_owner: "named growth owner"
  follow_up: "test that detects this failure before release"

Bring the affected report and one known failing record to the $99 martech audit, or request a scoped data reliability repair when the failing boundary and required acceptance are clear.

Written by

Jafrin Akter Alo

Data Engineer and Co-founder at Martechdevs and Growth and Analytics Partners

Jafrin writes practical guides to marketing technology, customer data, analytics and AI workflows. Martechdevs is a Growth and Analytics Partners company.

View LinkedIn profile