# Rollback and incidents

How to revert each kind of change, what cannot be reverted, and what to do when an agent changed something it should not have. Mechanics checked 2026-10-17 and re-checked 2026-10-20; windows change and are often configured per table or dataset, so read the actual setting before promising a recovery.

## Write the rollback in step 3, before approval

For each proposal row, record:

- The action: re-apply before values, revert a workspace edit, republish a version, restore from a recycle bin, restore a table from a clone, snapshot or time travel, cancel a send.
- Who can run it without the agent (a person with access in the vendor UI).
- The window: until when it works, from the actual setting, not the product default.
- What it does not undo: messages already delivered, spend already served, people who saw a page, data already synced to another system.

## Mechanics by system

- Google Tag Manager:
  - Before publish: `gtm_tag` `revert` reverts a tag's changes in the workspace (pass the current fingerprint). Or remove the proposal's own workspace.
  - After publish: publish the previous version (Versions, select it, Actions, Publish) or use Set as Latest Version to make it the draft first. Record the live version ID before publishing. Stape's `gtm_version` tool exposes `publish` and `setLatest`; with an Edit-only account, a person does this in the UI.
- Meta Ads: no undo. Re-apply before values (budget, bid, status, targeting) from the change log, in the API's unit. Pausing stops future delivery, not spend already served. Deleted custom audiences, pixel event rules and feeds must be rebuilt; deleting a feed also removes the products it ingested. Delivery already split or held out by an A/B test or lift study cannot be undone.
- Google Ads: re-apply before values, usually as a Google Ads Editor import built from the before snapshot. Confirm with `change_event` (past 30 days).
- HubSpot: re-apply before property values. For pages, the MCP page tools can restore a revision (with confirmation); check the editor's unpublish option at run time. Workflow enrolments triggered by a change do not reverse when the property is set back; check and unenrol separately.
- Salesforce: deleted records go to the Recycle Bin and can be restored in the UI for up to 15 days. There is no undelete tool in the hosted MCP servers.
- Braze: re-apply before content from the change log; the log is the reference because Braze's pages disagree on whether MCP edits appear in campaign edit history. `stop_campaign` stops future sends only; messages a launched campaign already delivered cannot be reversed. Deleted catalogs, fields and items are rebuilt from their source.
- Klaviyo: `cancel_campaign_send` can cancel or revert a send before delivery. Sends that went out, profile merges and deletion requests are permanent.
- Mixpanel: re-apply before Lexicon values (descriptions, visibility, hidden status, PII classification), up to 50 per bulk call. Deleted dashboards, cohorts and tags must be rebuilt.
- PostHog: disable or roll back the feature flag, end the experiment, stop the survey, disable the workflow. Users who already got a variant, survey, broadcast or announcement are not reversed. Person and recording deletes, deleted synced table data and opt-out changes already acted on cannot be undone.
- Snowflake:
  - Read `kind` and `retention_time` from `SHOW TABLES` for every table in the proposal.
  - Permanent tables: 1 day by default, up to 90 days on Enterprise, then Fail-safe.
  - Transient and temporary tables: 1 day at most, no Fail-safe. dbt builds Snowflake tables as transient by default (check the current dbt-snowflake docs), so most marketing models have a 1-day window.
  - Before any DML or DDL row, take a zero-copy `CLONE` and name it in the rollback field. Restore by swapping back from the clone, or query or clone the table `AT` or `BEFORE` the change within its window.
  - Never run `CREATE OR REPLACE` on a table without a copy. It drops the old table; to recover it, rename the new table, then `UNDROP` the old one within its retention.
- BigQuery:
  - Read the time travel window (set per dataset or project, 2 to 7 days, 7 by default) before promising it.
  - Before any DML or DDL row, take a table snapshot or table copy and name it in the rollback field. Restore a standard table from the snapshot.
  - After the window, 7 days of fail-safe, recoverable only through Cloud Customer Care.
- Fivetran: no rollback through the MCP server is documented. A historical re-sync, schema config change or pause needs a manual revert plan from the Fivetran admin; a deleted connection has to be recreated and re-synced.

## Irreversible by nature

Treat these as irreversible even when a "cancel" or "restore" exists:

- Messages delivered (email, SMS, push, broadcasts, announcements), surveys shown, experiment variants served.
- Ad spend already served.
- Content published and seen.
- Profile and record merges.
- Deletes past their window, person deletes and privacy deletion requests.
- Consent and opt-out changes that a send already acted on.
- Personal data uploaded to an ad platform audience.
- Data sent to an external destination or webhook.

## When an agent made an unexpected change

1. Contain the live effect now. A person with access does this in the vendor UI, or the agent does it under a containment permission written in `martech-context/decisions.md`:
   - Meta or Google Ads: pause the entity.
   - Braze: `stop_campaign`.
   - Klaviyo: `cancel_campaign_send` if not yet delivered.
   - PostHog: disable the flag, end the experiment, disable the workflow.
   - GTM: publish the previous version.
   - Record the time and who did it. Containment is not the revert; the revert still goes through a proposal in step 6.
2. Stop the write path. Remove the write-capable connection or switch it to read-only. For Braze, turn off MCP OAuth access or revoke MCP OAuth tokens in Settings, Admin Settings, OAuth (company Admin only; revocation is company-wide and cannot be undone). Do not let the agent "fix" its own change before step 4.
3. Preserve evidence. Save the call log, the agent transcript, the change log and the vendor's own history (Meta activity log, Google Ads `change_event`, GTM versions, CRM property history, Braze security event report) before anything is reverted.
4. Read the current state of every touched object from the system, not from the agent's summary.
5. Revoke credentials if the identity may be misused, following the vendor's documented process.
6. Draft the revert as a normal proposal (before = current state, after = last known good) and get approval. Apply it with this skill's procedure.
7. Record the incident: what changed, when, through which tool, identity and account, why the controls did not stop it, what was contained, what was reverted, what could not be.
8. Add a test that reproduces the cause (for example, the synthetic injection test, the drift test or the wrong-account test) before write access returns.

## Signals to watch after a change

Each proposal names its signals with a numeric trigger, an owner and the response, stated by the approver, not invented by the agent. For example: "pause ad sets A and B if spend by 12:00 local exceeds 60% of the approved daily total; owner: paid media lead".

- Spend pacing against the approved daily total (Meta, Google Ads), and learning status after budget or bid edits (Meta).
- Tag firing counts and conversion volume against the previous comparable period (GTM).
- Workflow enrolment counts and list or segment sizes (HubSpot, Klaviyo, Braze).
- Report totals and event visibility after Lexicon edits (Mixpanel).
- Row counts and freshness of affected tables (Snowflake, BigQuery, Fivetran destinations).