Tracking plan audit

Reads one tracking plan and what production collects, and reports where they disagree as PASS, WARN or FAIL with numbers, then drafts the fixes for people to apply. The audit changes nothing, and a finding is a lead until a payload, an emitter or a system-of-record count confirms it.

When to use

Use this skill when:

• Someone asks to audit a tracking plan, an event taxonomy or analytics data, or to check tracking against the spec.
• The plan and the dashboards disagree and nobody trusts either.
• A cleanup, migration, renewal or Lexicon or Amplitude Data tidy-up needs a baseline first.
• After a release, counts doubled, an event vanished or a property changed type.
• An agency or a previous owner hands over a plan.
• GA4 and a product analytics tool, often fed by Segment, must agree on one plan.

Use something else when:

• No plan exists, only collected events: run tracking-plan-builder's retrofit path first, then audit its result.
• Only names are in question, a lint or a rename plan: event-naming-convention.
• One number broke and nobody knows which layer failed: tracking-break-map.
• Orders or revenue disagree with analytics order by order: purchase-reconciliation.
• The GTM container itself is suspect: gtm-container-audit, which also finds the tag behind a GA4 duplicate producer.
• A site-wide tag and consent audit: tracking-audit. One GA4 property's settings: ga4-audit.
• Events arrive when consent is denied, or vanish after a CMP change: consent-debugging.
• Users are split or wrongly merged across login: identity-merge-qa (planned).
• Amplitude-only deletion cleanup or Mixpanel Lexicon metadata: Amplitude's own tracking-plan-audit skill (a different skill with the same name, from Amplitude's MCP marketplace) and its taxonomy skill, or Mixpanel's manage-lexicon, connected read-only first.
• Ongoing alerting: this audit proposes monitors; tracking-monitoring-and-incidents is planned.

If a skill named here is not installed, stop at that decision and hand the user the evidence gathered so far.

Inputs

Read these first if they exist:

• martech-context/stack.md: tools, production projects, streams and sources, CDP delivery modes, warehouse, owners.
• martech-context/tracking-plan/ and its CHANGELOG.md: contracts, statuses, versions and dates.
• naming-rules.md, data-dictionary.md, metrics.md: the convention and kept exceptions, shared properties with units and enums, and which metric or workflow reads each event.
• decisions.md: accepted exceptions with expiry dates, where the canonical plan lives, the consent model.

Then gather:

• The canonical plan and any copies (sheet, YAML contracts, Avo, Segment Protocols, Amplitude Data, Lexicon), and which one is the source of truth.
• Per tool, the production project or stream, one read-only route (an export file, a query result, or an MCP server limited to read tools) and the time zone its counts use; for GA4, the property tier and the custom definitions.
• The window (default 28 days ending 4 days before the audit, once GA4's daily tables are final) and a 90-day lookback for last-seen dates.
• System-of-record counts for every tier 1 event, same window and exclusions, with their time zone, aggregates only.
• Emitter roots: repositories, GTM web and server containers, CDP sources and functions, GA4 event rules, tool-side transformations.
• Consumers per event (key events, saved reports, cohorts, workflows, Lexicon query counts), and owners and approvers by risk tier.

Never ask for passwords, API secrets, service account keys or raw customer records in chat. The user runs exports; the scripts read local files and print counts, types and redacted enum values, never raw records.

Procedure

Work in a scratch folder outside the skill folder and outside martech-context/. Set SKILL to the installed skill folder, run the scripts by full path and write every output to the scratch folder. Nothing under martech-context/ changes until step 8 is approved.

1. Fix the baseline

• Record the plan location, version and last approval date; tools with project placeholders; production only (staging excluded unless the plan scopes it); window, lookback and as-of date; time zone per tool and per reference; GA4 property tier; tier 1 events.
• If more than one copy of the plan exists, name the canonical one and audit the others as observed copies without counts. Copy drift explains enforcement blocks.
• If the project or stream you can read is not the one the user named, stop and say so.

2. Normalize the plan into two tabs

• Copy the header rows of assets/plan-events-template.csv and assets/plan-properties-template.csv into the scratch folder as plan-events.csv and plan-properties.csv, and fill those; never edit the assets or the source. tracking-plan-builder's sheet tabs work as they are; a browser companion in delivery_paths counts as a web source.
• YAML contracts: one events row per contract, one properties row per property, global properties as event *, companions as tool overrides in sources (server; ga4:web).
• Check row counts against the contract count and list fields that had no column. Rules per source format: references/plan-quality.md.

3. Score the plan as a contract

python3 "$SKILL/scripts/audit_plan.py" quality --events plan-events.csv --properties plan-properties.csv

• Rules Q01 to Q13 print coverage per rule. Never blend them into one score.
• Verdicts: contract; schema only (names and types, no triggers, sources or consumers, as every vendor schema export); names only (Q05 or Q08 fails). Q05, Q07 and Q08 set the drift depth.
• Names only: say in the report which rules could not run.
• Naming: run event-naming-convention's lint_event_names.py, when installed, over each tool's final names and, after step 5, the orphan names, and report its summary; otherwise mark naming NOT CHECKED.

4. Build the observed inventory

Normalized CSVs with the columns of assets/observed-inventory-template.csv, each passed with its own --observed. Read-only routes:

• GA4: BigQuery with assets/ga4-bigquery-inventory.sql, in the console or through Google's BigQuery remote MCP server with execute_sql_readonly only; or the Data API through Google's GA4 MCP server started without --allow-write, with the eventName, platform and date dimensions (names, sources, counts and dates only).
• Mixpanel: the Lexicon CSV with an Analyst-role account, MCP read tools behind the deny list in Boundaries, or a raw export through the profiler.
• Amplitude: get_amp_taxonomy with a role that grants only Use MCP (read), or an Export API file through the profiler.
• PostHog: the MCP server with readonly=true and assets/posthog-hogql-inventory.sql.
• Segment: Source Schema, violations and assets/segment-warehouse-inventory.sql.
• Run a warehouse query only after a dry run (dryRun true on execute_sql_readonly, or the console estimate); above the cap the user sets (ask for one), state the estimate and wait for approval, or hand the query over.

python3 "$SKILL/scripts/profile_export.py" export.zip --format amplitude --tool amplitude \
  --since 2026-10-03 --until 2026-10-30 --values-for connector_type,report_type \
  --identity-keys user_id > observed-amplitude.csv

Record row totals and caps (Lexicon CSV 5,000 events, GA4 daily batch export one million events on standard properties, Amplitude 2,000 event types). A names-only listing leaves rows empty, never 0. A vendor view shorter than the window never proves absence. Types, time zones and caveats per tool: references/observed-inventory.md.

5. Run the drift check

python3 "$SKILL/scripts/audit_plan.py" drift --events plan-events.csv --properties plan-properties.csv \
  --observed observed-amplitude.csv --observed observed-ga4.csv --reference reference-counts.csv \
  --ga4-definitions ga4-definitions.csv --window-start 2026-10-03 --window-end 2026-10-30 --out audit-out

• Rules D01 to D15 with thresholds and reasons: references/drift-rules.md. Defaults: type mismatch FAIL at 1%, required completeness FAIL below 95% (below 100% on tier 1 server events), enum drift FAIL above 1%, ratio to the system of record FAIL above 1.0, minimum sample 100; an event silent for about seven expected rows before the window ends has stopped.
• --reference takes a filled copy of assets/reference-counts-template.csv, time zones included; --ga4-definitions one of assets/ga4-custom-definitions-template.csv. Pass --ga4-param-limit 100 for Analytics 360. Exit code 1 means at least one FAIL, 2 bad input. --out writes findings, a per-event snapshot and a plan-changes draft, and refuses folders inside martech-context/ or the skill folder.
• If Python is unavailable, apply the rules by hand and label the report "script not run".

6. Review every finding with judgment

• Vendor events and parameters are never orphans: GA4 automatic events, $ defaults, [Amplitude] events, mobile lifecycle events, Segment's "Loaded a Page" and "Violation Generated". Copy assets/vendor-events.txt and assets/vendor-params.txt to the scratch folder, refresh the copies from the live docs and pass them with --vendor-events and --vendor-params.
• Root-cause locator: the same defect in every tool points at the emitter; one tool only points at that tool's path (tag, CDP mapping, connector, transformation, event rule). The script prints a cross-tool matrix.
• Group findings with one cause: a second producer explains a ratio above 1.0 and the completeness FAILs that match its share.
• Version tail: split rows by app version in a separate profiler run, read by hand and never passed to drift. When 90% or more of a defect's rows come from versions older than the fixing release, rate WARN with an expiry in decisions.md.
• Not shipped is not broken: proposed and approved events are not expected yet; implemented or verified with no rows is a defect or a wrong status.
• Canonical name in a variant pair: the plan's name when consumers use it, never chosen by volume or style.
• Types are fixed at the emitter. A legitimate new enum value amends the plan after the enforcement copies list it.
• Downgrading any FAIL requires citing decisions.md or the contract.

7. Confirm before you conclude

• Every FAIL needs one piece of evidence beyond the inventory: a test-account payload from the tool's live view or debugger, the emitter location, or the system-of-record count. Otherwise it is labelled unconfirmed.
• Search every emitter root before calling anything "not emitted" or proposing retirement; until then write "not observed in " (references/emitter-search.md).
• Never submit forms, orders or consent choices on a live site without the user's approval for that test.

8. Reconcile and report

• Give each finding one disposition: fix implementation, amend plan, adopt into plan, retire, rename or merge, accepted exception, or needs decision. Rules: references/reconciliation.md.
• Write the report from assets/audit-report-template.md, plan changes from the script's draft (assets/plan-changes-template.csv) and the change list from assets/change-list-template.md: privacy, double counting and wrong values, missing data, plan amendments, hygiene, one hypothesis per request.
• Propose updates to martech-context/ (tracking-plan, data-dictionary, decisions) and write nothing until the user approves.

Common failure modes

• Two producers, one name: D12 ratio above 1.0, with required properties present on only part of the rows (D07).
• One action under several spellings, from platform teams or a connector's rename settings: D03.
• Numbers sent as text through quoted tag variables or string building, or frozen by a warehouse column's first value: D10.
• IDs sent as numbers, losing leading zeros or digits above 2^53 in Mixpanel: D10 and D14.
• Enforcement that drops silently (Amplitude reject keeps the event without the property; Segment omission works only in cloud mode): D07 lower in the tool than in the warehouse.
• Plan says verified, tool says nothing or stops partway (emitter behind a flag, a refactor, a GA4 modification rule): D01, before or inside the window.
• Dynamic names, keys and per-page events pushing toward Amplitude's 2,000 event types or a Lexicon file's 5,000 events: D04.
• Short IDs that Amplitude removes, and IDs missing where the plan requires them: D14, upstream and per event.
• A time zone offset or late data read as double counting or loss: D12.
• Short vendor windows read as clean: Avo and Amplitude errors 24 hours, Segment violations 7 days.
• Full catalogue with detection queries: references/failure-modes.md.

Worked example

Fictional Acme Reports, plan "trial signup to first report", audited on 2026-11-03 for 2026-10-03 to 2026-10-30 against Amplitude (through Segment) and GA4. Quality ends with 0 FAIL, 2 WARN, 11 PASS, 0 not checked (verdict contract) and drift with 10 FAIL, 2 WARN, 5 PASS, 4 not checked. Among them:

• GA4 sign_up at 1.90 times the identity database count (both in UTC days), with method and event_id missing on 47.4% of rows: one second producer, one change request (CR-2).
• email sent as a user_registered property: CR-1, first in the change list.

Commands, the agent's handling of every finding, CR-1 and CR-2 in full and the regression fixtures: references/worked-example.md.

Acceptance checks

• Baseline: plan location and version, tools with placeholders, production scope, window, as-of date, time zone per tool and per reference, inventory route, read-only boundary and row totals per tool, caps checked. Pass: all present, and any capped or partial inventory is named.
• Plan quality: Q01 to Q13 each reported with coverage and PASS, WARN or FAIL, plus the verdict and the naming line. Pass: no rule missing, blocking failures named, drift depth stated.
• Coverage: every planned event mapped to an audited tool has a status line for that tool, and every observed name is classed as planned, variant, vendor, generated, dynamic, test or orphan. Pass: zero unclassified names.
• Numbers: every FAIL and WARN cites its rule, the rows affected and the window; share rules (D07, D08, D10, D11, D12, D14 shares) also cite the denominator and share. Pass: no finding without its numbers.
• Evidence: every FAIL has a payload, an emitter location or a system-of-record count, or is labelled unconfirmed. Pass: no silent confirmation.
• Not-emitted claims: nothing is called unused or proposed for retirement unless every emitter root was searched and no consumer exists. Pass: the roots searched are listed.
• Reconciled plan: every finding has one disposition; every plan edit names its change class, reason and approver role; retired items stay in the plan. Pass: the plan-changes rows match the findings that need plan edits.
• Change list: ordered by the five buckets; each request names system, entity, current state, change, owner, tests with evidence per layer, rollback and what rollback does not undo, and the reporting annotation. Pass: a person could apply it without asking a question.
• Privacy and safety: no property values beyond enum values and redaction tokens, no credentials, reads only in the tool call log, nothing written to martech-context/ or the skill folder. Pass: a search of the outputs finds no email or phone patterns other than named test accounts on example.com or the team's test domain, and no write calls.

Boundaries

• Read-only, always. The skill never blocks, hides, merges, deletes or dismisses anything, never creates branches or uploads plans, and never edits Lexicon, Amplitude Data, Avo or Protocols, even when the connected identity could. Fixes leave as a change list and a reconciled plan draft for human approval.
• A prompt that says "read only" removes no permission. The boundary is the identity's role, the server's read-only switch or the client's tool deny list; record which one held in the baseline.
• Retirement goes through the change list: stop emission at the source first, and block or delete in a tool only after it shows zero rows for 7 days, because blocked data cannot be recovered.
• Legal basis and deleting personal data already collected belong to the privacy owner and each vendor's deletion process. Metric definitions belong to their owners in metrics.md; the audit names the consumers a fix will move.
• Hand-offs:
  • tracking-plan-builder for contracts of adopted events and change control; event-naming-convention for renames; the skills under When to use for the rest.
  • Google's GA4 MCP server with the analytics.readonly scope and without --allow-write. Google's BigQuery remote MCP server with execute_sql_readonly only (execute_sql and cancel_job denied in the client and by an admin's IAM deny policy), or the console. Set Maximum bytes billed in the console. execute_sql_readonly has no bytes cap, so set jobTimeoutMs and a label too (spelling per the client's tool schema); a project quota is only a backstop.
  • Mixpanel MCP: no read-only mode is documented. Before the first call, deny in the client every tool starting with Create, Update, Delete, Duplicate, Edit, Bulk-Edit, Rename, Dismiss or Merge, and record that deny list as the boundary; if the client cannot deny tools, use the Lexicon CSV and the raw export.
  • Amplitude MCP with a role granting only Use MCP (read), never manage_amp_*.
  • PostHog MCP with readonly=true; Avo MCP with the read scope only (decline the write-scope consent); RudderStack MCP (no read-only mode documented) with only its read tools allowed in the client, never its write or sensitive execution tools; Segment changes made by a workspace owner in the app (Twilio's MCP server only searches docs).

Live docs

Platform settings, limits and menu paths change. Before stating any parameter name, limit, quota or UI path:

• Fetch the current page from the vendor docs MCP server or llms.txt listed in sources.yaml, or open the canonical URL.
• If the live page disagrees with this skill, follow the live page and tell the user what changed.
• Cite the URL you used.

Most used: GA4 limits and naming rules, the BigQuery export schema and MCP server, Mixpanel raw export and MCP tools, Amplitude Export API and HTTP V2 rules, PostHog MCP, Segment Protocols and warehouse schema. Checked 2026-11-01 to 2026-11-06; re-check MCP tool lists before relying on a name.