# Worked example: Acme Reports (fictional) Acme Reports is the fictional B2B reporting product (app.example.com) whose "trial signup to first report" plan tracking-plan-builder wrote. Here the plan has been live for months and this skill audits it. Every name, ID and number is invented. ## Files and baseline - [../assets/example-acme-plan-events.csv](../assets/example-acme-plan-events.csv) and [../assets/example-acme-plan-properties.csv](../assets/example-acme-plan-properties.csv): the plan, canonical in `martech-context/tracking-plan/` as YAML contracts, normalized to two tabs. 8 events, 12 property rows. - [../assets/example-acme-observed.csv](../assets/example-acme-observed.csv): inventories for Amplitude (fed by Segment, project ``) and GA4 (web stream G-XXXXXXXXXX, dataset analytics_XXXXXXXXX, standard property, cut in UTC days with the `event_timestamp` filter). - [../assets/example-acme-reference-counts.csv](../assets/example-acme-reference-counts.csv): users created in the identity database, the same count for both tools, in UTC. - [../assets/example-acme-ga4-custom-definitions.csv](../assets/example-acme-ga4-custom-definitions.csv): the GA4 custom definitions, saved from `get_custom_dimensions_and_metrics`. - [../assets/example-acme-amplitude-export.json](../assets/example-acme-amplitude-export.json): 59 raw Amplitude-format events to show the profiler. - Window 2026-10-03 to 2026-10-30, lookback from 2026-08-02, as of 2026-11-03 (4 days after the window: Google can change the 2026-10-30 table through 2026-11-02, so it is final that day). Segment is mapped in the plan but not audited here, so the run says so. ## Commands From a scratch folder outside the skill folder and outside `martech-context/`: ```bash cd "$(mktemp -d)" SKILL="/path/to/skills/tracking-plan-audit" # the installed skill folder python3 "$SKILL/scripts/profile_export.py" "$SKILL/assets/example-acme-amplitude-export.json" \ --format amplitude --tool amplitude --since 2026-10-03 --until 2026-10-30 \ --occurrence-key event_properties.event_id --values-for connector_type,report_type,signup_method \ --identity-keys user_id > observed-sample.csv python3 "$SKILL/scripts/audit_plan.py" quality --events "$SKILL/assets/example-acme-plan-events.csv" \ --properties "$SKILL/assets/example-acme-plan-properties.csv" python3 "$SKILL/scripts/audit_plan.py" drift --events "$SKILL/assets/example-acme-plan-events.csv" \ --properties "$SKILL/assets/example-acme-plan-properties.csv" \ --observed "$SKILL/assets/example-acme-observed.csv" \ --reference "$SKILL/assets/example-acme-reference-counts.csv" \ --ga4-definitions "$SKILL/assets/example-acme-ga4-custom-definitions.csv" \ --window-start 2026-10-03 --window-end 2026-10-30 --out audit-out ``` ## What each command prints - Profiler: 59 records parsed, 57 in the window, 2 before `--since`. `data_source_connection_failed` comes out with rows 0 and last seen 2026-09-14 (the stopped signal). `user_registered` has 1 repeated `event_id`: a retry sent a new insert ID, so Amplitude kept both rows, since it only ignores a repeat of the same device_id and insert_id within 7 days. Transport duplicates are 0. `email` has 8 pattern hits and `[Amplitude] Page URL` 1 (an email in a query string); `plan_tier` has 1 "undefined" per affected event. The identifier rows show `user_id` null on 1 of 14 `report_generated` rows: Amplitude removed an ID shorter than 5 characters at ingestion, so the short ID itself shows only in the Segment copy. No value of a non-enum property is written. - Quality ends with `0 FAIL, 2 WARN, 11 PASS, 0 not checked. Nothing was changed.` and the verdict "contract". The WARNs: Q03, `report_exported` has no implementation owner; Q06, `report_exported.export_format` has no null meaning. - Drift ends with `10 FAIL, 2 WARN, 5 PASS, 4 not checked. Nothing was changed.` and exit code 1. Shortened lines: ```text FAIL D13 pii amplitude: user_registered.email: named like personal data; email, phone or sensitive URL patterns in 1,240 of 1,240 occurrences. Values are not printed. FAIL D12 volume-and-duplicates ga4: sign_up: 2,356 events against 1,240 from identity database (users created), ratio 1.90, browser route, days cut in UTC on both sides: more events than facts ... FAIL D10 type-mismatch amplitude: report_generated.duration_ms (source server): plan integer; numbers sent as text in 612 of 18,450 occurrences (3.32%) ... FAIL D11 enum-drift amplitude: data_source_connected.connector_type (source server): 'snowflake' 214 not in the allowed list ...; 214 of 5,120 rows (4.18%) ... FAIL D03 name-variant amplitude: 'Report Generated' (ios, 3,912 rows) beside planned 'report_generated' (server, 18,450 rows): one action under two spellings, 17.5% of its rows under the unplanned one. FAIL D07 required-incomplete amplitude: user_registered.signup_method (source server): present in 1,203 of 1,240 rows (97.0%), required on a tier 1 server event ... FAIL D07 required-incomplete ga4: sign_up.event_id (source web): present in 1,240 of 2,356 rows (52.6%), below 95.0%. FAIL D07 required-incomplete ga4: sign_up.method (plan signup_method, source web): present in 1,240 of 2,356 rows (52.6%), below 95.0%. FAIL D01 planned-unseen amplitude: data_source_connection_failed: stopped. Last seen 2026-09-14, no rows from 2026-10-03 to 2026-10-30; status verified. FAIL D05 lifecycle amplitude: trial_extended: status retired but 312 rows in the window from android ... WARN D02 unplanned-event amplitude: export_clicked: 9,880 rows in the window, not in the plan (rank 3 of 9 names by volume, 11.1% of amplitude rows, priority). WARN D09 unplanned-property amplitude: app_build: on 9 of 9 event names (100.0%); one unplanned global property ... INFO D01 planned-unseen amplitude: report_exported: status approved, not shipped yet; no rows expected. PASS D15 ga4-registration: 2 checks; every planned GA4 parameter that needs a custom definition is registered ... NOT CHECKED D12 volume-and-duplicates: no reference count for tier 1 events: data_source_connected (amplitude). ``` The matrix prints `user_registered: ratio to the reference amplitude 1.00, ga4 1.90. Above the reference only in ga4` and flags `event_id` only in GA4. In `audit-out/findings.csv`, F07 and F08 read affected 1,116 rows without it, of 2,356, share 47.37%. ## What the agent does with each finding - GA4 D12 and both GA4 D07 lines are one hypothesis: a second, page-based `sign_up` producer. The 1,116 rows without `method` equal 2,356 minus 1,240. Amplitude matches the identity database (ratio 1.00, both sides in UTC), so the emitter is fine and the defect sits on GA4's path. The container goes to gtm-container-audit, whose page-based conversion rule finds a GA4 event tag on the `/welcome` page view. One change request, CR-2, closes all three findings. - D13 `email`: CR-1, first in the list. The privacy owner decides on the 1,240 rows already collected, through Amplitude's deletion process. - D07 `signup_method` at 97.0%: the 37 rows without it are microsoft_sso signups through the invitation landing page. The matrix says "flagged in every tool", but the GA4 share is explained by CR-2; the emitter gap is CR-5. - D10 `duration_ms`: the report job runner's retry path builds the value as text. CR-4 fixes the emitter; the plan type stays integer. - D11 `snowflake`: the product owner confirms a supported connector since the 2026-10-01 release. Amend the plan (compatible, no version bump, tier 1 approvers because HubSpot routing reads the event), add the value to the Protocols enum first, and check the HubSpot routing mapping. - D03 `Report Generated`: iOS 4.11 added a client-side emission; the server is the source of truth. CR-3 stops it; iOS dashboards that read it are listed under "could break". - D01 `data_source_connection_failed`: the emitter search finds the call behind the flag `connection_errors_v2`, switched off in the 2026-09-15 release. Fix implementation (CR-6); status goes from verified to implemented until T1 and T2.1 to T2.3 pass again. - D05 `trial_extended`: a separate profiler run with `--source-key version_name`, read by hand and never passed to drift, shows all 312 rows from Android 4.9.x. `decisions.md` holds a tail exception for versions below 4.10 until 2026-12-31, so the agent downgrades to WARN, cites that entry, and drafts status deprecated with that end date. - D02 `export_clicked`: no consumer found; the product owner decides between a planned intent event (written with tracking-plan-builder) and retirement. - D09 `app_build`: the data owner decides between a global optional string in `data-dictionary.md` and stopping it. - D15 PASS: `method` fills a predefined GA4 dimension and `event_id` stays unregistered for export joins, so neither needs a custom definition. - Profiler duplicate: the repeated `event_id` on `user_registered` is must-not-fire case 4 (a retry with the same idempotency key). On the full export, with `--occurrence-key event_properties.event_id`, any repeat on this tier 1 event is a D12 FAIL and gets its own change request, CR-7, since its cause (the retry path) differs from the other requests. - Not checked: the agent asks for the connections table count (tier 1 `data_source_connected`), adds Part C of the GA4 query for `method` values, and runs the profiler with `--identity-keys` on a full export, plus Segment Query 4 for the short IDs Amplitude removed. ## CR-1 and CR-2 ```markdown ### CR-1 Stop sending email on user_registered (D13 FAIL, F01) - System to change: identity service, Segment server call. - Entity: amplitude, user_registered, email. - Current state: email present on 1,240 of 1,240 rows with email patterns, 2026-10-03 to 2026-10-30. - Change: remove email from the event properties the identity service builds; keep it in the identity database only. - Plan edit: none (the contract already lists email under pii_exclusions). - Could break: nothing that is correct today; check that no Amplitude cohort filters on email. - Test (staging, , approved by the privacy owner): T1 signup; email absent in the service log, the Segment debugger, Amplitude and the warehouse row; T4 marketing link with an email in the query string. - Rollback: revert the commit. Does not undo: the 1,240 rows already in Amplitude and the warehouse; the privacy owner runs each vendor's deletion process. ### CR-2 Remove the page-based GA4 sign_up producer (D12 FAIL F02, D07 FAIL F07 and F08) - System to change: GTM web container GTM-XXXXXXX. - Entity: ga4, sign_up, method and event_id. - Current state: 2,356 sign_up events against 1,240 users created (ratio 1.90, both counted in UTC days); method and event_id missing on 1,116 of 2,356 rows (47.4%), 2026-10-03 to 2026-10-30. - Change: pause, then remove, the GA4 event tag that sends sign_up on the /welcome page view; the dataLayer companion stays the only producer. - Could break: GA4 sign_up falls by about 47%; the key event and the imported ad conversion fall with it. The acquisition owner is told before release and the period is annotated. - Test (staging, approved by the web analytics lead): one signup; DebugView shows one sign_up with method and event_id; a /welcome reload sends none. - Post-release check: 7 days of export; ratio at or below 1.0, method and event_id at 100%. - Rollback: republish the previous container version. Does not undo: duplicates already in GA4 and in ad platforms. ``` ## Plan changes The `--out` draft holds five rows; the agent's decisions replace the "decide:" proposals: - PC-1 F04 `data_source_connected.connector_type` allowed values: add snowflake; compatible; no version bump; business, data and implementation owners. - PC-2 F09 `data_source_connection_failed` status: verified to implemented until CR-6 passes; documentation. - PC-3 F10 `trial_extended` status: retired to deprecated, end date 2026-12-31, citing `decisions.md`; documentation. - PC-4 F11 `export_clicked`: needs decision (adopt as an intent event or retire). - PC-5 F12 `app_build` (global): needs decision (add as an optional string or stop sending). ## Regression fixtures [../assets/regression-plan-events.csv](../assets/regression-plan-events.csv), [../assets/regression-plan-properties.csv](../assets/regression-plan-properties.csv), [../assets/regression-observed.csv](../assets/regression-observed.csv), [../assets/regression-reference-counts.csv](../assets/regression-reference-counts.csv) and [../assets/regression-ga4-custom-definitions.csv](../assets/regression-ga4-custom-definitions.csv) hold one case per event or property. After any script change, run from a scratch folder: ```bash python3 "$SKILL/scripts/audit_plan.py" quality --events "$SKILL/assets/regression-plan-events.csv" \ --properties "$SKILL/assets/regression-plan-properties.csv" python3 "$SKILL/scripts/audit_plan.py" drift --events "$SKILL/assets/regression-plan-events.csv" \ --properties "$SKILL/assets/regression-plan-properties.csv" --observed "$SKILL/assets/regression-observed.csv" \ --reference "$SKILL/assets/regression-reference-counts.csv" \ --ga4-definitions "$SKILL/assets/regression-ga4-custom-definitions.csv" \ --window-start 2026-10-03 --window-end 2026-10-30 ``` Expected: quality `1 FAIL, 0 WARN, 12 PASS, 0 not checked` (Q09 for the 44-character GA4 name and the reserved and malformed names below) and drift `28 FAIL, 17 WARN, 0 PASS, 15 not checked`. Per case: - GA4 `value` planned number, integer and decimal rows: no D10. `transaction_id` as integers in 50 of 1,000 rows: D10 FAIL "IDs sent as numbers". 3 repeated transaction IDs: D12 FAIL. - `$mp_web_page_view`, `$identify`, `[Amplitude] Page Viewed`, `Application Opened` and `Loaded a Page`: vendor. `ga_session_id`, `page_location` and Mixpanel `utm_source`: no D09. `app_build` on 12 of 13 names: one D09 global WARN. - `Signup Completed` beside `signup_completed`: D03 FAIL, 13.0% of the action's rows; stale `signupCompleted`: D03 WARN. Twelve `viewed_product_` names: one D04 FAIL. "Viewed Pricing Page", "Viewed Docs Page" and "Loaded Home Page": one D04 WARN for per-page names. `qa_test_event`: D02 WARN, test, not priority at 0.06% of Mixpanel rows. `legacy_banner_clicked` listed without counts: D02 WARN. - Deprecated `legacy_export_started` with rows: D05 WARN. Retired `old_invite_sent` with rows: D05 FAIL. - Required at 96% (tier 2 web): D07 WARN; at 94%: D07 FAIL; at 99.5% on a tier 1 server event: D07 FAIL. - `rating` mismatch in 2 of 40: D10 WARN, low sample. Numeric text on a planned string ID: no D10. Numeric text in 3% of a planned integer: D10 FAIL. - `unknown` under send_unknown_and_alert: INFO only. `quarterly` at 2.06%: D11 FAIL. `paypal` under reject: D11 FAIL. `share_channel` with 50 rows in the `` tail and every allowed value listed: D11 FAIL. - Approved and unseen: INFO. Implemented tier 3, never seen: D01 WARN. Verified, never seen: D01 FAIL. Rows 0 with last seen before the window: D01 FAIL, stopped. The 44-character GA4 name: D01 FAIL with the length as root cause. `export_scheduled`, 400 rows and silent from 2026-10-12: D01 FAIL, stopped inside the window. `project_created` from iOS silent from 2026-10-14 while web sends: D01 WARN for that source. `profile_updated` listed without counts: D01 INFO. GA4 names `error`, `ga_coupon_redeemed` and `plan compared`: Q09 FAIL for each (plus the parameter `ga_coupon_code`), and D01 with the name as root cause, FAIL on the two verified events and WARN on the implemented one. - Tier 1 `signup_completed` from web in Amplitude, planned server-only: D06 FAIL. GA4 `sign_up` from web with the `ga4:web` override: no D06. `trial_converted` with `source_system` backend and a `ga4 via gtm (browser companion)` path: no D06. `subscription_renewed` planned from a server and seen as GA4 web rows: D06 INFO, no FAIL. `dashboard_viewed` with no source label: one D06 NOT CHECKED line for Mixpanel. - Ratios: server 0.995 with UTC on both sides INFO; server 0.97 D12 FAIL; browser 1.20 D12 FAIL; browser 1.004 with UTC against America/New_York D12 WARN; `report_run` with route `backend` at 0.94 reads as a server route, D12 FAIL; `feedback_submitted` at 0.80 with no band D12 INFO; `dashboard_viewed` with route `edge` at 0.92 D12 NOT CHECKED, not judged. - `email` property: D13 FAIL. Pattern hits on `search_term`: D13 FAIL. - Segment `user_id` under 5 characters, upstream of Amplitude: D14 FAIL. Amplitude `signup_completed.user_id` null on 2 of 2,000 rows of a tier 1 server event: D14 FAIL. `account_upgraded.user_id` null on 10 of 970 rows, tier 2: D14 WARN. Mixpanel `distinct_id` placeholders at 2%: D14 FAIL; its short IDs: D14 WARN. - Segment column `revenue` string where the plan says number (money): D10 FAIL; `gift_wrap` string where the plan says boolean: D10 WARN. - Optional `template_id` never seen on 5,000 rows: D08 WARN. - D15: `signup_source` registered user-scoped on a tier 1 event: FAIL; `delivery_speed` unregistered: WARN; `value`, `currency`, `transaction_id` and `method` are predefined or identifiers and skipped. - NOT CHECKED: tier 1 events without reference counts; the reference row with no time zones; the reference row for the names-only `profile_updated`; GA4 without identifier rows; Mixpanel rows without a source label; Segment completeness with column types only; D13 for Segment, whose column rows carry no pattern counts; for PostHog, whose `onboarding_completed` has counts only, D01 (no last_seen dates), D07 to D11, D13 and D14. - By hand, outside the fixture: a profiler file split with `--source-key version_name` passed to drift stops with exit code 2, and so does `regression-observed.csv` passed twice, naming both lines, `--out` through a symlink into the skill folder or `martech-context/`, a window date that is not YYYY-MM-DD and an inventory with a header and no rows. The profiler exits 2 on a file it reads no events from, such as a CSV passed as `--format amplitude`. Second run, on what a PASS may claim. [../assets/regression-thin-observed.csv](../assets/regression-thin-observed.csv) holds only tool, source, event and rows for the Acme plan, and [../assets/regression-thin-reference-counts.csv](../assets/regression-thin-reference-counts.csv) one ratio with no band beside one that matches: ```bash python3 "$SKILL/scripts/audit_plan.py" drift --events "$SKILL/assets/example-acme-plan-events.csv" \ --properties "$SKILL/assets/example-acme-plan-properties.csv" \ --observed "$SKILL/assets/regression-thin-observed.csv" \ --reference "$SKILL/assets/regression-thin-reference-counts.csv" \ --window-start 2026-10-03 --window-end 2026-10-30 ``` Expected: `0 FAIL, 0 WARN, 7 PASS, 11 not checked`, exit code 0. The D01 PASS leaves out "kept sending to its end" and a D01 NOT CHECKED lists the 6 events without dates; D12 prints `1 event count matches the reference; 1 ratio has no documented band and was reported, not judged`; D13 prints NOT CHECKED, not a PASS. Third run, on a plan with integrity defects. [../assets/regression-broken-plan-events.csv](../assets/regression-broken-plan-events.csv) and [../assets/regression-broken-plan-properties.csv](../assets/regression-broken-plan-properties.csv) repeat an event and a property, name a misspelled event, differ only by case, and map two events to one GA4 name: ```bash python3 "$SKILL/scripts/audit_plan.py" quality --events "$SKILL/assets/regression-broken-plan-events.csv" \ --properties "$SKILL/assets/regression-broken-plan-properties.csv" python3 "$SKILL/scripts/audit_plan.py" drift --events "$SKILL/assets/regression-broken-plan-events.csv" \ --properties "$SKILL/assets/regression-broken-plan-properties.csv" \ --observed "$SKILL/assets/regression-thin-observed.csv" ``` Expected: quality `2 FAIL, 0 WARN, 11 PASS, 0 not checked`, exit code 1: Q07 FAIL with `event user_registered is defined on lines 2 and 7`, the repeated property and `user_registerd`, plus the case collision as its WARN; Q08 FAIL with `ga4 name trial_begin is mapped from events trial_started and trial_resumed`; verdict names only. Drift stops with exit code 2 and names the same lines.