# Acceptance tests: cases, evidence and sign-off A payload in a browser debugger proves one checkpoint. A contract is verified only when a controlled action has been traced through every layer to the report or workflow that uses it. ## Contents - How to run a trace - Case catalogue - Evidence per layer - Reconciliation after release - Monitoring and alert thresholds - Sign-off ## How to run a trace - Use a staging environment or a production test account flagged `is_internal` at creation. Never use a real customer. - Write the expected result for every layer before executing the action. A test written after the result is a description, not a test. - Record the `event_id`, `message_id`, `user_id` and object ID of the controlled action, then follow those IDs through each layer. - Run each case once for every source that emits the event (web, iOS, Android, server). ## Case catalogue Every contract gets the first six. Add the rest where the product supports the situation. Number negative cases T2.1, T2.2 and so on, in the order of the `must_not_fire` list, so a reviewer can match them one to one. - Normal success: one canonical event, all required properties present, correct account and workspace. - Each `must_not_fire` condition: no event. One negative case per listed condition, including the delivery retry item. - Normal failure: no success event; the failure event, if the plan has one, with a controlled error category and no raw error text. - Delivery retry of one occurrence (force a timeout after the fact is committed): two sends with one `message_id` at emission; one canonical event; destination counts unchanged. - Repeated genuine action: two events with different `event_id` values; the metric that counts "first per unit" still counts the unit once. - Prohibited field present in the source data (an email in a form, an error body with customer text): the field is absent from every destination payload, including URL parameters. - Companion or rename overlap, whenever `other_emitters` or `dual_emission` is set: send one action so the canonical event and its companion (or the old and new names) pass through the same CDP. Expect both in the CDP debugger with different `message_id` values, both in every mapped destination, one shared `event_id`, and one count in each metric. This is the case that catches a shared transport key. - Missing optional property: event accepted; report shows the documented null meaning, not zero. - Unknown enumeration value: the `unknown_value_policy` happens (rejected, or sent as `unknown` with an alert). - Identity: - anonymous visit, then signup: pre-signup events link to the new user in each product analytics destination - login on a second device: same user, no new known person - logout, then another person logs in on the same device: two separate people - user switches account or workspace: events carry the account at event time - empty or placeholder user ID reaches the emitter: identify call is not sent - Consent denied, then granted, then withdrawn: each destination named in `denied_behaviour` receives what it says, and nothing more. Check server routes too: a server event forwarded by the CDP is not stopped by the browser consent platform. For GA4, expect no request under basic consent mode, and under advanced mode a cookieless request with the denied state visible in Tag Assistant's consent view. - Automated actor: system-initiated event accepts a null `user_id`, and destinations that need a user or device ID do not receive it. - Boundary: an outcome just inside the metric window, and one exactly at the end, land on the correct side (an end-exclusive 168-hour window excludes an outcome at exactly 168 hours). - Derived CRM properties: two accounts sharing one email domain stay two CRM records; re-running the sync creates no new record; a second workspace in the same account follows the roll-up rule. - Mobile (see `mobile-and-offline.md`): offline action flushed later keeps its original `event_id`, `message_id` and `occurred_at`; an older app version still sends the previous definition and is distinguishable by `app_version`. - Money and billing (see `ecommerce-and-billing.md`): duplicate or out-of-order webhooks give one canonical event per business transition; two partial refunds give two events. ## Evidence per layer For each case record, in the contract's acceptance entry and in `links.staging_evidence`: - Emission: the payload as sent (SDK debug log, network request, server log line) with `event_id` and `message_id`. - Collection: the event as received by the CDP or collector (source debugger or live events view). - Destination: the event as received in each mapped destination, with the final name, properties, identity fields, timestamp and the value of its dedupe field. Use the destination's own debug or live view; check the current feature name in the vendor docs. - Canonical: the warehouse row, showing one row per event name plus `event_id` after transport duplicates were removed on `message_id`. - Use: the metric value or workflow state (CRM property set, email suppressed, audience joined) that the event was supposed to change. A negative case can record one line that names every layer it was checked in ("absent in the server log, the Segment debugger, Amplitude and the warehouse; metric unchanged"). A case passes only when every layer matches the expected result. A plausible chart built from an event with the wrong workspace is a failure. ## Reconciliation after release - Server-confirmed events: compare daily counts with the application database rows for the same fact, window and exclusions. Expect equality after deduplication, allowing for late arrivals inside the refresh window. - Browser events: compare with the server fact they relate to and expect a lower, stable ratio (consent, blockers, interrupted pages). A ratio that jumps or exceeds one points to double firing or a second producer. - Money: compare totals with the order or billing ledger in the same currency and unit. A ratio near 100 means cents and dollars are mixed. ## Monitoring and alert thresholds Derive each threshold from a field the contract already holds, then write the owner and response time beside it. - Server tier 1 event: alert when the deduplicated daily count differs from the application database row count for the same day once the contract's refresh window has passed. Any difference is a defect. - Browser companion: alert when its daily ratio to the server fact leaves the band observed over the trailing 28 days, or goes above 1.0. - Required properties on server events: alert when completeness drops below 100 percent. On client events, alert on a drop below the trailing 28-day level. - Delivery delay: alert when the 95th percentile of `received_at` minus `occurred_at`, by source and app version, exceeds the smallest maximum lateness in the contract's `late_arrival_policy`. A workflow that ignores late events is silently skipping customers at that point. - Enumerations with `reject` or `send_unknown_and_alert`: alert on the first unknown value. - Server sources: alert on any two rows with the same event name and `event_id` but different `message_id` values (a second producer of the same fact), and on any two rows with the same object ID and `occurred_at` within seconds but different `event_id` values (a new ID per retry). Rows that share a `message_id` are ordinary retries removed by the warehouse rule. - Transport keys: alert when two different event names share one `message_id`. - Enforcement tools: alert on any new violation type in Protocols, RudderStack or Amplitude Data unexpected events. Response by tier: - Tier 1 (revenue, ad conversions, CRM workflows, identity, consent): page the implementation owner; acknowledge within 4 business hours; the data owner annotates affected reports the same day. - Tier 2: ticket to the implementation owner, triaged within 2 business days. - Tier 3: reviewed at the weekly data review. Adjust the hours to the team's on-call policy, and write the final numbers in the contract; do not leave "set from consequence" as the threshold. ## Sign-off - The data owner signs off that every case passed with evidence at every layer. - The business owner signs off that the metric or workflow produced the expected result for the controlled example. - Set the contract `status` to `verified` and add a changelog entry with the date and approvers.