# Measurement and holdouts (area 7) Attributed revenue is credit, not impact. A lifecycle programme proves incremental impact only with a control that was never messaged, sized to detect the effect that would change a decision, and excluded in every sender. This file holds the claims inventory, the holdout options per platform, the design and analysis rules, the statistics and rules MH01 to MH10. Designing a new experiment belongs to incrementality-test-design (planned); the audit judges claims and checks the holdouts that exist. Platform facts were read on 2026-11-01 unless marked "verify live". ## Claims inventory (MH01) List every impact claim in reports, decks and dashboards: - the claim as stated ("lifecycle drove USD 84,000 in September") - the method: attribution model, holdout, A/B test, before and after - windows and filters, the source report and its date - the verdict: credit, not incremental impact; supported; not proven; or invalid, with the rule that breaks it Facts that change verdicts: - Klaviyo attribution defaults for new accounts: email clicks 5 days, email opens 5 days, SMS clicks 5 days, SMS opens (RCS) 1 day, SMS deliveries 12 hours, push opens 24 hours. Bot clicks and Apple Privacy opens can be excluded from attribution. A settings change recalculates history and can take up to 36 hours. Save the settings and an export before anyone changes them. - Braze's Global Control Group report shows change from control, incremental uplift and a confidence figure, and has no data without a saved group. - Open-based attribution after Apple Mail Privacy Protection credits machine opens. ## Holdout options per platform ### Braze Global Control Group - Members receive no campaigns or Canvases on any channel except API campaigns, and except campaigns or Canvases carrying a tag listed in the exclusion settings. API-triggered Content Cards are withheld from members. - Size: no smaller than 1,000 users and no more than 10% of the audience. Under 10,000 users, raise the percentage to reach 1,000 but not above 15%. The value is an integer from 1 to 15. - Changes mean disabling the group and creating a new one; when the group grows, previous members stay in it. Export membership before any change. - Shuffle no more than once a month. Run at least one month; a quarter is a good start. - Random bucket segments can overlap the group (SG06). - Profiles sharing an email address: if one profile is not in the group and is eligible, the email is still sent to that address. - Campaign and Canvas control variants are separate from the global group. ### Customer.io - An email A/B test variation can be a holdout, or a random cohort branch can split a workflow. - The test's sending behaviour must be Queue Draft or Send Automatically. Holdout people register conversions within the conversion window. - The live variant always wins on opens and clicks, so judge on conversions. If the holdout wins, the message action is removed from the workflow. - No workspace-level global holdout is documented (page updated 2026-10-22); verify live. A global holdout is usually a profile attribute set at creation and filtered in every workflow. ### Iterable (read 2026-11-06) - Holdout groups exist only on blast campaign experiments with a custom conversion event, and not on send time experiments. Held-out users get a send skip event with reason `ExperimentHoldOut`. - Journey-level holdouts: verify live. ### HubSpot (verify live) - No native holdout was found. The usual pattern is a random property set at creation, plus a suppression list on every optional workflow and email. ### Klaviyo (verify live) - Read the current holdout and control options for flows and campaigns. Otherwise use a random property split set at profile creation. ## Design rules for a valid holdout - Assignment unit matches the outcome: whole accounts or workspaces for account outcomes, so colleagues do not cross arms (MH06). - Assignment happens before eligibility and never changes; post-assignment exclusions in one arm break the comparison. - Intent-to-treat: analyse everyone assigned, messaged or not (MH08). - Primary outcome and window fixed before launch: a product or commercial result, not opens or clicks (MH07). - Guardrails fixed too: unsubscribes, complaints, support contacts. - Required messages are never withheld: service, security, billing and legally required messages still reach holdout members (MH04). - Every sender excludes the holdout from optional messages; record any crossover (MH03). - Membership is exported before any reshuffle (MH10). ## Statistics - Sample ratio mismatch (MH05): chi-square with 1 degree of freedom against the intended split; FAIL when p is below 0.0005, the threshold Microsoft's experimentation platform uses. Causes fall into assignment, execution, log processing and analysis. Do not read the lift until the cause is found. - Count the assignment unit in the SRM check: workspaces for a workspace-level holdout, not the users inside them, whose numbers differ by workspace size. - Chi-square needs every expected count to be 5 or more. Below that, holdout_math.py uses a two-sided exact binomial test and says so. - Minimum detectable effect: `MDE = (z(1 - alpha/2) + z(power)) x sqrt(p x (1 - p) x (1/n_t + 1/n_c))`, with baseline rate p, alpha 0.05 and power 0.8 by default. - Interval for a difference in rates (unpooled): `diff +/- z(1 - alpha/2) x sqrt(p_t(1 - p_t)/n_t + p_c(1 - p_c)/n_c)`. - Small counts: when an arm has fewer than 5 conversions or 5 non-conversions, the unpooled interval is far too narrow and can exclude zero when an exact test does not. holdout_math.py then prints NOT RELIABLE, a Newcombe score interval built from two Wilson intervals and a two-sided Fisher exact p. Report not proven unless both exclude no difference. A 5% holdout of a small B2B base reaches this quickly: 30 of 380 against 0 of 20 gives an unpooled +5.18 to +10.61 points, but a score interval of -8.38 to +11.05 and Fisher p = 0.38. - "Not proven" means the interval includes zero. It does not mean the programme has no effect. - These formulas assume a binary outcome per unit, independent units and no spillover. Revenue, repeated outcomes and clustered units need a different analysis. ## holdout_math.py ```bash python3 scripts/holdout_math.py plan --baseline 0.21 --eligible-per-period 1200 \ --control-share 0.10 --periods 3 --target-mde-pp 3 python3 scripts/holdout_math.py result --treat-n 3310 --treat-conv 745 \ --control-n 290 --control-conv 58 --expected-control-share 0.10 ``` - `plan` prints the MDE, the periods needed for the target at this split and at 50/50, and MH09 WARN when the MDE is larger than the target. Without `--target-mde-pp` it reports MH09 as NOT CHECKED: ask the owner for the smallest effect that would change the decision before judging power. - `result` prints rates, the difference, the relative lift, the interval, "not proven" when it includes zero, and MH05 when `--expected-control-share` is given. With small counts it adds the NOT RELIABLE line, the score interval and the Fisher p, and bases the verdict on them. Exit status 1 on SRM FAIL. - `plan` adds a note when the control arm expects fewer than 5 conversions or non-conversions: the readout will need the small-count method. Reference numbers from the script (baseline 21% activation, 1,200 eligible workspaces a month, alpha 0.05, power 0.8): - 10% holdout over 3 months: MDE 6.34 points. Over 6 months: 4.48 points. - 20% holdout over 3 months: 4.75 points. - 50/50 over 3 months: 3.80 points. - To detect 3 points: 13.4 months at 10%, or 4.8 months at 50/50 (about 2,894 workspaces per arm). A 5 to 10% holdout of a small B2B audience cannot detect the effects lifecycle programmes usually have. Say so, and give the owner the choice: a time-boxed 50/50 test of one journey, or accept a coarse readout. ## Rules ### MH01 claims-inventory - WARN: an attributed figure presented as incremental impact without a control ("claim not supported"). - WARN "credit double counted": two or more senders report attributed conversions for the same window (Klaviyo and HubSpot, Customer.io and Braze) and the figures are summed. Deduplicate by order or conversion ID before summing. Evidence: the overlap share (conversions credited by more than one sender over all credited conversions), or a summed figure above the cohort's total revenue for the window. ### MH02 persistent-holdout - WARN: an optional programme with no persistent holdout; or a holdout outside vendor size guidance with no power calculation. - inventory_audit.py lists live marketing journeys that do not exclude the global holdout. ### MH03 contamination - FAIL: any optional message from any sender reached a holdout member while assigned. Detect: message_pressure.py `--holdout`, split by sender. Confirm: seed S08. - Membership changes over time: Braze adds new users to its Global Control Group, and a reshuffle moves people. Give the holdout file `assigned_at` and `released_at` columns so only sends inside each member's membership count. Without them the script says so; check the earliest flagged send against the assignment date before reporting contamination. - For an account-level holdout, list every profile in each held-out account, keyed like the send log, with the account's assigned_at. A list of account IDs alone matches no send. - When no holdout member appears in the send log, the script reports MH03 NOT CHECKED, not PASS: the keys or the salt differ, or no member received anything, not even a required message (MH04). ### MH04 required-messages - FAIL: service, security, billing or legally required messages withheld from the holdout. ### MH05 sample-ratio - FAIL: p below 0.0005 against the intended split (chi-square, or the exact binomial test when an expected count is below 5). ### MH06 assignment-unit - WARN: person-level assignment for account-level outcomes. ### MH07 outcome-and-window - WARN: opens or clicks as the primary metric, or a window and guardrails not fixed before launch. ### MH08 intent-to-treat - FAIL: the analysis compares openers with non-openers, or only delivered recipients, instead of everyone assigned. ### MH09 power - WARN: the minimum detectable effect is larger than the smallest effect that would change the decision. ### MH10 governance - WARN: no membership export before a reshuffle, reshuffles more often than monthly (Braze), or a test shorter than one month.