# Two-candidate pilot and pre-signing checks

Run a pilot only when the hard-requirement check left two candidates standing. A pilot measures whether the intended operator can produce a correct, repeatable answer on the plan the company will keep. A vendor demo measures neither.

## Contents

- Before the pilot
- The analysis contract
- Fixtures
- Identity check on the current tool
- Operator tasks
- Scoring
- Duration
- Pre-signing checks

## Before the pilot

- Write the decision, the population and the outcome as one sentence a buyer can test. Example: "For trial workspaces with a complete seven-day window, find where successful setup fails and decide which onboarding step to change."
- Classify every requirement as mandatory or preferred. A missing mandatory item ends the pilot for that candidate.
- Send both tools the same event contracts from the tracking plan and the same identity rules. A pilot that compares two different implementations compares implementations, not tools.
- Use the plan tier the company will buy. If a trial unlocks higher-tier features, list them and turn them off or exclude them from scoring.
- Pilot paid add-ons (account analysis, exports) on a startup plan that includes them or on a written, time-boxed add-on trial. Never commit to an annual tier in order to pilot.

## The analysis contract

Write it before building either chart, so a second person can check both:

```yaml
question: Which trial workspaces reach their first successful export?
entity: workspace_id
cohort_entry: first trial_started for the workspace
conversion: export_completed with status success
window: within 7 elapsed days of cohort entry
ordering: entry before conversion
repeat_trials: excluded
internal_workspaces: excluded
observation: only workspaces with a full 7-day window
output: eligible workspaces, converted workspaces, conversion rate
```

## Fixtures

Create known journeys in a test environment and write the expected result for each before opening either tool:

- Normal conversion by one person.
- Conversion where different members complete different steps in one workspace.
- One person active in two workspaces, acting in both. The account result must not mix them.
- A failure, a retry after failure, and a conversion outside the window.
- Anonymous visit, signup, second device, logout, then another person on the same device.
- A late-arriving backend event.

Account fixture worth copying: Person A connects a data source in workspace Alpha; Person B builds a report in Alpha; Person A shares a report in workspace Beta; Person B shares in Alpha. Expected: Alpha activated once, Beta not activated, and no person completed every Alpha step alone. A person-level funnel will disagree, and that is correct.

## Identity check on the current tool

When step 1 finds an identity gap, run the identity fixtures against the tool already installed before any purchase:

- Anonymous visit, then signup: the pre-signup events join the new person under the vendor's documented identity rules.
- Second device, after login: both devices resolve to one person.
- Logout: the next anonymous activity on that device is not attributed to the person who logged out.
- Shared device: a second person logging in on the same device stays a separate person.
- Account switch: one person acting in two workspaces keeps the workspaces separate.
- Backend event for the same person: it joins the browser person on the same ID.

Write the expected result for each before opening the tool, read the vendor's current identity docs (listed in `sources.yaml`) for the identify, alias or reset call it expects, and record pass or fail with the evidence. A failure here is fixed in the implementation; a new tool would inherit it.

## Operator tasks

The person who will run the reports does these after a short introduction, without the implementer driving:

- Build the funnel from the analysis contract and explain the denominator.
- Change the conversion window, exclude staff and compare two cohorts.
- Connect an acquisition source to retained usage, if that decision is in scope.
- Locate the event behind an unexpected number (the engineer's task).
- Repeat the funnel a week later from memory.

Record time to a correct answer, the help needed, and interpretation errors. "This tool felt more intuitive" is not evidence. "The lifecycle manager rebuilt the funnel unaided in twelve minutes" is.

If assistant access is a requirement, give each vendor MCP server the same bounded question with the definitions supplied, require it to show the query or tool output, and compare with the fixture result. Connect read-only where the server supports it.

## Scoring

Score three things separately, never as one average:

- Correctness: fixture results match the expected values; account and person results are both right.
- Usability: the intended operator reproduced the answers without help.
- Operability: someone can detect missing events, manage access, set usage alerts and retrieve raw records through the route the plan includes.

When both pass all three, choose the one with the cheaper complete month-13 forecast and the easier exit. Do not invent a requirement to break a tie.

The easier exit is the one that passed the exit test in "Raw export and the exit test" in [hard-requirements.md](hard-requirements.md): one week of events, the anonymous-to-known identity mapping and the tracking plan exported on the kept plan, loaded into a scratch warehouse table, and one funnel rebuilt to match. Record the route for each, for example:

- Mixpanel: Data Pipelines (events, user profiles, identity mappings) and the Lexicon CSV export.
- Amplitude: the Snowflake destination with "Export all merged Amplitude IDs", and the Taxonomy API.
- PostHog: batch exports of the events and persons models.

A candidate that needs a paid add-on for any of the three has the harder exit unless that add-on is in the forecast.

If a pilot failure traces to bad source data, fix the shared input, rerun the case in both tools, and keep the original failure in the record.

## Duration

- Collection and report configuration can be tested in days.
- Outcome rates need a matured cohort: the conversion window plus the cohort entry period. A two-day trial cannot turn pending customers into failures.
- Book the pilot end date after the last cohort matures, not after the vendor's trial clock.

## Pre-signing checks

Each item needs evidence attached to the memo:

- The recurring decision is named, with whether it counts people, accounts, workspaces or attempts.
- Fixtures reconcile with known application outcomes, including retry, logout and account switch.
- The intended user rebuilt a segment and explained the denominator.
- Required reporting, experiment and export capabilities exist on the plan after trial access ends, confirmed on the live plan page or in writing.
- Events, identified events or MTUs, sessions, replays, flag requests and experiment users are forecast separately for normal, growth, fault and month 13.
- At-limit behaviour is known for each meter, with an alert below 100% and an owner.
- Late-arriving data and a repeated export range were tested where export matters, and the exit test passed on the kept plan.
- Maintenance hours, warehouse compute and any overlapping subscription are in the cost.
- The decision, the rejected options and the evidence that would justify revisiting are recorded.