# Profiler reference

`scripts/profile_crm_export.py` reads CSV exports and writes local files under `--out`. It never calls a network service, never writes anywhere else (no bytecode cache either) and never changes a CRM. Python 3.9 or later, standard library only.

Run it from a working folder outside any repository and outside `martech-context/`, calling it by its path in the skill folder, so exports and outputs never land inside the installed skill or a project: `python3 "$SKILL_DIR/scripts/profile_crm_export.py" --help` prints the same options. The fixture commands in [worked-example.md](worked-example.md) run the same way.

## Code layout

The entry script imports five modules from the same folder, so keep `scripts/` together. Each can be read in one sitting:

- `profile_crm_export.py`: arguments, default rules, the self-test and the run order.
- `hygiene_core.py`: constants, normalisers (email, phone, domain, names, Jaro-Winkler) and value parsing. No file access.
- `hygiene_io.py`: the only module that opens input files: delimiter and encoding detection, header presets and mapping.
- `hygiene_checks.py`: populations, owners and associations, and the X, F, E, R, O, L and A checks. No file access.
- `hygiene_duplicates.py`: blocking, comparison, bands, the union guard and the D checks. No file access.
- `hygiene_outputs.py`: the only module that writes, always under `--out`; it reads `--previous` and an earlier calibration-sample.csv there.

None imports a network module.

## Options

- `--crm hubspot|salesforce|attio|generic`: header presets and CRM-specific checks. X03, D07, O05 and L10 apply to Salesforce; E07 needs HubSpot's marketing contact status column. Default generic.
- Input files, each optional, at least one object: `--contacts`, `--companies`, `--deals`, `--leads` (Salesforce), `--users`, `--contact-roles` (Salesforce OpportunityContactRole), `--queues`. A cross-object check runs only when both files are given; otherwise it is NOT CHECKED.
- `--map OBJECT.FIELD=Header`, repeatable: wins over the presets. Canonical field names are at the end of this file.
- `--rules FILE`: a copy of `assets/rules-template.json`. Unknown keys are listed in summary.md and ignored.
- `--unique-key OBJECT=Header`, repeatable: the D01 key. The rules file's `unique_keys` does the same; the command line wins.
- `--expected-count OBJECT=N`, repeatable: the CRM's count for the export's filter. Enables X08 and the coverage label.
- `--as-of YYYY-MM-DD`: the date for every age and window. Default today in UTC. Always pass it, so a rerun gives the same result.
- `--previous FILE`: last month's profile.json; trend.md compares with it.
- `--calibration N`: pairs per band per object in calibration-sample.csv, default 20. 0 leaves an existing file untouched and summary.md says so.
- `--out DIR`: default crm-hygiene-out, relative to the working folder.
- `--internal-domains a,b` and `--freemail-domains a,b`: added to the rules file lists.
- `--date-order mdy|dmy`: for slash dates the script cannot tell apart. Without it, checks on that column are NOT CHECKED.
- `--allow-reserved-domains`: keeps example.com, example.net, example.org and the .example, .test, .invalid, .localhost names in the populations. Fixtures only; never on real exports.
- `--unmask`: adds clear name and email columns to duplicates.csv for a person reviewing locally. The agent never passes it and never opens a file written with it.
- `--json`: totals and check lines as JSON on stdout.
- `--self-test`: 47 built-in assertions on the normalisers (Jaro-Winkler, email syntax and internationalised domains, phone keys, domains, Salesforce IDs and X03 case damage, bounce classes including SMTP reply text, amounts, masking, blank keys), the duplicate rules (the union guard, same name with different domains, band settings including `d04_band`), the owner settings on O01, L10 without IsConverted, a check with an empty population (NOT CHECKED, never PASS) and a Salesforce lead with only a bounce date.

## Input files

- One CSV per object with a header row. The delimiter is sniffed among comma, semicolon, tab and pipe. Encodings are tried in order: UTF-8 with or without a byte order mark, cp1252 (X04 WARN), Latin-1.
- users.csv: the ID owner columns store (HubSpot: the owner ID, headed Owner ID; Salesforce: the user Id), name, email, active (true or false) and kind (person, integration or queue). Owner values are matched by ID, name or email, so HubSpot exports that show owner names work. Rows with kind queue count as queues.
- Salesforce leads: include converted leads, with IsConverted and ConvertedContactId. They are set aside as archived records (summary.md gives the count): only the X checks, F07 and L10 read them. Without IsConverted no lead can be set aside: summary.md says so and L10 is NOT CHECKED.
- queues.csv: ID and name. contact-roles.csv: OpportunityId and ContactId.
- Multi-value cells (association IDs, additional emails) split on semicolons, commas, pipes or spaces.
- Dates: ISO 8601 with or without a time, or slash dates with the order detected per column (a first part above 12 means day first, a second part above 12 means month first).
- Amounts: currency symbols removed; 1,234.56 and 1.234,56 both read as 1234.56; 12,5 reads as 12.5. Anything else is unreadable and left out of L03 with a note.
- Booleans: true, false, yes, no, 1, 0 in any case. HubSpot's "Marketing contact" and "Non-marketing contact" labels are read as true and false.

## Outputs

- summary.md: baseline (files, rows, encoding, delimiter, sha256 prefix, coverage), rules version and hash, mapping report (mapped, not mapped, ambiguous), scorecard grouped by status with numerator, denominator, population and threshold, the duplicate summary with the excess convention, unique affected records per object, next steps. Ends with "Nothing was changed."
- findings.csv: check_id, status, object, record_id, field, detail. One row per record per check. The detail is a category or an internal reference, never a raw name, email or phone.
- duplicates.csv: group_id (CT, CO, DL or LD prefix), band, object, record_id, rule_ids, best_score, key_conflict, suggested_role (always labelled a suggestion), suggestion_reason (on the first member), created_at, owner (emails inside owner labels masked), association_count, source, masked_key, unique_key. key_conflict says whether the members hold different unique keys; the union guard refuses such groups, so a yes means the guard failed and the group must not be merged. Copy unique_key into the merge plan's unique_key_value.
- possible-pairs.csv: pair_id, band, object, record_id_a, record_id_b, rule_id, score, key_conflict, reason. The reason starts with the basis (for example "D05 same name, different domains") and ends with "key conflict" for a direct conflict or "refused" for a link the union guard stopped. Salesforce lead to contact matches (D07) appear here with object leads-contacts.
- columns.csv: per column, fill rate, placeholder rate, distinct count and the three most common shapes (letters become a or A, digits 9). Never values.
- profile.json: schema `crm-hygiene-profile/1` with as_of, crm, rules version and hash, inputs with sha256, mapping, coverage, every check with its record IDs, duplicate groups and pairs, every record ID and create date per object (for next month), the unique key per record where one is declared and the converted lead IDs (check_merge_plan.py reads both), unique affected records and totals. It holds record IDs and keys but no names, emails or phones; keep it with the report for the next run.
- trend.md, written on every run so an old one never survives a rerun. With `--previous`: a warning first when the rules changed; check deltas with resolved records, records that existed last time and now fail, and records created since; records missing since last time (merged or deleted); duplicate groups with a member created since the previous as-of, by creation source; groups still open. Without it, one line says there is no trend.
- calibration-sample.csv: pair_id, band, object, record_id_a, record_id_b, rule_id, basis, score, reviewer_mark, notes. Pairs come in a stable order seeded by `--as-of`. A rerun into the same folder keeps the reviewer_mark and notes already typed for every pair it still samples, so after tightening one rule only the new pairs need marks. Edit it in a text editor; a spreadsheet can damage the IDs (X02).

## Reading the scorecard

- Each line: check ID and name, object (and field), numerator of denominator, population, threshold, detail, the first record IDs.
- A D line counts the records in pairs where that rule fired. "Rule fired in" names the groups and pairs it touched, whatever band they ended in: D04 shows a certain group when a name match sits inside a D01 pair. The Duplicates section of summary.md gives each object's totals once.
- R01 lists dormant records even when their share is under the WARN threshold.
- X06 prints as WARN with NOT RETRIEVED; every check that needs the column is NOT CHECKED.
- D08 is INFO for shared phones and blocks too large to compare, and WARN when one unique key or email value sits on more records than `oversized_block`: that key is not unique. D01 or D02 then shows WARN too.

## Exit codes, determinism and privacy

- Exit 0: no FAIL. Exit 1: at least one FAIL. Exit 2: unreadable input, a file with a header and no records, no record ID column, or invalid arguments.
- The same inputs, rules and `--as-of` give the same files: rows are sorted, calibration uses a seed from `--as-of`, and no timestamp other than as-of is written. The one deliberate carry-over is the reviewer marks in calibration-sample.csv.
- stdout prints counts, statuses and record IDs only. Masking keeps the first character of each part (j***@e***.org, a*** m***). Phone numbers never appear in clear.
- Any output cell starting with =, +, - or @ gets a leading apostrophe, so a spreadsheet does not run it.

## Performance and limits

- Measured with synthetic data on a laptop on 2026-11-01: about 50 seconds and 0.8 GB of memory for 103,000 contacts. Time grows with the number of records that share a name inside one company domain.
- Blocks over 50 records (`oversized_block`) are not compared and are reported under D08.
- Above about 300,000 rows per object, or when memory is short, profile in the warehouse instead (identity-resolution-and-customer-view if installed) and say so in the baseline.
- Presets are guesses for English HubSpot headers, Salesforce API names and Attio attribute titles. Pipedrive, Zoho and Dynamics have none: use `--map`.
- Registrable domains use a short list of two-level suffixes, not the full Public Suffix List.
- Name similarity is Jaro-Winkler for Latin script only; names in other scripts match on exact equality.

## Canonical fields for --map

- Every object: record_id, created_at, updated_at, owner, source, created_by, last_activity_at, last_engagement_at. A preset is a guess: the HubSpot deal preset reads is_closed from "Is Deal Closed?" or "Is Closed", and derives it from "Is Closed Won" and "Is closed lost" when neither is exported; check the mapping report against your export.
- Contacts and leads: email, additional_emails, first_name, last_name, full_name, phone, mobile, title, company_name, associated_company_ids, associated_deal_ids, lifecycle_stage, lead_status, marketing_status, opted_out, do_not_call, hard_bounced, bounced_at, bounce_reason, last_form_submission_at, last_click_at, send_count, country, state, postal_code, is_converted, converted_contact_id.
- Companies: name, domain, additional_domains, website, phone, country, city, state, postal_code, industry, lifecycle_stage, parent_id, associated_contact_ids, associated_deal_ids.
- Deals: name, amount, currency, close_date, stage, pipeline, is_closed, is_won, is_lost, deal_type, lost_reason, associated_company_ids, associated_contact_ids, next_step.
- Users: user_id, name, email, active, kind. Contact roles: opportunity_id, contact_id. Queues: queue_id, name.

## After changing the script

Run `--self-test`, then the worked example, the December rerun and the five regression commands in [worked-example.md](worked-example.md), and compare the totals and each case's outcome. If a seeded case stops producing its documented status, fix the script or the seeded row, not the expectation.