# Production failure modes and how to detect them

Twenty-seven failure modes seen in live CRMs, the rules that catch each one, and the safe direction for the repair. Safe repair always starts by stopping or narrowing the writer, then restoring values from history inside the vendor's restore window, then fixing the reports that read the damaged values. Restoring values while the writer still runs is temporary.

## Identity and duplicates

- FM-01 An integration creates duplicate companies or accounts (OBJ-02, OBJ-05).
  - Why: its create path matches on name or domain, or does not search a unique key first. HubSpot does not deduplicate API-created companies by domain.
  - Detect: companies sharing a primary domain whose record source is that integration, rising after each sync run; Salesforce accounts created by the integration user with matching websites; Attio records created by an app actor.
  - Repair: rekey the integration on a unique source ID and switch it to search-then-update; send unmatched records to a review queue; hand the duplicates to a dedupe review, never a bulk merge from the audit.
- FM-02 Automatic company association on a keyed portal (OBJ-03).
  - Why: HubSpot creates a company for each unknown email domain and picks one company on shared domains.
  - Detect: companies with an empty source-ID property created by auto-association; contacts on shared domains attached to the wrong company.
  - Repair: turn the setting off before rekeying anything, then list the auto-created companies for review.
- FM-03 Mailbox sync floods the CRM with companies (OBJ-03, OBJ-05).
  - Why: Attio creates a company for each non-public email domain by default.
  - Detect: companies created by mailbox sync with one person and no deals.
  - Repair: change the creation setting (it does not remove existing records), then review the created set.
- FM-04 A merge breaks downstream IDs (OBJ-06).
  - Why: Attio merges always create a new record ID; HubSpot merges create a new ID unless the primary ID preservation beta is on; Salesforce deletes the merged records.
  - Detect: sync errors such as record not found after merges; warehouse joins keyed on CRM record IDs.
  - Repair: add a remap step after every merge, or rekey downstream systems on a source unique key. Merges cannot be undone.

## Field ownership

- FM-05 Two writers fight over one field (FLD-02, INT-06, AUTO-09).
  - Why: owner rotation against a Salesforce or enrichment owner sync; billing against customer success; enrichment against manual edits.
  - Detect: `writer_history.py` flip-flops; property history sources alternating.
  - Repair: one writer per field, written precedence, the losing writer's mapping removed or set to fill-empty; restore values for the flip-flop records from history.
- FM-06 Opt-in or consent overwritten by an app (FLD-03).
  - Why: an event or webinar app writes opt-in on attendance; an import or reverse ETL writes consent fields.
  - Detect: consent history shows integration writes after a preference-centre change; opted-out contacts flipped to opted-in.
  - Repair: remove the mapping first, restore opt-outs from history, tell the privacy owner; expect opted-in counts to fall.
- FM-07 Near-duplicate fields split one meaning (FLD-04).
  - Why: each new integration adds its own field instead of using the register (plan, current_plan, stripe_plan).
  - Detect: clusters with different writers, and different reports reading different members.
  - Repair: pick the surviving field with its owner, repoint reports and workflows, then archive the rest inside the restore window.
- FM-08 A workflow silently stops after an option is removed (AUTO-04).
  - Why: filters reference option values a cleanup deleted or renamed.
  - Detect: workflow filter values not present in the property's options; zero enrolments since the option change in the audit log.
  - Repair: update the filter to current values, then backfill the records missed since the change, with approval.

## Lifecycle and pipeline

- FM-09 A copied form marks prospects as customers (LIFE-04, LIFE-06).
  - Why: a form copied from a customer portal keeps lifecycle stage Customer.
  - Detect: Customer records created by a public form with no closed-won deal or billing ID.
  - Repair: fix the form setting first, then correct the affected cohort by its form and date window; tell the reporting owner which conversion numbers will move.
- FM-10 Lifecycle repair by re-import does nothing (LIFE-05).
  - Why: HubSpot imports, forms, the API, workflows and the Salesforce integration only move lifecycle forward.
  - Detect: an import reports rows updated while the lifecycle value and its history are unchanged.
  - Repair: a manual-only correction workflow that clears then sets, with a reason property.
- FM-11 Required stage fields are assumed to guard every writer (PIPE-04).
  - Why: HubSpot documents conditional stage properties for manual UI moves only.
  - Detect: deals in a stage with required properties empty, placed by an integration, import or workflow.
  - Repair: validate in the writer, and keep a saved view per stage of incomplete deals.
- FM-12 A stage was recreated under an integration (PIPE-06).
  - Why: recreating or copying a stage changes its internal ID.
  - Detect: integration errors, or deals landing in a default stage.
  - Repair: map by internal ID, and run stage changes as migrations with the integration owner.
- FM-13 An ambiguous won stage (PIPE-03).
  - Why: a second 100% stage in a HubSpot pipeline (HubSpot treats every 100% stage as won), an open Salesforce stage at 100%, or a closed Salesforce stage in an open forecast category.
  - Detect: two 100% stages in a HubSpot pipeline; won counts that change with the stage filter; close dates set on the earlier stage.
  - Repair: set the stage below 100% so it is open, or merge it into the won stage; tell the reporting owner which won counts and close dates move, and restate the forecast with them.

## Access and credentials

- FM-14 Departed owners keep live work, or rotation leaves records unowned (ACC-03, AUTO-07).
  - Detect: open deals owned by inactive users, or by a departed person whose account is still active; unowned records in the routing population.
  - Repair: reassign by an agreed rule before deactivating; add a monitored fallback owner to rotation.
- FM-15 An integration runs on a person's account (ACC-02, INT-05).
  - Detect: the integration identity is a named user; a HubSpot private app created by a departed user and not rotated since; Salesforce OAuth tokens under a human user.
  - Repair: move the integration to a dedicated least-privilege identity, then revoke the personal credential.
- FM-16 Too many privileged accounts, agencies never removed (ACC-01).
  - Detect: the privileged list with account types; agency or shared logins.
  - Repair: record the before state (permissions are not versioned), then remove privilege with a named approver.
- FM-17 Access tested as an admin only (ACC-05, INT-02).
  - Why: the integration identity lacks field access, so values arrive blank with no error.
  - Detect: a mapped field blank in the destination; the Salesforce Field Access Summary for the integration user.
  - Repair: grant field access through a permission set on the integration user; retest as that user.

## Integrations and capacity

- FM-18 Two-way sync with a blanket winner (INT-02).
  - Why: different fields have different owners, but one direction setting covers them all.
  - Detect: field mappings plus history showing integration clears and older values returning.
  - Repair: set direction per field from the ownership register; define blank and clear behaviour.
- FM-19 Sync filters or inclusion segments treated as exclusion or security (INT-06).
  - Why: HubSpot data sync filters limit only the initial sync; a Salesforce inclusion segment is an operational filter.
  - Detect: excluded records present and updating in the other system.
  - Repair: control access through the integration user's permissions, not filters.
- FM-20 A backfill exhausts Salesforce API capacity (INT-03, INT-04).
  - Why: daily requests are measured over a rolling 24 hours and shared by every integration.
  - Detect: `DailyApiRequests` near the maximum; HubSpot Salesforce integration suspensions.
  - Repair: budget the job inside a reserve ([rules-access-integrations-reporting.md](rules-access-integrations-reporting.md)), run it off-peak, re-read limits between batches.
- FM-21 Automation layers fight on one object (AUTO-09).
  - Why: Salesforce Workflow Rules or Process Builder, record-triggered flows and Apex write the same field.
  - Detect: active items in several layers on one object writing one field.
  - Repair: consolidate into one record-triggered flow per object and timing, with Trigger Order set.

## Automation risk

- FM-22 Destructive or consent-changing automation with no policy (AUTO-06).
  - Why: a cleanup workflow deletes contacts or sets marketing status in bulk.
  - Detect: HubSpot action types `0-18224765`, `0-31`, `0-43347357`, and any unclassified action type the scripts list (an app action can delete or change consent too); Salesforce Delete Records elements.
  - Repair: replace deletion with a suppression list or flag; turning the workflow off does not restore deleted records.
- FM-23 Credits or send limits silently skip an essential step (AUTO-10).
  - Why: usage-based features pause when credits run out; sequence enrolment is skipped at the sender's send limit.
  - Detect: usage notifications; skipped actions in workflow history.
  - Repair: move the essential step to a non-metered action or add a fallback branch.
- FM-27 Automatic cleanup removes workflows (AUTO-08).
  - Why: HubSpot Enterprise account cleanup (beta) removes unused workflows automatically; restore lasts 90 days and returns an inactive copy.
  - Detect: the cleanup setting on with no owner; the deleted workflows list.
  - Repair: name an owner and review the candidate list before each cleanup run.

## Reporting

- FM-24 Report totals wrong by grain or report type (REP-02, REP-03).
  - Why: deal amount summed after a contact join; Salesforce "with" report types drop parents; unique counts on names.
  - Detect: reconciliation against a record-ID list.
  - Repair: rebuild the report at deal grain, or use a cross filter for "has related records" questions.
- FM-25 Currency mix in pipeline totals (REP-04).
  - Why: company currency changed after deals existed, or mixed currencies summed.
  - Detect: deal amounts unchanged after a currency switch; totals with mixed currency codes.
  - Repair: a documented conversion policy and a converted amount field; restate history with finance.
- FM-26 Unknown source dominates new records (REP-06).
  - Why: forms lose UTMs, imports carry no source, server-side handlers drop the tracking cookie.
  - Detect: the share of new contacts with an unknown or offline source.
  - Repair: fix capture at the form or handler; report imported contacts as their own group.