# Failure modes Thirty-two failures seen in production martech integrations, each with how it happens, how to detect it, the rules that prevent it and the test case that exposes it ([test-plan.md](test-plan.md)). ## Receipt and acknowledgement Acknowledged before durable receipt: - How: the receiver returns 2xx, then crashes or the worker fails; the sender stops retrying and the event exists nowhere. - Detect: the sender's delivery log shows more deliveries than stored receipts for the same window; a kill test before the commit should return a non-2xx. - Rules: WH03. Tests: duplicate_delivery, unknown_outcome_retry. 2xx returned to dodge bans: - How: a receiver returns 200 for everything to avoid Pipedrive's ban; failed processing becomes invisible and unrecoverable. - Detect: receiver 2xx count above durable receipts; known bad payloads never reach the exception queue. - Rules: RT07. Test: malformed_payload. Unknown event types crash the receiver: - How: a new event type falls into an error branch and returns 5xx, so the sender retries and eventually disables the subscription. - Detect: 5xx responses grouped by event type; subscriptions disabled after a vendor release. - Rules: WH05. Test: unknown_event_type. Signature verified on re-serialised JSON, or downgraded: - How: the framework parses and re-serialises the body before verification, or the receiver accepts a weaker legacy scheme when the pinned one is missing. - Detect: valid sender requests failing verification; an altered-body test passing. - Rules: WH02. Test: signature_invalid. Payload chooses the tenant: - How: a valid signature, but a payload-supplied account or portal ID selects another customer's credentials or partition. - Detect: receipt and effect keys missing the account ID; a cross-account test writes to the wrong partition. - Rules: WH04. Test: cross_account_isolation. ## Keys and duplicates Delivery ID used as the business key: - How: HubSpot's classic `eventId` is not guaranteed unique and a delivery-attempt ID changes per retry, so legitimate events are dropped or duplicates pass. - Detect: distinct `eventId` count against distinct (objectId, propertyName, occurredAt); two effects for one episode key in the destination. - Rules: IP02, IP08. Test: duplicate_delivery. Two Stripe Event objects for one change: - How: Stripe can generate two separate Event objects for one occurrence; dedupe on event ID alone lets both through. The opposite fix also fails: a unique constraint on `data.object.id` plus `type` drops every later `customer.subscription.updated` or repeated `invoice.payment_failed` for the same object, so billing state goes stale. - Detect: received events grouped by `data.object.id` plus `type` with a count above 1 inside a short window; two effects for one business version; legitimate later updates marked as duplicates in the receipt log. - Rules: IP08 (the effect key, refetch and version compare make the second event a no-op). Test: duplicate_delivery. Delivery dedupe mistaken for effect idempotency: - How: events are deduplicated, but a worker timeout after the CRM created a task leads to a second create. - Detect: destination tasks or orders per episode key above 1; creates seconds apart after a timeout. - Rules: IP04, IP05. Test: unknown_outcome_retry. ## Retries Retry storm across layers: - How: SDK retries, iPaaS step retries and platform autoreplay multiply; three retries at five layers is 243 times the load in AWS's example. - Detect: attempts per operation in logs above the designed product; 429 spikes after an outage. - Rules: RT01. Test: rate_limited. Retry-After read in the wrong unit: - How: a client reads seconds as milliseconds or the reverse, or a receiver tells a HubSpot workflow `Retry-After: 60` meaning seconds. HubSpot records the value in milliseconds, so it retries almost at once, while RFC 9110 and Klaviyo use seconds. - Detect: logged wait times against the header value and the vendor's documented unit; retries seconds apart in the HubSpot workflow action log after a 429. - Rules: RT05, RT02. Test: rate_limited. ## Ordering Stale overwrite from out-of-order events: - How: events arrive out of order (Stripe, HubSpot and Braze state no ordering guarantee) and an older state overwrites a newer one. - Detect: destination value older than the source's current version after an out-of-order test; a revision field going backwards. - Rules: OR01, OR02. Test: out_of_order. ## Ownership and loops Global winner on a two-way sync: - How: a connector setting such as "Salesforce wins" applies to every field, so billing-owned or sales-owned values are overwritten by the other side. - Detect: property history alternating between two sources on one field; the field map shows two_way with owner_wins on a decision-driving field. - Rules: FO03, FO01. Tests: loop_check, non_owned_untouched. Integration loop: - How: a write triggers a webhook or workflow that triggers the same write; enrichment timestamps re-trigger enrichment. - Detect: one record updated more than 3 times an hour by the integration user; repeated no-op writes; Pipedrive `change_source` api repeating on the same entity. - Rules: FO rules, compare-before-write. Test: loop_check. Null overwrite from a failed join: - How: an upstream join fails and the model returns nulls; Hightouch sends nulls by default and RudderStack maps null to `#N/A` for Salesforce, clearing good values. - Detect: a spike of blank owned fields in the read-back against the previous run; NULL_OVERWRITE findings. - Rules: FO04. Test: blank_semantics. ## Identity Match on domain, name or email for accounts: - How: several companies share a domain, names collide, and emails change and move between people. - Detect: one destination record holding several source IDs; ID01 FAIL. - Rules: ID01. Test: ambiguous_match. Create on lookup failure: - How: a failed or ambiguous lookup silently creates a new record, building duplicates over time. - Detect: records created by the integration user with no crosswalk entry; duplicates by normalised key. - Rules: ID02, ID03. Tests: unmatched_record, ambiguous_match. Key mangled by a spreadsheet or case folding: - How: leading zeros stripped, IDs lowercased, Salesforce 15-character IDs joined case-insensitively, Segment Unify comparing identifiers case-insensitively. - Detect: KEY_MANGLED and CASE_COLLISION findings; the count of distinct keys changing after lowercasing. - Rules: normalisation rules in [contract-rules.md](contract-rules.md). Test: ambiguous_match. ## Removals Add-only sync for a current-membership list: - How: rows leaving the model are never removed, so ineligible people stay in audiences or lists. - Detect: the destination count only grows; ineligible members in the read-back; RM01. - Rules: RM01. Test: removal. Mass removal from an empty or broken source: - How: a mirror sync faithfully deletes everything when a broken join returns zero rows; Hightouch's volume stop covers only journey Trigger-mode syncs. - Detect: planned removals above 10% of the population or 1,000 records; source row count far below the trailing median. - Rules: the minimum-population guard. Test: removal. ## Hidden failures Partial batch failure hidden in a success: - How: batch APIs report per-record errors inside a 2xx; Marketo reports errors in HTTP 200 bodies; Salesforce Bulk results separate successful, failed and unprocessed rows. - Detect: per-record outcome counts that do not add up to the input; failed or unprocessed result files that are not empty. - Rules: RT03. Test: partial_batch_failure. Silent drop of malformed server events: - How: GA4 Measurement Protocol returns no HTTP error code even for malformed events. - Detect: validation server messages in test; DebugView or realtime counts against sent counts. - Rules: RT03. Test: malformed_payload. n8n "Continue" writes stale data: - How: On Error set to Continue (continueRegularOutput) passes the last valid data to the next node, so a failed lookup still feeds a create. - Detect: IX02 in the export; error items counted against writes in execution history. - Rules: IX02. Test: ambiguous_match. Salesforce rejects or blocks integration writes: - How: concurrent jobs or batches touch child records of one parent and fail with `UNABLE_TO_LOCK_ROW` (Salesforce describes it as a deadlock or timeout, worse when a summary field locks the parent); duplicate rules, which run on create, update and upsert, block the write (`DUPLICATES_DETECTED`); the integration user's field-level security hides a mapped field, so writes to it fail and reads leave it out. - Detect: failed result rows grouped by error code per run; a check before go-live that every mapped field is readable and editable by the integration identity, run as that identity. The two codes above were read on Salesforce's API status code list on 2026-11-06; confirm any other code there at run time, and match field-level security failures by the field name in the message, since the code differs by API and operation. - Rules: RT03, OR02 (group child records by parent and serialise per parent), OP03. Tests: partial_batch_failure, permission_check. Make stores nothing when it fails: - How: Store incomplete executions is off by default; a full store with data loss enabled discards failed data; errors on the first module create no incomplete execution unless a Retry handler is attached; Skip marks the run successful. - Detect: IX21, IX22 and IX25 in the blueprint; scenario history showing successes with missing destination records. - Rules: IX21, IX22, IX25. Test: unknown_outcome_retry. Incremental poll skips late or same-second changes: - How: a scheduled read (reverse ETL, iPaaS polling trigger, custom poller on last-modified time) uses a strict `>` cursor, so changes in the same second as the cursor are skipped, and so are records that reach the source's search index after the run. HubSpot says new or updated objects can take a few moments to appear in search results (CRM search guide, checked 2026-11-06). - Detect: a later full pull or read-back finds records modified inside an already processed window that were never written, or counts per modified-time window differ. - Rules: BF03, BF04. Test: late_change. ## Recovery Search-based recovery truncated: - How: HubSpot search caps a query at 10,000 results, excludes archived objects and can lag fresh changes. - Detect: a 400 when paging past 10,000 results, or a window that stops at exactly 10,000; source and destination totals that still differ after recovery. - Rules: RL05, BF04. Test: replay_no_duplicate. Backfill fires customer outreach: - How: bulk updates enrol records into workflows, sequences or journeys that trigger on create or update. - Detect: workflow enrolments or sends during the backfill window; tasks created for historical changes. - Rules: BF05. Test: backfill_no_outreach. Replay skips eligibility: - How: Zapier history replay re-runs only errored steps and never Filter or Paths; Make retries resume from the failed module; an unsubscribed contact gets the message. - Detect: messages sent after an unsubscribe inside the recovery window. - Rules: BF06. Tests: replay_no_duplicate, consent_change. ## Operations Silent subscription loss: - How: Pipedrive deletes a webhook after 3 consecutive days without success; Klaviyo disables a system webhook after 48 hours in error and sends only new events after re-enabling; Make answers 410 Gone for webhooks unattached for more than 5 days. - Detect: a daily subscription status check; the zero-volume alert against the trailing median. - Rules: OP06. Test: environment_check. Shared quota exhausted by a backfill: - How: HubSpot daily limits are per account across apps and Salesforce counts every app over a rolling 24 hours, so a backfill starves lead routing. - Detect: usage headers or the Limits resource near the cap; 429 or 403 `REQUEST_LIMIT_EXCEEDED` in other integrations. - Rules: RL01, RL03. Test: rate_limited. Credential tied to a person: - How: the integration authenticates as an employee or a personal iPaaS account; it breaks when they leave or their role changes, and its writes look like a human's. - Detect: `credential_owner` person; field history attributing integration writes to a named person. - Rules: OP03. Test: credential_revoked. API version or contract drift: - How: dated API versions, Stripe account versions and deprecations change payload shapes or access; Google Ads UploadClickConversions has failed since 15 June 2026 for developer tokens with no offline conversion uploads between 17 December 2025 and that date, and most new Customer Match integrations must use the Data Manager API since 1 April 2026. - Detect: unpinned versions in the spec; new errors after a vendor release; gaps in the deprecation calendar. - Rules: OP08, OP04. Test: schema_change.