# Delivery reliability: idempotency, ordering, retries and receivers A delivered event is not a completed business action. Design receipt, processing and effect as three separate steps, each with its own evidence. Vendor contracts at the end carry the date each was read; confirm them live before quoting a number. ## Two keys, one ledger (IP08) - Delivery key: the sender's event ID. It deduplicates repeated deliveries of the same event. - Business effect key: flow ID + source object ID + business version (for example `billing_to_crm + 0001006 + revision 18`). It deduplicates the effect, whichever delivery or replay caused it. - A second event for the same change (Stripe can generate two Event objects for one occurrence) refetches the same state, computes the same version and is skipped as a no-op. Never put a unique constraint on the source object ID plus event type: events such as `customer.subscription.updated` and `invoice.payment_failed` repeat for one object, and the constraint would drop every later one. - Never build either key from attempt or arrival data: attempt numbers, delivery IDs that change per retry, receipt time, `now`, random values or a fresh UUID per attempt (IP02). Each retry would look new. Action ledger columns: - `episode_key` (unique), `flow_id`, `source_event_ids` - `state`: approved, submitting, acknowledged, needs_reconciliation, done - `destination_ref`, `first_seen`, `last_attempt`, `attempts` Rules: - IP05: claim the effect with a unique constraint or an atomic insert before the side effect. Read-then-create races: two workers both read "not found" and both create. - IP06: no increments as effects. Set values, or count unique events; "add 1 to the counter" is not idempotent. - IP03 (lint): ledger retention at least the longest of the source's retry window, its manual resend window, the iPaaS replay window and the backfill overlap. Examples: Stripe automatic retries up to 3 days, Dashboard resend 15 days, CLI resend 30 days; Zapier replay within 60 days; HubSpot app webhooks retry over 24 hours and the webhooks journal keeps 3 days. ## Unknown outcomes (IP04) A timeout or dropped connection after sending a create is an unknown outcome: the destination may have accepted it. Before retrying: - Read the destination by the stable key, or - Use the vendor's idempotency mechanism: Stripe `Idempotency-Key`, Salesforce upsert by external ID with `updateOnly` where creation is not allowed, an ERP external ID upsert. Stripe v1 idempotency, for reference: - The first response after execution begins is saved and replayed, including 500 errors. - Keys up to 255 characters, with no personal data in them; parameters are compared and a mismatch errors. - Keys can be pruned after 24 hours; reuse after that creates a new request. Validation failures and concurrent conflicts are not saved. POST only. - Stripe API v2 retries complete failed or partly failed work instead of replaying the first response. The IETF Idempotency-Key header draft (draft-ietf-httpapi-idempotency-key-header-08, 22 October 2026) describes a structured string header, unique per request payload, with 400 when missing, 422 when reused with a different payload and 409 while the first request is still processing. Resources publish their own expiry policy. ## Vendor keys have windows (IP07) - Stripe v1 idempotency keys: prunable after 24 hours. - Segment deduplicates on `messageId` (not payload) with at least 24 hours of IDs stored. - Meta deduplicates pixel and Conversions API events on `event_id` plus `event_name` within 48 hours of the first event. - Klaviyo deduplicates events per profile, metric and `unique_id`; the first processed event wins. The default identity truncates time to the second, so same-second events can collide. - Google Ads click conversion uploads use `order_id` per conversion action. - Hightouch event syncs need a unique event key or events are omitted; Fivetran Activations needs a sync key unique on both sides. A replay older than the vendor's window needs the team's own ledger. ## Ordering - OR01: arrival order is not business order. For current state, refetch the authoritative object, or compare versions and drop stale writes. Stripe, HubSpot and Braze all state no ordering guarantee. - OR02: serialise writes per account or per object (queue partition, advisory lock). - OR03: store occurrence time and receipt time separately. - Stripe's `created` is in seconds, so several events share a timestamp; it cannot order them. ## Error classes (RT03) Classify by status and body, not status alone: - Transient: 5xx, timeouts, connection resets. Retry with backoff. - Quota: 429. Salesforce quota exhaustion and its concurrency limit both arrive as 403 `REQUEST_LIMIT_EXCEEDED`. Stripe returns 429 with code `lock_timeout` for object contention, which is not a quota problem. - Permanent: 400, 404, validation errors. Never retry (RT06); send to the exception queue. - Unknown outcome: timeout after send. Reconcile first (IP04). - Hidden failure: Marketo returns HTTP 200 with `success` false and an errors array; GA4 Measurement Protocol returns no HTTP error even for malformed events; batch APIs return per-record failures inside a successful envelope; Salesforce Bulk API 2.0 splits results into successful, failed and unprocessed rows. ## Backoff (RT04) - Exponential with full jitter: `sleep = random(0, min(cap, base x 2^attempt))`. - Default for custom code: base 1 s, factor 2, cap 60 s, 6 attempts, then the exception queue. - Scheduled re-drive from the exception queue at 15 minutes, 1 hour and 6 hours. This is a separate, bounded schedule. - A local token bucket for retries keeps one failing dependency from flooding the others. ## One retry layer (RT01, lint) - The product of nested synchronous attempts (SDK, HTTP client, iPaaS step, platform autoreplay) must be 10 or below. Three attempts at five layers is 243 times the load, the AWS Builders' Library example. - Turn off retries in all but one layer. For example, set Stripe SDK automatic retries to 0 when the worker retries. - Written in the inventory as `retry_layers`, for example `sdk:3;worker:5` (product 15, FAIL). ## Retry-After (RT05) - RFC 9110: an HTTP date or a number of seconds. RFC 6585: a 429 may carry it. - Klaviyo returns it in seconds. - HubSpot workflow webhook actions honour the Retry-After your receiver returns on a 429 and record it in milliseconds: `Retry-After: 60` means 60 ms to HubSpot. A receiver behind a HubSpot workflow sends milliseconds. - Log the header value, the unit applied and the wait taken, so a unit error (1,000 times too long or too short) shows up. ## Never silence a sender (RT07) - Never return 2xx to stop retries or to avoid a ban. Pipedrive bans a webhook for 30 minutes after 10 first-attempt failures and loses events during the ban; the fix is a fast, durable receiver, not a fake success. ## Receivers (WH rules) - WH02: verify the signature on the raw bytes with the pinned scheme. Frameworks that parse and re-serialise JSON break signatures. Never accept a weaker legacy scheme when the pinned one is missing. - WH04: after the signature, validate the account, portal or tenant; never let a payload field choose whose credentials or partition to use. - WH03: commit the event durably, then return 2xx, then process asynchronously. - WH01 (lint): acknowledgement p99 at or under half the sender's timeout (HubSpot app webhooks 5 s, Pipedrive 10 s, Klaviyo system webhooks 5 s). - WH05: acknowledge and store unknown event types; do not route them into an error branch that returns 5xx (the sender retries, then disables the subscription). Reject malformed payloads per the sender's contract. - Use constant-time comparison for signatures. ## Publishing webhooks (WH06) Follow Standard Webhooks (Apache-2.0) when the team sends webhooks: - Headers `webhook-id`, `webhook-timestamp` and `webhook-signature` (v1 HMAC, v1a ed25519). - Secrets 24 to 64 bytes with a `whsec_` prefix; signature lists allow rotation without downtime. - Retries over days with exponential backoff and jitter; 2xx is success; 410 Gone disables the endpoint. - Timeouts of 15 to 30 seconds; payloads under about 20 kB. ## Vendor delivery contracts HubSpot app webhook subscriptions (developers.hubspot.com webhooks guide, 2026-11-01): - Up to 10 retries over 24 hours for connection failures, timeouts and any 4xx or 5xx. - The receiver must answer a batch within 5 seconds; a request carries up to 100 events. - No ordering guarantee; duplicates possible; the classic `eventId` is not guaranteed unique; use `occurredAt` for timing. - At most 10 requests in flight; `maxConcurrentRequests` must be above 5. - Settings can be cached for up to 5 minutes after a change; at most 1,000 subscriptions per app. - Webhook POSTs do not count against the app's API rate limits. HubSpot request signatures (request validation page, 2026-11-01): - v3: `X-HubSpot-Signature-v3` and `X-HubSpot-Request-Timestamp`; HMAC SHA-256 over method, decoded URI, body and timestamp with the app's client secret, Base64; reject timestamps older than 5 minutes. - v2 (workflow webhook actions, app card data): SHA-256 hex over secret, method, URI and body, with no timestamp. v1: secret plus body. HubSpot workflow webhook action (knowledge base, checked 2026-11-02): - Data Hub Professional or Enterprise; POST or GET over HTTPS; request signature or API key authentication. - Retries for up to three days, starting one minute after failure, with gaps up to eight hours. 4xx responses are not retried except 429, where HubSpot honours the Retry-After your receiver returns and reads it as milliseconds. - Optional action rate limiting pauses later actions in the workflow too. HubSpot webhooks journal (webhooks journal guide, checked 2026-11-02): - Subscription-related changes for the past three days, read through API calls that return journal file URLs, which expire. - Install-specific subscriptions; reading needs `developer.webhooks_journal.read`, managing subscriptions `developer.webhooks_journal.subscriptions.write`, plus the object's read scope. - The journal API allows 100 requests per second per app (subscriptions 50, snapshots 10); a 429 carries Retry-After. Stripe webhooks (docs.stripe.com/webhooks, checked 2026-11-06): - Automatic retries for up to three days with exponential backoff in live mode; a sandbox retries three times over a few hours; a 3xx counts as failure. - Manual resend: Dashboard up to 15 days after the event, CLI `stripe events resend` up to 30 days; a resend does not cancel automatic retries. - No ordering guarantee; fetch missing objects from the API. - Duplicates: log processed event IDs. Stripe can generate two Event objects for one occurrence and says to identify the pair by `data.object.id` plus `event.type`; use that pair only to flag duplicates that arrive close together, or as a key for once-per-object types (created, deleted, `invoice.paid`). Repeating types rely on refetch plus version compare. - `created` is in seconds, so distinct events can share a timestamp; Stripe says not to use it for ordering or for deciding whether an event was processed. - `Stripe-Signature` carries `t=` and `v1` signatures; ignore other schemes; libraries default to a 5-minute tolerance; never use 0; each retry gets a new timestamp and signature; secrets can roll with up to 24 hours of overlap. - Return 2xx quickly and process through a queue; up to 16 event destinations per account; thin events recommended for new integrations. Pipedrive webhooks v2 (pipedrive.readme.io, 2026-11-06): - Non-2xx or no response within 10 seconds is a failure; three more attempts follow at 3, 30 and 150 seconds. - Ten first-attempt failures ban the webhook for 30 minutes; events during the ban are lost; retries do not raise the ban counter. - A webhook with no successful delivery for 3 consecutive days is deleted; at most 40 webhooks per user; outgoing webhooks are outside the API rate limit. - Meta fields include `id` (the triggering event), `change_source` (app or api), `attempt`, `webhook_owner_id`, `merged_to_id` and `merged_from_id`; optional basic auth. Klaviyo (webhook pages on developers.klaviyo.com, checked 2026-11-02): - Flow webhooks: 429, 500, 502 and 503 are retried up to 17 times over 24 hours with exponential backoff, shown as Waiting to Retry; any other failure code moves the call to Skipped. - System webhooks need 200, 201 or 202 within 5 seconds; failed requests retry with exponential backoff up to a 1-hour delay; after more than 48 hours in error the webhook is disabled, and re-enabling sends only events processed after that. Recover the gap with Get Events filtered by metric. - System webhook requests carry `Klaviyo-Signature`, an HMAC SHA-256 keyed with the webhook's secret over the body and the `Klaviyo-Timestamp` value. Segment (twilio.com/docs/segment, 2026-11-01): - Deduplicates on `messageId` with at least 24 hours of IDs and a 99% no-duplicate guarantee across sources; warehouses remove later duplicates by `messageId`. - Failed cloud-mode destination calls retry for four hours with randomised exponential backoff; Business tier customers can request a Replay from archived logs. Make (help.make.com, 2026-11-01): - Store incomplete executions is off by default. When on, RateLimitError, ConnectionError and ModuleTimeoutError retry automatically at 1, 11, 21, 51, 81 and 111 minutes, then about 4 hours 51 and 7 hours 51 after the original run. The Retry handler defaults to 3 attempts 15 minutes apart; at most 3 incomplete-execution retries run in parallel per scenario. - No incomplete execution is created for an error on the first module (unless a Retry handler is attached), when storage is full (data loss disabled turns the scenario off; enabled discards the data), past the plan's duration limit, or in the initialization or rollback phase. - Skip drops the bundle and marks the run successful; Rollback reverts only transactional modules; Process data in order makes runs sequential and stops new runs while incomplete executions wait. - Instant webhooks run in parallel by default, up to 300 incoming requests per 10 seconds, then 429; a webhook queue holds up to 667 items per 10,000 monthly credits, at most 10,000; a full queue answers 400; webhooks unattached for more than 5 days answer 410 Gone; a webhook-triggered scenario deactivates on its first error; a mid-scenario webhook response hides later errors. Zapier (help.zapier.com and docs.zapier.com, 2026-11-01): - Replay from Zap history re-runs only errored steps and never re-runs Filter or Paths; replaying an entire run starts at the trigger, uses the current published Zap and counts tasks again. - Autoreplay (paid plans) retries a failed step up to 5 times, at about 5 minutes, 35 minutes, 1 hour 35, 4 hours 35 and 10 hours 35 after the first error; error emails wait for the final failure; runs on hold are not autoreplayed. - Replays must happen within 60 days of the trigger event. n8n (docs.n8n.io and the official n8n skills, 2026-11-01): - Retry On Fail retries on any error, with no per-status filter; `maxTries` is capped at 5 and `waitBetweenTries` at 5,000 ms, fixed waits. - On Error options: Stop Workflow (default), Continue (passes the last valid data on), Continue (using error output). - Error workflows start with the Error Trigger node and are set per workflow under Settings; Stop And Error forces a failure. Braze `/users/track` (braze.com docs, 2026-11-06): - 3,000 requests per 3 seconds on data-point contracts (MAU contracts add an hourly limit); up to 75 objects per request across attributes, events and purchases; asynchronous with no ordering guarantee; every custom attribute sent consumes a data point, so send changes only. GA4 Measurement Protocol, Meta and Google Ads (vendor docs, 2026-11-01): - GA4 MP returns no HTTP error codes even for malformed events; the validation server returns `validationMessages` and its events never reach reports. - Meta deduplicates pixel and Conversions API events on `event_id` plus `event_name` within 48 hours. - Google Ads click conversion uploads need `partial_failure` true, support `validate_only`, and use `order_id` per conversion action to prevent duplicates. Since 15 June 2026 UploadClickConversions fails with `CUSTOMER_NOT_ALLOWLISTED_FOR_THIS_FEATURE` for developer tokens that sent no offline conversion uploads between 17 December 2025 and 15 June 2026; Google points to the Data Manager API (upload page, checked 2026-11-06). ## Rules checked by the linter - IP01 FAIL: `creates_records` yes or `side_effect` not none, and `idempotency_key` blank. - IP02: `idempotency_key` matching attempt, delivery, received, arrival, now, random, retry or uuid4 is FAIL. WARN when the flow creates records or has a side effect, its method is not event_pipeline, and every `+`-separated part of the key names a delivery key (event_id, message_id, webhook_id or delivery_id, with or without the underscore; a part naming the flow itself is ignored). Key effects on flow + source object + business version. - IP03 FAIL: `ledger_retention_days` below `max_replay_window_days`. WARN when either is blank on a flow that creates records or has a side effect. - RT01: product of nested attempts above 10 is FAIL; more than one layer above 1 attempt while the product is 10 or less is WARN; a blank `retry_layers` is WARN, because blank means unknown (write `none` for no retries). - RT02 WARN: `retry_after_unit` blank for method custom or ipaas. - WH01: `inbound_webhook` yes and `ack_p99_seconds` above half of `sender_timeout_seconds` is FAIL; either value blank is WARN. Applied by hand: IP04 to IP08, RT03 to RT07, WH02 to WH06, OR01 to OR03.