# Rate limits, budgets and backfills

Size every load against the endpoint's real allowance, leave room for everything else that shares it, and prove the pilot before the full run. Each vendor heading below gives the date its figures were read. Limits change: confirm on the source page and record the date in the spec.

## Formulas

- batches = ceil(records / batch)
- calls = batches x calls_per_batch + records x extra_calls_per_record + batches x readback_calls_per_batch
- total = ceil(calls x (1 + retry_overhead)), with a default retry overhead of 10%
- rate = min(limit / seconds) over every window that applies, x target utilisation (default 0.7)
- duration = total / rate
- available = remaining - reserve - other expected consumers
- records that fit now = floor(available / (total / records))

Count everything: discovery reads, pagination, association fetches, retries and the read-back. If destination IDs are not resolved yet, or records need different operations, the simple full-batch assumption no longer holds.

## Budget rules

- RL01: plan inside the available budget (remaining minus reserve minus other expected consumers). With no measured figure, reserve 20% of the daily allowance.
- RL02: plan at 70% or less of the tightest window limit shared by every app on that allowance. HubSpot's 10-second limit is per app, its daily limit per account; search is 5 requests per second per account.
- RL03: use the vendor's reset rule. HubSpot daily limits reset at midnight in the account's time zone; Salesforce counts a rolling 24 hours; Marketo resets at 12:00 AM CST; Stripe limits are per second plus a rolling 30-day read allocation.
- RL04: prefer batch and bulk endpoints. HubSpot batch operations take 100 records. Salesforce Composite Batch counts each subrequest, and Bulk API 2.0 jobs need a job-level call model.
- RL05: search-based recovery splits windows so each returns fewer than 9,000 results; HubSpot caps any query at 10,000 (paging past it returns a 400) and excludes archived records.
- RL06: keep error responses under 5% of daily requests (HubSpot's rule) and alert when 429s exceed 1% of calls in an hour.

## Vendor limits

HubSpot (usage guidelines and CRM search pages, 2026-11-06):

- Privately distributed apps: Free and Starter 100 requests per 10 seconds per app and 250,000 per day per account; Professional 190 and 625,000; Enterprise 190 and 1,000,000. The API limit increase raises the burst to 250 and adds 1,000,000 a day, at most two increases.
- Marketplace OAuth apps: 110 requests per 10 seconds per installing account, CRM search excluded; no daily figure listed.
- The 429 body's `policyName` says which limit was hit. Headers `X-HubSpot-RateLimit-Max`, `-Remaining`, `-Interval-Milliseconds`, `-Daily` and `-Daily-Remaining` show usage (daily headers are absent for OAuth; search responses carry none).
- CRM search: 5 requests per second per account, 200 results per page, 10,000 results per query, up to 5 filter groups with 6 filters each and 18 in total, archived records excluded, and new changes can take a moment to appear.
- Batch operations take 100 records. Batch upsert uses `idProperty` set to email or a unique value property; partial upserts are not supported with email.

Stripe (rate limits page, 2026-11-06):

- 100 requests per second in live mode, 25 in a sandbox; most endpoints 25 per second; Search 20 reads per second; 50 concurrent requests by default.
- A 429 carries `Stripe-Rate-Limited-Reason` (global-rate, endpoint-rate, global-concurrency, endpoint-concurrency, resource-specific); code `lock_timeout` means object contention.
- Read allocation: on average at most 500 read requests per transaction over a rolling 30 days, minimum 10,000 a month. Writes have no allocation.

Salesforce (limits quick reference, REST API guide, Bulk API 2.0 guide and the help article on rolling limits, checked 2026-11-06):

- The daily allocation is org-specific (edition, licences and add-ons), enforced on all calls to the org, not per user, and tracked over a rolling 24-hour window rather than reset at midnight. Calls from certain Salesforce connected apps, such as the mobile app, don't count.
- A new integration user adds nothing unless its licence carries API calls, so read the total from the Limits resource after any licence change.
- Past the allocation, calls can be blocked: REST returns 403 with `REQUEST_LIMIT_EXCEEDED`. Paid orgs can run a limited amount over before a hard cap. The same code covers the concurrency limit, so read the Limits resource before calling it a daily quota problem.
- The REST Limits resource returns `DailyApiRequests` Max and Remaining (needs View Setup and Configuration; accurate within five minutes). REST responses carry `Sforce-Limit-Info` with `api-usage=used/limit`.
- Composite Batch runs up to 25 subrequests and each counts against limits; Bulk API and Bulk API 2.0 calls count toward the allocation.
- Bulk 2.0 results split successful, failed and unprocessed rows; unprocessed rows were never attempted, failed rows were. Results are not recorded for batches beyond the daily batch allocation, so missing keys there are unknown, not failed.

Marketo (Adobe Experience League, 2026-11-06):

- 50,000 calls per day per subscription, reset at 12:00 AM CST; 100 calls per 20 seconds; 10 concurrent calls. Adobe's best-practices page says a daily allocation of at least 10,000, so read the instance's own figure under Admin, Web Services and pass it as `--daily-limit`.

Klaviyo (rate limits page and endpoint references, checked 2026-11-02):

- Create Event: 350 per second burst, 3,500 per minute steady. Bulk Create Events: 10 per second, 150 per minute, up to 1,000 events per request. Retry-After is an integer number of seconds. Private keys share the account quota; OAuth apps get a quota per installed app instance.

Braze (users/track page, 2026-11-06):

- 3,000 requests per 3 seconds for data-point contracts; MAU contracts add an hourly limit; up to 75 objects per request.

## Batch and bulk notes

- Salesforce Bulk API 2.0 job model: create the job, upload data, close it, poll, then download successful, failed and unprocessed results. Reconcile result counts to the input: they must add up.
- In Bulk CSV updates, empty cells are ignored and `#N/A` sets null.
- Partial batch failure: read per-record outcomes inside every 2xx batch response; send failed records to the exception queue and continue.

## Search caps and recovery windows

- A 400 when paging past 10,000 results, or a window that stops at exactly the cap (10,000 on HubSpot), means the window is truncated, not complete.
- Split by time so each window returns under 9,000 results, and overlap windows (BF04).
- Search excludes archived records and can lag fresh changes, so search-based recovery must be followed by a read-back.

## Backfill procedure

- BF01 Pilot first: 100 records or 1% of the population, whichever is smaller, and never fewer records than the edge cases in the test plan. Read back before the next batch.
- BF02 Parents before children: resolve and reconcile parent IDs (companies) before loading children (contacts, deals, line items).
- BF03 Checkpoint only after durable writes; persist a cursor only after its entries are stored.
- BF04 Every scheduled incremental read (reverse ETL, iPaaS polling trigger, custom poller on last-modified time) and every incremental recovery compares inclusively from the last durable cursor minus an overlap of at least 1 hour or twice the vendor's documented propagation delay, whichever is longer. Dedupe by key plus version, and advance the cursor only after durable writes (BF03). A strict `>` cursor skips same-second changes and records that reach a search index late.
- BF05 Pause or filter workflows, sequences and journeys that fire on create or update before any backfill or replay. Re-enable after the read-back passes.
- BF06 A replay re-checks eligibility at the action boundary. Zapier history replay never re-runs Filter or Paths steps, so eligibility must be enforced in the action step or the replay must run as a whole run.

Pilot composition: include every edge case the test plan names, for example several subscriptions on one account, a trial, a past-due account, a cancelled account, mixed currencies, no destination match and two destination matches.

## Worked budget 1: one record at a time against batches

40,000 known HubSpot contacts, then 3,000 changed records on a later day, on a Professional private app (625,000 a day):

- One record per call: 40,000 requests, 6.4% of the daily allowance.
- Batches of 100: 400 requests, 0.064%.
- The later 3,000 changes in batches of 100: 30 requests.
- Request counts compare budgets; they are not runtime guarantees. A 10-second ceiling says nothing about latency or bursts.

## Worked budget 2: Salesforce with other consumers

- Remaining 50,000; reserve 20,000; other consumers 12,000; 3 calls per record; 8,000 records requested.
- Available = 50,000 - 20,000 - 12,000 = 18,000. Needed = 8,000 x 3 = 24,000.
- Plan 6,000 records now and defer 2,000 to the next rolling period.

```bash
python3 scripts/size_sync.py --records 8000 --batch 1 --calls-per-batch 3 --retry-overhead 0 \
  --daily-limit 150000 --remaining 50000 --reserve-calls 20000 --other-calls 12000
```

Output ends with `FAIL RL01: 24,000 calls exceed the available 18,000; 6,000 records fit now (3 calls each), defer 2,000 to the next period after the reset`.

## size_sync.py options

- `--records N`: records in this load. `--batch` (default 1), `--calls-per-batch` (default 1), `--extra-calls-per-record` (lookups, associations), `--readback-calls-per-batch` (verification reads).
- `--retry-overhead` (default 0.10), `--target-utilization` (default 0.7).
- `--preset NAME` or `--window-limit N --window-seconds S` for custom limits; `--daily-limit` overrides the preset.
- `--remaining` (from usage headers or the Limits resource), `--reserve` (fraction, default 0.2 when a daily allowance is known) or `--reserve-calls`, `--other-calls`.
- `--window-hours H`: warn when the load takes longer.
- `--search-results N`: checks a search-based recovery against the preset's result cap and prints the windows needed.
- `--compare-single`: also sizes the writes one record per call.
- `--list-presets`, `--json`.
- Exit 1 on any FAIL, 2 on bad input. `--preset salesforce` exits 2 and tells you to read the Limits resource, because the allocation is org-specific.

Presets live in `assets/rate-limit-presets.json`. To add one, copy an entry and fill `vendor`, `scope`, `windows` (every window that applies, each with `limit` and `seconds`), `daily_limit` (or null), `daily_reset`, `batch_max`, `concurrency`, `result_cap`, `page_size`, `checked` (the date you read the page), `source` (the URL) and `notes`. The script prints the check date and source with every result.