Rate limiting is a capacity contract, not an exceptional edge case. A resilient integration accepts work without losing it, schedules outbound calls within the provider's documented rules, and slows down when the provider signals pressure. The queue protects intake; idempotency protects side effects; retry policy and observability determine whether the backlog recovers or becomes a retry storm.
Model limits at the scope the provider enforces
Record limits by endpoint, tenant, credential, region, and time window when the documentation distinguishes them. Capture response headers that report remaining capacity or reset time and honor `Retry-After` when supplied. Do not copy a limit from another plan or account. Keep the configured rate below the published ceiling until real traffic shows how bursts, pagination, and concurrent workers behave.
Decouple durable intake from outbound execution
Authenticate and validate the inbound envelope, persist it with a stable event ID, enqueue a reference, and return the acknowledgement required by the source provider. Workers should retrieve the durable record and perform one bounded destination operation. If the queue publish fails, the receiver must not report success unless another durable recovery mechanism exists.
A queue absorbs demand; it does not create downstream capacity. Admission control and worker pacing still decide whether the backlog converges.
Pace workers with shared, tenant-aware state
A token bucket or scheduled dispatcher can smooth calls, but the limiter must be shared by every worker using the same quota. Partition fairly so one noisy tenant cannot starve others. Cap concurrency as well as requests per interval, because long-running calls can exceed connection or provider capacity even when request counts appear compliant. Reduce the rate when throttling rises and recover gradually.
Retry only failures likely to recover
Retry throttling, timeouts, and documented transient server failures with exponential backoff, jitter, and a maximum attempt or elapsed-time budget. Do not retry validation, authentication, permission, or schema errors without a state change. Implement retry in one layer; nested SDK, worker, and workflow retries can multiply requests unexpectedly. Mutating calls must be idempotent before automatic retry is enabled.
Design the dead-letter queue as an operating process
Choose the redrive threshold from provider behavior and recovery objectives rather than a universal attempt count. Store event ID, tenant, destination operation, sanitized error, attempt history, first and last failure time, and code version. Alert when messages arrive or age beyond policy. Before redrive, classify the cause, repair configuration or data, and replay a small canary at a controlled velocity.
Batch only when semantics and APIs support it
Use a documented bulk endpoint when it preserves the required per-record result and idempotency behavior. Bound batches by the provider's current item and payload limits, isolate tenant data, and record which items succeeded. Split and retry only failed or transient records when the API supports partial results. Do not delay urgent work merely to fill a batch, and do not invent a batch wrapper around an endpoint designed for single writes.
Prove recovery under controlled load
Test sustained demand above the configured rate, sudden bursts, quota changes, 429 responses with and without retry guidance, timeouts after an unknown write result, worker crashes, duplicate delivery, poison messages, DLQ redrive, and one tenant monopolizing traffic. Monitor queue depth and age, dispatch rate, throttle rate, retry amplification, success latency, DLQ inflow, and reconciliation mismatches. Capacity is healthy only when the oldest work returns toward the target after the burst ends.