Retry Storm Prevention: Backoff, Jitter and Retry Budgets

Retry storm prevention became a live topic again this autumn. GitHub’s report on its August 17, 2026 incident says a latent client retry bug sharply amplified traffic to one internal authentication endpoint. Uber then published how it stops retries from multiplying across deep call chains. Both describe the same trap. Retries feel safe because they usually work.

The trap is arithmetic, not bad luck. One retry per hop doubles load at every level of a call chain. This guide shows the naive code, the failure maths, a small working retry helper with a budget, how to test it, and when not to use it. Code targets Node.js 22 with ES modules and no dependencies.

Why do retries turn a small failure into an outage?

Retries add work to a service at the moment it has the least spare capacity. Marc Brooker of AWS calls retries “selfish”: they spend more server capacity to raise one caller’s chance of success. That trade is fine for rare faults. It is harmful during overload.

The multiplication is the real danger. Uber’s analysis shows that with one retry per hop, a failing node at depth d receives roughly 2^d times the baseline load. AWS gives the same warning for stacked layers: retrying at every layer can raise database load by 243x across five layers, which may prevent recovery.

A hypothetical example

Consider a hypothetical checkout flow. The web tier calls an order service, which calls a payment service, which calls a ledger database. Assume each layer makes up to three attempts. If the ledger fails every call, one user click produces 3 x 3 x 3 = 27 ledger calls. At 1,000 clicks per second, the ledger sees 27,000 calls per second. This example is illustrative arithmetic, not measured data.

What did the GitHub incident show?

GitHub’s own report is specific about the retry role. A traffic peak pushed load balancers in one datacenter past their limits, and a service-mesh sidecar hit its concurrency limit. A latent client retry bug then amplified traffic to one authentication endpoint and slowed recovery. The report says clients were “not bound enough to keep a partial degradation from amplifying into a broader overload”.

GitHub’s response is the useful part. Engineers reduced gateway retries and blocked retry-triggering requests to the most affected endpoint, which produced broad recovery. They then ramped traffic back gradually. The remediation list includes reviewing retry and backoff limits across gateways and clients.

What does the naive retry code look like?

The naive version retries a fixed number of times with no delay or a fixed delay. It looks harmless in review.

// Naive: every failure retries immediately, at every layer that wraps it
async function callWithRetry(fn) {
  for (let i = 0; i < 5; i++) {
    try { return await fn(); } catch (e) { /* try again */ }
  }
  throw new Error('failed');
}

It retries every error, waits zero time, and has no budget, so a total outage means five calls per request. Nothing stops a caller above from wrapping it in another loop.

How does retry storm prevention combine backoff, jitter and budgets?

Each technique fixes a different problem, and none of them replaces the others. Backoff spaces retries out. Jitter stops clients from retrying in lockstep. A budget limits total volume.

1. Cap attempts. Google's SRE book recommends a per-request retry budget of up to three attempts. After that, return the error to the caller.
2. Add exponential backoff with full jitter. Full jitter sleeps a random time between zero and the capped exponential delay. In Brooker's simulations, jittered backoff substantially cut client work and server load compared with plain exponential backoff, which the article calls "the clear loser."
3. Add a retry budget. The SRE book describes a per-client limit that retries only while retries stay under 10% of requests. It estimates this cuts growth to about 1.1x in the general case, against just under 3x in the worst case without it.
4. Retry at one layer only. The lower layer should return an "overloaded, do not retry" signal, as the SRE book advises.
5. Retry only safe operations. Use idempotent calls or idempotency keys.

What does retry storm prevention look like in working code?

The helper below combines capped attempts, full jitter and a token-bucket budget. The budget follows gRPC's retry throttling design: each failure removes one token, each success adds a fraction, and retries pause once the count falls to half the maximum or below. We wrote and ran it on Node.js 22.

// retry.mjs - Node.js 22, ES modules, no dependencies
export class RetryBudget {
  // gRPC-style token bucket: failures spend 1 token, successes refill tokenRatio.
  // Retries pause while tokens <= maxTokens / 2.
  constructor({ maxTokens = 10, tokenRatio = 0.1 } = {}) {
    this.max = maxTokens;
    this.ratio = tokenRatio;
    this.tokens = maxTokens;
  }
  onSuccess() { this.tokens = Math.min(this.max, this.tokens + this.ratio); }
  onFailure() { this.tokens = Math.max(0, this.tokens - 1); }
  canRetry() { return this.tokens > this.max / 2; }
}

// Full jitter: sleep a random time in [0, min(cap, base * 2^attempt)].
export function fullJitterDelay(attempt, { baseMs = 100, capMs = 5000 } = {}, rand = Math.random) {
  return rand() * Math.min(capMs, baseMs * 2 ** attempt);
}

export async function withRetry(fn, {
  maxAttempts = 3, baseMs = 100, capMs = 5000, budget,
  isRetryable = () => true, sleep = (ms) => new Promise((r) => setTimeout(r, ms)),
  rand = Math.random,
} = {}) {
  for (let attempt = 0; ; attempt++) {
    try {
      const result = await fn(attempt);
      budget?.onSuccess();
      return result;
    } catch (err) {
      budget?.onFailure();
      const last = attempt + 1 >= maxAttempts;
      if (last || !isRetryable(err) || (budget && !budget.canRetry())) throw err;
      await sleep(fullJitterDelay(attempt, { baseMs, capMs }, rand));
    }
  }
}

Share one RetryBudget per downstream dependency, not one per request.

import { withRetry, RetryBudget } from './retry.mjs';

const paymentBudget = new RetryBudget({ maxTokens: 10, tokenRatio: 0.1 });

export function charge(order) {
  return withRetry(() => callPaymentApi(order), {
    maxAttempts: 3,
    baseMs: 100,
    capMs: 2000,
    budget: paymentBudget,
    isRetryable: (err) => err.code === 'ETIMEDOUT' || err.status === 503,
  });
}

Does jitter alone stop the storm?

No. We checked this with a small deterministic simulation. The assumptions are ours: 90 clients each send one request per tick, the server accepts 100 per tick, and for ten ticks its capacity drops to 20. Each request gets at most three attempts. The simulation is a teaching model, not a benchmark of any real system.

Simulated retry strategies during a ten-tick capacity drop (our model, fixed seed)

Strategy	Total attempts for 5,400 requests	Amplification	Peak attempts in one tick
No retries	5,400	1.00x	90
Fixed one-tick retry	7,256	1.34x	254
Full jitter	7,340	1.36x	257
Full jitter plus budget	5,771	1.07x	206

The result matches the theory. Jitter changed when retries arrived, not how many were sent, so total volume stayed near 1.36x. Only the budget cut volume, to 1.07x. Our one-tick granularity hides jitter's smoothing of peaks, so read the peak column with caution. Use both tools.

What trade-offs and failure modes should you plan for?

A budget protects the service at the cost of some user-visible failures. During a short fault, retries would have succeeded, and the budget may block some of them. That is the intended trade.

Other failure modes to plan for:

• Per-process budgets. A budget in each pod caps that pod, not the whole fleet.
• Hidden retry layers. SDKs, service meshes and gateways often retry by default. gRPC enables retries by default, though it ships with no default retry policy. Audit every layer before adding your own.
• Retrying non-idempotent writes. A retried charge without an idempotency key can duplicate the charge.

When should you not use client-side retries?

Skip retries when the operation is not idempotent and has no key, when errors are deterministic (validation, authorization, bad input), or when the caller has a tight deadline that a retry delay would break. Skip them too when the failure is overload rather than a transient fault. Retries assume errors are random and recoverable. In overload, bad hosts or sharding problems, errors often persist, so extra retries add load without improving success.

Also consider whether a circuit breaker or load shedding belongs in the design. AWS's guidance prefers token-bucket retry limiting to circuit breakers, which it says add hard-to-test modal behavior and slow recovery.

How did Uber go beyond retry budgets?

Uber found budgets insufficient for deep call chains with heavy fan-out. Its post says retry budgets reduce multiplication to roughly (1+B)^d, where B is the budget and d is depth, which still compounds. Uber's fix is error ownership. A service owns an error only if none of its outbound calls failed. Callers retry only errors that a service has claimed, and the first node that sees no claim stops the storm.

Uber reports a November 18, 2025 production incident where a service five levels deep failed. Simple retry budgets would have caused a 46%-135% traffic increase on the degraded service. Error ownership stopped retries at the immediate callers. Its maximum retry storm radius fell from as high as 25 to 3 across user-facing APIs.

How do you test retry storm prevention?

Test the failure path, not just the happy path. Our test file for the helper checks five properties using Node's built-in assert.

1. A flaky call succeeds after two failures and makes exactly three calls.
2. A permanently failing call stops after maxAttempts.
3. A non-retryable error (HTTP 400 in the test) makes exactly one call.
4. With a shared budget, ten permanently failing requests made 13 calls in total, not the 30 that three attempts each would allow.
5. Every jittered delay stays between zero and the capped exponential value.

Beyond unit tests, inject faults in staging and watch request volume to the failing dependency. If it rises well above baseline, a retry layer is leaking. Pair retries with server-side rate limiting and stampede protection on hot cache keys.

How do retry storm prevention techniques compare?

Retry-control techniques compared

Technique	What it limits	What it misses	Complexity
Fixed attempt cap	Attempts per request	Fleet-wide volume; multiplication across layers	Very low
Exponential backoff	Retry rate over time for one client	Synchronization between clients	Low
Full jitter	Synchronized retry waves	Total retry volume	Low
Token-bucket retry budget	Retries as a share of traffic	Cross-service multiplication in deep chains	Medium
Circuit breaker	All calls to a failing dependency	Partial failures; adds modes that are hard to test	Medium
Error ownership (Uber)	Retries to services that cannot fix the error	Needs mesh-level infrastructure	High

Interview questions this topic prepares you for

Why can retries make an outage worse?

Retries add load to a service that is already short of capacity. If each layer in a call chain retries, load multiplies per layer, so a deep dependency can receive many times the original traffic. The overload can then sustain itself after the trigger is gone. Strong answers mention capped attempts, jitter, budgets and single-layer retries.

Follow-up they may ask: How would you find hidden retry layers in an existing system?

What is the difference between jitter and a retry budget?

Jitter randomizes delays so failed clients do not retry together; it changes timing, not total volume. A retry budget limits how many retries clients may send relative to normal traffic; it changes volume. A production design usually needs both, plus an attempt cap and idempotent operations.

Follow-up they may ask: Where would you store a budget shared across many pods?

When is a retry unsafe?

A retry is unsafe when the operation has side effects and no idempotency key, because the first attempt may have succeeded before the timeout. It is also pointless for deterministic errors like validation failures. Safe retries need idempotent operations, a retryable error class and a bounded attempt count.

Follow-up they may ask: How do you make a payment call safe to retry?

• One retry per hop makes a failing node at depth d receive roughly 2^d times baseline load, according to Uber's analysis.
• Google's SRE book recommends up to three attempts per request and a per-client retry ratio under 10%.
• Jitter changes when retries arrive; only a retry budget reduces how many are sent.
• gRPC's retry throttling pauses retries when its token count falls to half the maximum.
• GitHub's August 17, 2026 report names a latent client retry bug as an amplifier of the overload.
• Retry only idempotent operations, and at only one layer of the stack.

Frequently Asked Questions

What is a retry storm?

A retry storm happens when clients retry failed requests so aggressively that the retries themselves overload the service. Each failure creates more traffic, which causes more failures. The system can stay overloaded even after the original fault is fixed, because the retry load alone keeps it saturated.

How does jitter prevent retry storms?

Jitter adds randomness to each retry delay so clients that failed together do not retry together. It breaks up synchronized waves of traffic. It does not reduce the total number of retries, so it works best combined with a retry limit or a retry budget that caps overall volume.

What is a retry budget?

A retry budget caps retries as a share of normal traffic instead of per request. Google's SRE book describes a per-client limit of 10% retries, plus up to three attempts per request. gRPC's retry throttling uses a token bucket for the same purpose, pausing retries when failures outnumber successes.

Should you retry at every layer of a microservice stack?

No. Retry at one layer, usually the one directly above the failing component. If three layers each make three attempts, one user request can produce 27 calls at the bottom of the stack. Lower layers should return an explicit do-not-retry signal when they are overloaded.

Which errors are safe to retry?

Retry timeouts, connection resets and explicit unavailable or overloaded responses, and only for idempotent operations or requests carrying an idempotency key. Do not retry validation errors or most 4xx responses, because repeating them cannot succeed. A retried non-idempotent write can create duplicate side effects.

Conclusion

Retry storm prevention comes down to bounding a tool that helps and harms in equal measure. Cap attempts, add jitter, share a budget and retry once per chain. Then test the failure path, because a retry bug stays hidden until the day it matters. For related patterns, read our guides on idempotency keys, rate limiter design and cache stampede prevention.

Sources

• GitHub Blog — GitHub availability report: August 2026
• Uber Engineering — How Uber Protects Against Retry Storms
• AWS Builders Library — Timeouts, retries, and backoff with jitter
• AWS Architecture Blog — Exponential Backoff And Jitter
• Google SRE Book — Handling Overload
• gRPC — Retry guide