Site icon NexGismo

Hedged Requests: Cut p99 Without Doubling Your Load

Hedged Requests: Cut p99 Without Doubling Your Load
TL;DR
  • Problem: One slow replica makes p99 latency far worse than the average, especially when a request fans out to many calls.
  • Pattern: Send a second copy to another replica after a delay. Use the first reply. Cap hedges with a budget and a breaker.
  • When to use: Idempotent reads with several healthy replicas and occasional stalls on one of them.
  • Trade-off: About 4-5% more calls in normal times. Without a cap, a fleet-wide slowdown can double load.

Hedged requests cut tail latency by sending a duplicate call to a different replica once the first call is slower than usual, typically the p95 delay. The first reply wins. Cap the extra calls with a budget and stop hedging when most calls are slow, or hedging amplifies an outage.

Hedged requests are a tail-latency technique. The client sends a second copy of a request to another replica after a short delay and accepts the first response. They trade a small amount of extra load for a much shorter latency tail.

Hedged requests are a fix for a problem that averages hide. On 9 November 2026, HubSpot published how it hedges reads on erasure-coded HDFS. On its case-study cluster, typical p99 reads took about 10 ms, yet reads as slow as 5 seconds occurred nearly every day. One slow DataNode caused more than 2,000 client errors in about half an hour. The fix was a hedge. The risk is that a hedge is also extra traffic, and extra traffic is the last thing a slow system needs. This post shows the pattern, the safeguards, working Node.js code, and numbers from a simulation we ran.

Why do hedged requests cut tail latency?

Hedged requests work because slow replies are rare and uncorrelated across replicas. A second copy sent elsewhere will usually finish before the stalled first one. The Tail at Scale paper by Dean and Barroso describes deferring the secondary request until the first has been outstanding longer than the 95th-percentile latency, which adds only about 5% more load.

The same paper reports a benchmark that reads 1,000 keys across 100 servers. A hedge sent after a 10 ms delay reduced the 99.9th-percentile latency from 1,800 ms to 74 ms, with about 2% more requests.

Fan-out is why this matters. If a request must collect replies from 100 servers and each is slow 1% of the time, the paper states that 63% of user requests will take more than one second. The arithmetic is 1 – 0.99100 = 0.634. The authors frame the cure as “latency tail-tolerant” design: build a predictable system from less predictable parts.

What requirements should a hedging layer meet?

Write the requirements first. The functional need is simple: return the first valid reply from any replica. The non-functional needs decide whether the design is safe.

  1. Bounded extra load: at most about 5% more calls in normal operation (our assumption, matching the p95 rule).
  2. No amplification: when many replicas are slow, hedging must switch itself off.
  3. Idempotent calls only: a hedge means the server may execute the call twice.
  4. Cancellation: abort the losing copy when the winner returns.
  5. Observability: track hedge rate, hedge win rate, and copies per request.

Why does naive hedging fail?

Naive hedging fails because the delay timer fires for almost every request when the whole fleet slows down. Each request then sends two copies, so offered load doubles at the worst moment. Queues grow, replies get slower, and more timers fire.

Consider a hypothetical service with ten replicas at 55% utilisation. Every replica becomes 1.7 times slower, perhaps after a bad deploy. The fleet now runs near 94% busy. Hedging every slow request pushes demand past 100% of capacity. The queue never drains.

We simulated exactly this. Results are in the table further down. With no cap, hedging turned a p99 of about 1.2 seconds into about 156 seconds. gRPC and Envoy both ship guards for this reason, covered below.

A second failure is semantic. A hedge on a non-idempotent call can charge a card twice. This is a design rule, not a tuning problem: hedge reads, or writes with an idempotency key.

How do you design a safe hedged request layer?

Use a delay, a budget, and a breaker, in that order. The data path for one call looks like this: client starts call A on replica 1, starts a timer, and waits. If A answers first, the timer is cancelled. If the timer fires, the client checks the breaker, then the budget, and only then sends call B to a different replica. The first reply wins and the other is aborted.

  1. Pick the delay from data. Use the observed p95 of that call, then revisit it weekly. HubSpot’s per-cluster threshold ranges from 250 ms to 1,500 ms.
  2. Send to a different replica. A hedge to the same slow node adds load and no benefit.
  3. Add a token-bucket budget. Each request earns 0.05 token. Each hedge costs one token. The cap is then 5% of requests.
  4. Add a breaker. If a large share of recent calls hit the delay, hedging is the wrong tool. Stop hedging for a while.
  5. Bound the whole chain with one deadline. gRPC does this: the call deadline covers the whole chain of hedged requests.
  6. Cancel the loser so it stops consuming capacity.

What does working code for hedged requests look like in Node.js?

The code below targets Node.js 22 as an ES module. It uses AbortController for cancellation and Promise.any for the race. The call function you pass in must be safe to run twice.

// Node.js 22 (ES modules). Hedged call with a budget and a breaker.
export class HedgeBudget {
  constructor({ ratio = 0.05, burst = 10 } = {}) { this.ratio = ratio; this.burst = burst; this.tokens = 0; }
  onRequest() { this.tokens = Math.min(this.tokens + this.ratio, this.burst); } // earn 0.05 token per request
  tryTake() { if (this.tokens >= 1) { this.tokens -= 1; return true; } return false; }
}

export class HedgeBreaker {
  constructor({ window = 200, minSamples = 100, maxSlowShare = 0.25 } = {}) {
    Object.assign(this, { window, minSamples, maxSlowShare }); this.samples = [];
  }
  record(wasSlow) { this.samples.push(wasSlow); if (this.samples.length > this.window) this.samples.shift(); }
  get open() {
    if (this.samples.length < this.minSamples) return false;
    return this.samples.filter(Boolean).length / this.samples.length > this.maxSlowShare;
  }
}

// call(signal, attempt) must be idempotent: it may run twice.
export async function hedged(call, { delayMs, budget, breaker }) {
  budget.onRequest();
  const ctrls = [new AbortController()];
  const first = call(ctrls[0].signal, 0);
  let timer;
  const hedgeTimer = new Promise((resolve) => { timer = setTimeout(resolve, delayMs, 'timer'); });
  const winner = await Promise.race([first.then((v) => ({ v })), hedgeTimer]);
  clearTimeout(timer);
  if (winner !== 'timer') { breaker.record(false); return winner.v; }
  breaker.record(true);
  const second = (!breaker.open && budget.tryTake()) ? (ctrls.push(new AbortController()), call(ctrls[1].signal, 1)) : null;
  const racers = [first, ...(second ? [second] : [])];
  const result = await Promise.any(racers);
  ctrls.forEach((c) => c.abort()); // cancel the loser
  return result;
}

We ran four tests on Node v22.23.4. A fast call sends one copy. A slow first call is beaten by the hedge in about 70 ms instead of 300 ms. An empty budget sends no hedge. An open breaker sends no hedge even when the budget has tokens. All four passed.

What did the simulation show?

We wrote a discrete-event simulation in Python 3.13 and ran it ourselves. The parameters are assumptions, not measurements of any real system: 10 replicas with one FIFO queue each, service time exponential with a 10 ms mean, 55% utilisation, and 1% of replies delayed by an extra 300 ms stall. The hedge delay was the baseline p95, which came out at 71.3 ms. The losing copy was not cancelled, which is the conservative case. Each row averages three seeds of 100,000 requests.

Simulated latency and load, hedged vs unhedged (milliseconds; our assumptions above)
Scenario Policy p50 p99 p99.9 Hedged Copies per request
Normal No hedge 15.7 219.0 348.7 0% 1.000
Normal Hedge, no budget 17.2 101.9 147.8 6.7% 1.067
Normal Hedge, 5% budget 16.4 103.3 303.4 4.0% 1.040
Normal Budget + breaker 16.5 105.6 302.7 4.0% 1.040
Fleet slowdown x1.7 No hedge 173.0 1,237.5 1,596.8 0% 1.000
Fleet slowdown x1.7 Hedge, no budget 77,443 155,894 159,519 99.9% 1.999
Fleet slowdown x1.7 Hedge, 5% budget 472.3 2,961.8 3,358.6 5.0% 1.050
Fleet slowdown x1.7 Budget + breaker 192.6 1,261.4 1,748.1 0% 1.000

Three findings stand out. First, in normal operation hedging cut p99 from 219 ms to about 102-106 ms for 4-7% extra calls. Second, an unbudgeted hedge collapsed the fleet. Third, a budget alone was not enough: during the slowdown it kept p99 at 2,962 ms, worse than the 1,238 ms without hedging, because even 5% extra load hurts at 94% utilisation. Only the breaker, which stopped hedging when about a quarter of recent calls hit the delay, matched the unhedged baseline.

The budget also costs something in normal times. At 4.0% hedged, the p99.9 was 303 ms against 148 ms for the uncapped hedge at 6.7%, because the budget ran dry on some stalls. That is the price of a hard cap.

How do gRPC, Envoy and HubSpot bound hedging?

Each of them limits hedging, but in different places. gRPC limits attempts and tokens. Envoy limits the share of active requests. HubSpot adds a per-server circuit breaker.

In gRPC, maxAttempts is required and values above 5 are capped at 5. With retryThrottling, hedges are sent only while the token count is greater than half of max tokens. A server can also send grpc-retry-pushback-ms, and a negative value tells the client not to retry. The gRPC page says nothing about idempotency, so check that yourself.

Envoy triggers hedging only from a per-try timeout, and the timed-out request is not cancelled. Its docs recommend retry budgets over a maximum-active-retries breaker.

HubSpot’s breaker trips per RegionServer. Its defaults are 2 impaired DataNodes in a 60,000 ms window, a 30,000 ms cooldown, and a hard cap of 200 hedges per second. In its data, hedges ran about once per million reads with a 90% win rate, and host-minutes with a read over one second fell from a median of 124 per day to 3, a 98% reduction.

How common hedging implementations limit extra load
Implementation Trigger Load limit Cancels loser?
Tail at Scale paper First request older than p95 About 5% extra by design Tied requests cancel on start
gRPC hedging hedgingDelay elapsed maxAttempts up to 5; token throttling Cancels outstanding on a fatal status
Envoy Per-try timeout Retry budget on the cluster No: late response still awaited
HubSpot HDFS Threshold 250-1,500 ms Breaker plus 200 hedges per second Not stated
Our Node.js sketch Fixed delay Token bucket plus slow-share breaker Yes, via AbortController

When should you not hedge?

Skip hedging when the call has side effects and no idempotency key. Skip it when every replica is already saturated, since the second copy only deepens the queue. Skip it when the backend is a single shared resource such as one primary database, because a hedge cannot reach a different copy.

Cost matters too. HubSpot notes each hedge on an erasure-coded read costs six extra reads from six DataNodes plus decode CPU, about six times the disk and network traffic of the original read. A hedge is cheap only when the duplicate is cheap.

The Tail at Scale authors also describe tied requests, where two servers each know about the other and one cancels when the other starts. That shrinks wasted work further, but it needs server cooperation.

How do you test hedging before production?

Test the failure modes, not the happy path. Inject a 300 ms delay on one replica and confirm p99 drops. Then slow all replicas and confirm that copies per request stays near 1.0 once the breaker opens. Alert on three metrics: hedge rate, hedge win rate, and copies per request. If the win rate falls well below the 90% HubSpot reports, the delay is probably too low. Related reading on NexGismo: how failed health checks turned a partial fault into a wide outage, and replay traps in durable execution, which also depends on safe duplicate execution.

Interview questions this topic prepares you for

How would you reduce p99 latency for a service that fans out to 100 backends?

Explain that per-call tail probability compounds: 1 – 0.99^100 is about 63%. Propose hedging idempotent reads after the p95 delay, to a different replica, with a budget near 5%, a breaker that disables hedging when many calls are slow, and cancellation of the loser. Mention measuring copies per request.

Follow-up they may ask: What happens if every replica is slow at once?

What is the difference between a retry and a hedge?

A retry starts after failure or timeout. A hedge starts while the first attempt is still running, and the first reply wins. Retries recover from errors. Hedges cut latency from slow successes. Both need budgets, because both add load when the backend is already struggling.

Follow-up they may ask: How do you make a hedged write safe?

How do you choose the hedge delay?

Start at the observed p95 of the specific call, then adjust from data. Too low wastes capacity on requests that would have finished. Too high leaves latency unrecovered. Watch the hedge win rate and the extra-load percentage, and revisit after traffic changes.

Follow-up they may ask: Should the delay be fixed or adaptive?

The NexGismo verdict: My recommendation: hedge only idempotent reads, and never ship the hedge without the breaker. In our simulation the budget alone still made the slowdown worse than doing nothing. The breaker is the part that makes hedging safe, and it takes about twenty lines of code. Treat the 5% budget as a ceiling, not a target.

Frequently Asked Questions

What are hedged requests?

Hedged requests send a second copy of a call to another replica when the first has not answered within a set delay, then use whichever reply arrives first. Dean and Barroso describe waiting until the first request is older than the 95th-percentile latency, which keeps the extra load near 5%.

Are hedged requests the same as retries?

No. A retry starts after the first attempt fails or times out and usually abandons it. A hedge runs alongside the first attempt, and the first reply wins. gRPC documents both as separate policies, and Envoy triggers hedging only from a per-try timeout.

Is it safe to hedge write requests?

Only if the write is idempotent. A hedge means the server may run the same call twice. Use an idempotency key so the second execution has no extra effect. Without one, hedge reads only. Never hedge a call that charges money or sends a message.

What hedging delay should I use?

Start near the observed p95 latency of the call, as the Tail at Scale paper suggests, then measure the hedge rate and win rate. HubSpot tunes a per-cluster threshold between 250 ms and 1,500 ms. A delay that is too low wastes capacity.

Can hedging make an outage worse?

Yes. In our simulation, unbudgeted hedging doubled the load during a fleet-wide slowdown and pushed p99 from about 1.2 seconds to over 155 seconds. A hedge budget plus a circuit breaker that stops hedging when most calls are slow avoided that collapse.

Conclusion

Hedged requests are one of the cheapest ways to shorten a latency tail. They also show how a good idea becomes an incident without guard rails. Start with one read path that has several replicas. Set the delay from its p95, add a 5% budget and a breaker, and watch copies per request for a week. Then widen the rollout. For more system design write-ups like this one, see our short link design guide, which applies the same check-the-failure-mode approach to identifiers.

Sources

Exit mobile version