Readiness Probe Dependencies: Lessons From Inngest’s Outages

Readiness probe dependencies are one of those settings nobody reviews until they cause an outage. The pattern looks harmless: connect to everything at boot, then flip a ready flag. It works for months. Then one auxiliary system starts timing out, and every pod that restarts gets stuck. Inngest’s July 2026 root cause analysis documents exactly this failure twice in one month, plus a feature-flag fallback that rerouted traffic onto a legacy path. The write-up is unusually candid about timelines and contributing factors, which makes it a good case study. This post walks through what happened, using only Inngest’s published account. It then turns each lesson into Node.js code you can apply to your own Kubernetes, ECS or serverless services. I also ran a small simulation of the startup failure in Node.js v22 so you can see the difference in timing, not just read about it.

What happened in Inngest’s July 2026 outages?

Inngest reported four incidents in July 2026, and the root causes split into three themes: feature-flag defaults, startup dependencies and configuration drift. According to the Inngest RCA, the incidents hit on July 10, July 16 (twice) and July 23.

1. July 10 — flag defaults. A third-party feature-flag provider degraded, so newly started containers fell back to default flag values. Function execution was degraded for 60 minutes and run scheduling for 125 minutes.
2. July 16 — startup dependency. An upgrade-readiness path in the state store proxy waited on a connection during startup. Restarted proxies could not become ready, and checkpointing suffered for about 168 minutes.
3. July 16 — configuration drift. A reconcile from declarative definitions removed broker permissions that existed only in production, so execution metrics stopped publishing from 15:45 to 19:05 UTC.
4. July 23 — startup dependency again. A storage backend behind one auxiliary message-broker topic stopped allocating capacity. The event-processing service required that topic at startup, so event processing stopped for about 37 minutes.

Two of the four incidents share one root shape. A dependency that was not needed to serve the main workload was still allowed to block startup. That is the lesson most worth stealing.

Why do readiness probe dependencies cause fleet-wide outages?

Readiness probe dependencies cause fleet-wide outages because every replica shares the same dependency, so they all fail together. Kubernetes documents that a failing readiness probe removes the pod from all matching Service endpoints (Kubernetes probe docs). One pod out of rotation is fine. Every pod out of rotation is an outage.

The slow-drain failure mode

The Inngest incidents add a subtle twist: the failure did not appear at once. On July 16, existing proxy connections kept serving traffic. Damage built up only as instances were evicted and restarted, because each fresh instance had to connect at startup and timed out. Inngest notes that impact began at 09:32 UTC, two hours after the path was enabled at 07:11.

This is why the bug survives testing. Inngest says it had tested proxy restarts and several connection-failure scenarios. It had not tested a timeout at startup on that specific path (Inngest RCA). Your staging environment probably has the same gap. Google’s SRE book covers slow startup as one of the classic causes of cascading failure (Addressing Cascading Failures).

Auxiliary paths are the dangerous ones

On July 23, Inngest describes the lifecycle publishing topic as “an auxiliary path.” Yet a failure to connect to it prevented the service from starting at all. In the RCA’s words, this “turned the loss of a single non-critical topic into a full event-processing outage” (source). AWS warns about the same trap: without fail-open protection, a health check that tests a dependency turns it into a hard dependency (Amazon Builders’ Library: Implementing health checks).

How do you fix readiness probe dependencies in Node.js?

Fix readiness probe dependencies by classifying every dependency as critical or optional, then blocking startup only on the critical ones, each with a timeout. Here is the pattern that bit Inngest, reduced to a minimal Express-style service.

// BROKEN: readiness waits for every dependency, critical or not
async function start() {
  await Promise.all([
    db.connect(),              // needed to serve requests
    lifecycleTopic.connect(),  // auxiliary publishing path
    metrics.connect(),         // observability only
  ]);
  ready = true;                // never reached if any connect() hangs
}

app.get('/readyz', (req, res) => res.sendStatus(ready ? 200 : 503));

If lifecycleTopic.connect() hangs, ready never becomes true. Kubernetes keeps the pod out of rotation. After enough restarts, no pods are serving.

// FIXED: block only on critical deps, with a timeout; start optional deps in the background
const withTimeout = (p, ms) => Promise.race([
  p,
  new Promise((_, rej) => setTimeout(() => rej(new Error(`timeout ${ms}ms`)), ms)),
]);

const deps = {
  db:             { critical: true,  connect: () => db.connect() },
  lifecycleTopic: { critical: false, connect: () => lifecycleTopic.connect() },
  metrics:        { critical: false, connect: () => metrics.connect() },
};
const status = {};

async function start() {
  for (const [name, d] of Object.entries(deps)) {
    if (d.critical) {
      await withTimeout(d.connect(), 2000); // fail fast: the pod restarts, which is correct
      status[name] = 'up';
    } else {
      status[name] = 'connecting';
      withTimeout(d.connect(), 500)
        .then(() => { status[name] = 'up'; })
        .catch((e) => { status[name] = `degraded (${e.message})`; }); // alert on this, don't block
    }
  }
  ready = true;
}

app.get('/readyz', (req, res) => res.status(ready ? 200 : 503).json({ ready, status }));

What the simulation shows

We ran both versions in the sandbox on Node.js v22.23.4, with the auxiliary dependency simulated as a promise that never resolves. The broken version was still not ready when the 3-second test budget expired. The fixed version reported ready in 26 ms, then marked the auxiliary path as degraded after its 500 ms timeout.

$ node startup.mjs
node v22.23.4
broken: { ready: false, elapsedMs: 3009 }
fixed : {
  ready: true,
  elapsedMs: 26,
  status: { db: 'up', lifecycleTopic: 'connecting' }
}
fixed after 600ms: { db: 'up', lifecycleTopic: 'degraded (timeout 500ms)' }

The timings are from a toy simulation, not a benchmark. The point is the shape: one design waits forever, the other degrades and keeps serving. The same rule applies in Python services: in a Django or FastAPI app, keep optional clients out of your startup hooks and readiness view.

Are feature flag default values a hidden startup dependency?

Yes. If your code reads flags at startup, the flag provider is a startup dependency, and its default values decide how your service behaves during the provider’s outage. The OpenFeature specification makes this explicit. Requirement 1.4.10 says evaluation calls must always return the default value in the event of abnormal execution.

That is a sensible contract. It means a flag call never crashes your app. But it also means a stale default silently changes behaviour when the provider is down. On July 10, Inngest’s fallback defaults switched a subset of internal traffic from gRPC to a legacy HTTP path. DNS query volume rose roughly tenfold and internal traffic on a key path effectively stalled.

Audit defaults against production

The legacy path had been kept as a per-customer opt-out. It was a fallback that almost never ran in production. Amazon’s Builders’ Library describes this exact risk: fallback paths are hard to test and often hide latent bugs that surface months later (Avoiding fallback in distributed systems). A cheap guard is a CI check that compares code defaults with live values.

// Fail CI when a flag's code default differs from what production serves
const productionValues = await exportFlagsFromProvider();   // your flag service export
const codeDefaults     = collectDefaultsFromSource();       // getBooleanValue(key, DEFAULT)

const drift = Object.keys(productionValues)
  .filter((key) => productionValues[key] !== codeDefaults[key]);

if (drift.length) {
  console.error(`${drift.length} flags change behaviour if the flag service is unreachable:`, drift);
  process.exitCode = 1;
}

$ node flag-audit.mjs   # sample data: three long-rolled-out flags still defaulting to false
3 of 3 flags change behaviour if the flag service is unreachable
exit 1

Inngest’s own follow-up was similar in spirit: it switched a subset of flags to static values to remove the provider dependency on critical paths (Inngest RCA). My recommendation: once a flag is fully rolled out, either delete it or make its default equal to the production value the same week.

Should a fallback path exist at all?

A fallback path should exist only if it runs regularly in production; otherwise it is an untested outage mode. Amazon’s Builders’ Library says it “almost never” uses fallback, and recommends converting fallback into failover that runs continuously (AWS). The article recounts an Amazon retail outage from around 2001 where a cache fallback sent every web server to the database at once.

Inngest’s July 10 incident follows the same arc. The legacy HTTP path was safe as a per-customer opt-out, but it was never sized to carry the fleet. When flag defaults sent traffic there, the network could not cope. If you keep a legacy path, test it under real load or remove it. This pairs well with designing retries safely; see our guide to idempotency keys, since Inngest’s same release also contained a regression where retried idempotent requests failed.

How fast should alerts on degraded dependencies fire?

Alerts on degraded dependencies should page at a priority that matches customer impact, not the priority of the component. On July 16, Inngest’s medium-priority alert fired at 09:35 UTC, three minutes into impact. Because it was medium priority, it did not trigger on-call escalation, and the incident was not formally identified until 11:40 UTC.

The metrics incident had the opposite problem: no alerting on the publishing path existed. Errors began at 15:45 UTC and the incident was declared at 17:44. Compare July 23, where automated alerts fired within minutes and the root cause was identified by 14:35 UTC. Severity routing is part of your architecture, just like timeouts. For broader latency hygiene, our API performance optimization checklist covers the request-path side.

Readiness probe dependencies: which ones should block startup?

Use this decision table, built from the Inngest RCA, the Kubernetes probe documentation and the AWS Well-Architected graceful degradation guidance (REL05-BP01).

Readiness probe dependencies: what to block on, what to degrade

Dependency type	Block startup?	Fail readiness?	When unavailable	Inngest example
Primary datastore needed for every request	Yes, with timeout	Yes	Restart or stay out of rotation	—
Auxiliary publishing topic	No	No	Connect in background, alert	July 23 event lifecycle topic
Upgrade or migration path behind a flag	No	No	Serve from the stable path	July 16 upgrade-readiness path
Feature-flag provider	No	No	Use defaults equal to production values	July 10 flag outage
Metrics or tracing backend	No	No	Keep serving; alert on export errors	July 16 metrics publishing
Parameter store	No	No	Use cached or baked-in defaults (AWS)	—

The same logic applies outside Kubernetes. On ECS, a load-balancer health check plays the readiness role. On Lambda there is no probe, but a cold start that awaits an optional client still adds latency to every first request; our server vs serverless explainer covers where those boundaries sit.

• Inngest reported four incidents in July 2026, and two were caused by dependencies that blocked service startup.
• A failing Kubernetes readiness probe removes the pod from all matching Service endpoints, so a shared dependency can drain every replica.
• Startup-dependency failures often appear gradually, as pods restart, which is why restart tests in staging can miss them.
• OpenFeature requires flag evaluation to return the caller’s default on abnormal execution, so defaults define outage behaviour.
• Amazon’s Builders’ Library recommends avoiding rarely exercised fallback paths or converting them into continuously used failover.
• Alert priority must reflect customer impact; Inngest’s medium-priority alert delayed formal incident identification by about two hours.

Frequently Asked Questions

Should a readiness probe check dependencies?

Only dependencies the pod truly needs to serve requests. Kubernetes removes a pod from Service endpoints when readiness fails, so checking a shared dependency can pull every pod out at once. Check local state and critical connections; report optional dependencies as degraded in metrics instead of failing the probe.

What is the difference between liveness and readiness probes?

A liveness probe tells the kubelet when to restart a container. A readiness probe decides whether the pod receives traffic, and failing it removes the pod from Service endpoints without a restart. Liveness should never depend on external systems; readiness can, but only for critical ones.

What happens when a readiness probe fails?

Kubernetes marks the pod not ready and removes it from the endpoints of every matching Service, so it stops receiving traffic. The container keeps running. If every replica fails readiness together, for example because they share a broken dependency, the Service has no endpoints left and the outage is total.

What is a startup probe in Kubernetes?

A startup probe protects slow-starting containers. While it is configured and has not yet succeeded, Kubernetes disables liveness and readiness checks for that container. It runs only during startup, unlike liveness and readiness probes, which run periodically for the life of the container.

What should a feature flag default value be?

The default should be the behaviour you want when the flag service is unreachable, which for a fully rolled-out flag is usually the production value. OpenFeature requires evaluation calls to return the default on abnormal execution, so a stale default silently changes behaviour during an outage.

Conclusion

Inngest’s July 2026 write-up is worth reading in full, because it documents failures most teams quietly share. The core lesson is small enough to act on today. List every dependency your service touches at boot. Mark each one critical or optional. Make only the critical ones block readiness, each with a timeout, and connect the rest in the background with alerts. Then check your feature-flag defaults against production and delete any fallback path you never exercise. None of this needs new infrastructure. It needs one review session and a CI check. If you run Node.js services on Kubernetes or ECS, start with your /readyz handler; it is probably checking more than it should.

Sources

• Inngest — Root Cause Analysis: July 2026 Service Incidents
• Kubernetes — Configure Liveness, Readiness and Startup Probes
• OpenFeature — Specification: Flag Evaluation API
• Amazon Builders’ Library — Avoiding fallback in distributed systems
• Amazon Builders’ Library — Implementing health checks
• AWS Well-Architected Framework — REL05-BP01 Implement graceful degradation
• Google SRE Book — Chapter 22: Addressing Cascading Failures