Site icon NexGismo

Cancel LLM Stream on Client Disconnect in Node.js: The Fix

Ashutosh Tiwari
Cancel LLM Stream on Client Disconnect in Node.js: The Fix
TL;DR
  • What: Most Node.js chat endpoints keep generating LLM tokens after the user closes the tab.
  • Why it matters: You burn GPU time, rate-limit quota and sometimes money on answers nobody reads.
  • What to do: Abort from res.on('close') when res.writableFinished is false, and pass the signal to the SDK.
  • Key stat: In our Node 22 test, the fix stopped generation at 11 of 40 tokens; the common pattern produced all 40.

To cancel LLM stream on client disconnect in Node.js, create an AbortController per request, pass its signal to the OpenAI or Anthropic SDK call, and abort it inside res.on('close') only when res.writableFinished is false. Do not use req.on('close'): since Node.js 16 it fires when the request body is read, not when the user leaves.

— Ashutosh Tiwari, NexGismo

Client-disconnect cancellation is the practice of stopping server-side work, such as an upstream LLM generation, as soon as the HTTP client that asked for it goes away. In Node.js it is usually built from an AbortController whose signal is passed to every downstream call and aborted when the response connection closes early.

If you’ve tried to cancel LLM stream on client disconnect in Node.js, you probably wrote the obvious code, and it is wrong in a way no test catches. A user clicks “Stop” or closes the tab, and your server keeps pulling tokens from OpenAI or Claude until the model finishes. The bug is quiet: no error, no log line, just extra generation time. Developers have been reporting broken disconnect detection on the Node.js tracker since February 2023, and the issue reporter estimated that the affected proxy stack alone sees about 14 million weekly downloads. This post shows the broken pattern, explains why Node behaves that way, and gives fixed code for Express, the OpenAI SDK, the Anthropic SDK and Next.js on Vercel. We also ran the patterns side by side in a sandbox, so you can see exactly how many tokens each one generates after a disconnect.

Why does my Node.js server keep streaming LLM tokens after the client disconnects?

Your server keeps streaming because nothing tells the upstream LLM request to stop. Closing a browser tab drops the TCP connection to your server. It does not touch the separate HTTPS connection your server holds open to the model provider.

So the for await loop keeps reading chunks. Each res.write() goes to a dead socket and is silently discarded. The loop only ends when the model hits its stop condition or max_tokens.

The fix needs two pieces. First, a reliable signal that the client left. Second, a way to pass that signal into the SDK so it closes the upstream request. Both SDKs support the second part; the first part is where most code goes wrong.

Why is req.on(‘close’) the wrong way to cancel LLM stream on client disconnect?

Because in modern Node.js, the request’s 'close' event does not mean “the client went away”. A Node.js issue documents a breaking change in Node.js 16.0.0 that made 'close' fire once the request body has been read. Up to Node.js 15.5.0, an 'aborted' event reported real client aborts.

That 'aborted' event was then deprecated in v17.0.0 and v16.12.0, with docs pointing to 'close' instead. The official deprecation note, DEP0156, recommends checking destroyed and listening for 'close'. But on the request object, 'close' now means something else.

What does the broken code look like?

This is the pattern you’ll find in many tutorials. It looks correct and type-checks fine:

// BROKEN: Express 5 + OpenAI streaming
app.post('/api/chat', express.json(), async (req, res) => {
  const controller = new AbortController();
  // Looks right. It is not.
  req.on('close', () => controller.abort());
  res.setHeader('Content-Type', 'text/event-stream');
  const stream = await openai.chat.completions.create(
    { model: MODEL, messages: req.body.messages, stream: true },
    { signal: controller.signal },
  );
  for await (const chunk of stream) {
    res.write(`data: ${JSON.stringify(chunk.choices[0]?.delta ?? {})}\n\n`);
  }
  res.end();
});

Depending on when the listener attaches, it fails in one of two opposite ways. Either it fires almost immediately and truncates every answer, or it has already fired and never fires again. We measured both.

What happened when we tested each pattern?

We ran a fake LLM generator (40 tokens, one every 50 ms) behind four server variants on Node.js v22.23.4 and Express 5.3.0 in the NexGismo sandbox. A fetch() client either aborted at 500 ms or read to the end. “Server tokens” is how many tokens the generator produced.

Server-side tokens generated per pattern (our sandbox run, Node.js v22.23.4)
Server pattern Client aborts at 500 ms Client reads to the end Verdict
Raw http, req.on('close') added after reading body 40 / 40 (never stopped) 40 / 40 Leaks every disconnect
Raw http, req.on('close') added before reading body 0 / 40 (fired at 12 ms) 0 / 40 Breaks every request
Express 5.3.0 + express.json(), req.on('close') 1 / 40 (fired at 26 ms) 1 / 40 Breaks every request
res.on('close') + !res.writableFinished (raw and Express) 11 / 40 40 / 40 Correct

The surprising row is Express. With express.json(), the request close fired 26 ms in, so every answer was cut to a single token even when nobody disconnected. In local testing that looks like a model problem, not a server bug. The correct pattern stopped generation 22 ms after the client aborted, at 11 tokens.

Here is the full test script. Run it as node exp.mjs res-close disconnect; the modes are req-close or res-close, and the scenarios are disconnect or complete. The Express variant swaps in app.use(express.json()) with the same handler bodies.

import http from 'node:http';
const t0 = Date.now(); const ts = () => String(Date.now()-t0).padStart(4)+'ms';
// fake LLM: emits a token every 50ms for 40 tokens unless aborted
async function* fakeLLM(signal){ for(let i=0;i<40;i++){ if(signal.aborted) return; await new Promise(r=>setTimeout(r,50)); yield `tok${i} `; } }
const mode = process.argv[2]; // 'req-close' | 'res-close'
let generated = 0;
const server = http.createServer(async (req,res)=>{
  let body=''; for await (const c of req) body+=c;
  const ac = new AbortController();
  if (mode==='req-close') req.on('close', ()=>{ console.log(ts(),'req close fired -> abort'); ac.abort(); });
  else res.on('close', ()=>{ console.log(ts(),'res close, writableFinished=',res.writableFinished); if(!res.writableFinished) ac.abort(); });
  res.writeHead(200,{'content-type':'text/event-stream'});
  for await (const tok of fakeLLM(ac.signal)) { generated++; res.write(`data: ${tok}\n\n`); }
  console.log(ts(),'generation loop ended, tokens generated =', generated, 'aborted =', ac.signal.aborted);
  res.end();
});
server.listen(0, async ()=>{
  const port = server.address().port;
  const scenario = process.argv[3]; // 'disconnect' | 'complete'
  const ctl = new AbortController(); let received=0;
  try {
    const r = await fetch(`http://127.0.0.1:${port}/chat`,{method:'POST',body:JSON.stringify({prompt:'hi'}),signal:ctl.signal});
    if (scenario==='disconnect') setTimeout(()=>{console.log(ts(),'client aborts'); ctl.abort();},500);
    for await (const c of r.body) received++;
  } catch(e){ console.log(ts(),'client:',e.name); }
  setTimeout(()=>{ console.log(ts(),`RESULT mode=${mode} scenario=${scenario} serverTokens=${generated}`); server.close(); process.exit(0)}, 2600);
});

How do you cancel LLM stream on client disconnect in Express with OpenAI?

Listen on the response, not the request, and check writableFinished. The response’s 'close' also fires after a normal res.end(), so the flag is what separates “finished” from “user left”. The Node.js issue thread lands on this same workaround.

// FIXED: abort only when the RESPONSE closes before it finished
import express from 'express';
import OpenAI from 'openai';
const openai = new OpenAI();
const app = express();
function abortOnDisconnect(res) {
  const controller = new AbortController();
  res.on('close', () => {
    // 'close' also fires after a normal res.end(); writableFinished tells them apart
    if (!res.writableFinished) controller.abort(new Error('client disconnected'));
  });
  return controller.signal;
}
app.post('/api/chat', express.json(), async (req, res) => {
  const signal = abortOnDisconnect(res);
  res.setHeader('Content-Type', 'text/event-stream');
  try {
    const stream = await openai.chat.completions.create(
      { model: process.env.MODEL, messages: req.body.messages, stream: true },
      { signal, maxRetries: 0 }, // don't retry a request nobody is waiting for
    );
    for await (const chunk of stream) {
      res.write(`data: ${JSON.stringify(chunk.choices[0]?.delta ?? {})}\n\n`);
    }
    res.end();
  } catch (err) {
    if (signal.aborted) return; // expected: the user left
    if (!res.headersSent) res.status(502);
    res.end();
  }
});

The OpenAI Node SDK’s request options accept an AbortSignal as signal, so aborting closes the upstream request. The same file shows maxRetries defaults to 2. The SDK docs also warn that timeouts are retried by default, so a request can outlive its timeout. Setting maxRetries: 0 on interactive streams avoids retrying for a user who has already left.

  1. Create one AbortController per incoming request.
  2. Attach res.on('close') before the first await on the model.
  3. Abort only when res.writableFinished is false.
  4. Pass the signal as the SDK’s second argument, not inside the request body.
  5. Swallow the abort error in catch; log anything else.

If you’re on Node.js 24, the event semantics are the same; our Node.js 24 LTS upgrade guide covers what else changed.

How do you cancel a Claude stream with the Anthropic TypeScript SDK?

Call stream.abort() on the MessageStream. The SDK’s helper docs say you can break out of a for await loop or call stream.abort(). They describe abort() as equivalent to .controller.abort(), and say it also aborts in-flight network requests.

import Anthropic from '@anthropic-ai/sdk';
const anthropic = new Anthropic();
app.post('/api/claude', express.json(), async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  const stream = anthropic.messages.stream({
    model: process.env.CLAUDE_MODEL,
    max_tokens: 1024,
    messages: req.body.messages,
  });
  res.on('close', () => {
    if (!res.writableFinished) stream.abort(); // documented: same as stream.controller.abort()
  });
  stream.on('text', (delta) => res.write(`data: ${JSON.stringify(delta)}\n\n`));
  stream.on('abort', () => console.log('generation cancelled'));
  try { await stream.done(); } catch { /* APIUserAbortError on abort */ }
  res.end();
});

The stream emits an 'abort' event carrying an APIUserAbortError, which is a clean place to persist a partial answer. The SDK supports Node 18 and newer. For tool-runner agents, the same docs show passing { signal } so cancellation reaches both API calls and tool run() methods. That matters for multi-step agents; see our single vs multi-agent orchestration guide for where those loops get long.

How do you cancel LLM streams in Next.js and on Vercel?

In a Next.js route handler you don’t touch res at all. You forward req.signal. The Vercel AI SDK’s stopping-streams guide shows passing abortSignal: req.signal to streamText, with an onAbort callback for cleanup.

// app/api/chat/route.ts (Next.js App Router, Node.js runtime)
import { streamText, createTextStreamResponse, toTextStream } from 'ai';
export async function POST(req: Request) {
  const { prompt } = await req.json();
  const result = streamText({
    model: process.env.MODEL!,
    prompt,
    abortSignal: req.signal,          // forward the platform's disconnect signal
    onAbort: ({ steps }) => console.log('aborted after', steps.length, 'steps'),
  });
  return createTextStreamResponse({ stream: toTextStream({ stream: result.stream }) });
}

On Vercel, that signal only fires if you opt in. Vercel’s Functions API reference says request cancellation is available only in the Node.js runtime and is enabled per path:

{
  "functions": {
    "app/api/chat/route.ts": { "supportsCancellation": true }
  }
}

Without it, the AI SDK docs warn that stop() ends the client stream, but server-side generation may continue. With it, Vercel terminates the function on disconnect. Work not wrapped in waitUntil or after is lost, per the same page. Separately, Vercel gives code up to 500 milliseconds after a SIGTERM before the instance is killed. Keep abort handlers short.

Fastify users can use the hook from the official ‘Detecting When Clients Abort’ guide, which checks request.raw.aborted inside a 'close' listener. Fastify argues that the deprecated aborted property is still more reliable than destroyed.

Does cancelling an LLM stream actually save money?

Not always, and this is the part most tutorials skip. We could not find an official OpenAI or Anthropic page that states exactly how a cancelled stream is billed. The most specific public write-up we found is TrueFoundry’s AI gateway cost-tracking docs, which describe policies “based on measured provider behaviour”:

The same page notes that output tokens for a cancelled stream are never reported, so the gateway estimates roughly one token per 3.5 characters. On one prompt, the provider’s exact input count was 154 tokens versus 111 from a character heuristic. Your own usage logs will be off in the same way for cancelled requests.

Cancellation still pays off even where billing doesn’t change. It frees server memory and sockets, stops downstream tool calls, and releases provider concurrency. If you enforce per-user quotas, count cancelled requests too; our rate limiter system design guide covers token-bucket options.

The NexGismo verdict: If you need to cancel LLM stream on client disconnect, treat req.on('close') as a bug in any streaming handler. My recommendation is a small abortOnDisconnect(res) helper, one per request, passed as the SDK signal with retries off. Then check your provider’s invoices before assuming cancellation saves money.

Frequently Asked Questions

How do I stop an OpenAI stream when the client disconnects in Node.js?

Create an AbortController per request, pass its signal as the second argument to openai.chat.completions.create(), and call controller.abort() from res.on(‘close’) only when res.writableFinished is false. The SDK’s request options accept an AbortSignal, so aborting closes the upstream HTTP connection and ends your for await loop.

Why does req.on(‘close’) fire immediately in Express?

Since Node.js 16, the request’s ‘close’ event means the request body has been fully read, not that the client left. Body parsers such as express.json() read the body before your handler runs, so the event can fire within milliseconds. In our Express 5.3.0 test it fired at 26 ms on every request.

Do I still pay for tokens after cancelling an LLM stream?

It depends on the provider. TrueFoundry’s gateway documentation says OpenAI bills the full completion even when a stream is cancelled, Vertex AI and Azure OpenAI do not bill cancelled streams, and Anthropic and Bedrock bill the partial output. Treat this as a third-party measurement and check your own invoices.

How do I cancel a Claude stream in the Anthropic TypeScript SDK?

Call stream.abort() on the MessageStream returned by anthropic.messages.stream(), or break out of the for await loop. The SDK documents abort() as equivalent to stream.controller.abort(), and it also aborts in-flight network requests. The stream emits an ‘abort’ event you can use for cleanup.

Does req.signal fire on Vercel when the user closes the tab?

Only if you opt in. Vercel’s docs say request cancellation works in the Node.js runtime and must be enabled per path with supportsCancellation: true in vercel.json. Without it, the AI SDK’s stop() ends the client stream but server-side generation may continue.

What should you do next?

To cancel LLM stream on client disconnect reliably, start small. Search your codebase for req.on('close' in any route that streams. Replace it with a response-side check and pass the signal into the SDK call. Then run the disconnect test from this post against your own endpoint and count the tokens. If the numbers look like the first three rows of our table, you’ve found the leak. Finally, compare a week of provider usage before and after the change, because billing behaviour differs by provider. If you hit a framework we didn’t cover, such as Hono, NestJS or Koa, the rule is the same: listen to the response or platform signal, never the request body stream.

Sources

Exit mobile version