
LLM Context Length Exceeded in Node.js: Count, Trim, Recover
LLM context length exceeded errors in Node.js: detect OpenAI and Claude overflows, count tokens correctly, and trim history without breaking tool calls.
- What: “LLM context length exceeded” is a 400 error returned when your prompt, history and tools no longer fit the model’s window.
- Why it matters: Chat apps and agents grow every turn, so this bug shows up in production, rarely in the demo.
- What to do: Count tokens before sending, trim whole turns from the oldest end, pin the task, and catch the error as a fallback.
- Key stat: GPT-5 has a 400,000-token window but accepts at most 272,000 input tokens.
To fix an LLM context length exceeded error in Node.js, stop sending the full chat history. Count the request’s tokens first (the provider’s counting endpoint or a local tokenizer), reserve room for the reply, then drop the oldest complete turns while keeping the system prompt, the original task and tool call/result pairs intact. Catch the 400 as a last resort.
An LLM context length exceeded error is the rejection an LLM API returns when the tokens in a request exceed the model’s context window. The window covers the system prompt, every message, tool definitions, tool results and, on most providers, the output the model is about to generate.
The LLM context length exceeded error is the bug almost every chat feature ships with. The first version stores messages in an array, sends the whole array on every turn, and works perfectly in testing. Then a real user keeps a support thread open all afternoon, or an agent loops through dozens of tool calls, and the API starts answering with a 400. OpenAI’s Node SDK surfaces it as code: 'context_length_exceeded', as this developer forum report shows. Anthropic returns invalid_request_error with “prompt is too long”.
The usual advice is “trim your history”. That’s right, but most guides stop there and miss three things that break real apps. Character-based estimates are badly wrong for JSON and base64. Naive trimming orphans tool results, so the API rejects the request again. And newer Claude models don’t always throw: some stop mid-answer instead. This post shows the broken code, the fixed code, and the numbers we measured in a sandbox.
Why does the LLM context length exceeded error happen?
It happens because everything in the request shares one fixed token budget, and a stateless chat API makes you resend the whole conversation every turn. Anthropic’s docs list what counts: the system prompt, every message including tool results, images and documents, your tool definitions, and the output for the turn including thinking.
Two details catch people out. First, prompt caching doesn’t help. Anthropic states that cached prompt prefixes still occupy the context window; caching changes the price, not the size. Second, the advertised window is not always your input limit. OpenAI’s GPT-5 page lists a 400,000-token context window, a 272,000-token input maximum and 128,000 max output tokens. Budget against the bigger number and you will overflow.
Here’s the broken pattern. It appears in countless tutorials:
// BROKEN: history grows forever
import OpenAI from "openai";
const client = new OpenAI();
const history = [];
export async function chat(userText) {
history.push({ role: "user", content: userText });
const res = await client.chat.completions.create({
model: "gpt-5",
messages: history, // whole conversation, every turn
});
const reply = res.choices[0].message;
history.push(reply);
return reply.content;
}
Nothing here is wrong on turn one. Once the history passes the input cap, every call fails, and the user can’t recover without starting a new chat.
What does the LLM context length exceeded error look like in OpenAI and Claude?
Both providers return HTTP 400, but the shape differs, so detect both explicitly. OpenAI’s error object carries type: 'invalid_request_error', param: 'input' and code: 'context_length_exceeded'. Anthropic’s docs say that if the input alone exceeds the window, the API returns a 400 invalid_request_error (“prompt is too long”) on every model.
The Claude gotcha is the second case. On Claude 4.5 models and newer, if input plus max_tokens exceeds the window, the API accepts the request and stops with stop_reason: "model_context_window_exceeded" when generation hits the limit. No exception is thrown. You get a cut-off answer that looks like success unless you check stop_reason.
// Detect both failure modes
export function isContextOverflow(err) {
if (err?.status !== 400) return false;
const code = err.code ?? err.error?.code; // OpenAI
const msg = String(err.error?.error?.message ?? err.message ?? "");
return code === "context_length_exceeded" || /prompt is too long/i.test(msg);
}
export function wasTruncatedByWindow(claudeMessage) {
return claudeMessage.stop_reason === "model_context_window_exceeded";
}
My recommendation: log both conditions with the request’s token count. A rise in either is your early warning that some conversation type has outgrown its budget.
How do you count tokens before sending a request in Node.js?
Use the provider’s own counter when you can, and a local tokenizer when you need speed. For Claude, messages.countTokens accepts the same payload as a real request. Anthropic says it is free to use and rate-limited at 5,000 requests per minute on the Start tier, with limits separate from message creation.
import Anthropic from "@anthropic-ai/sdk";
const anthropic = new Anthropic();
const { input_tokens } = await anthropic.messages.countTokens({
model: "claude-sonnet-5-5",
system,
tools,
messages,
});
Three limits apply. The count is an estimate that may differ by a small amount from real usage. The endpoint rejects server tools, the MCP connector and url or file image sources. And it adds a network round trip to every turn.
Counting OpenAI tokens locally with js-tiktoken
For OpenAI models, js-tiktoken runs in pure JavaScript. OpenAI’s cookbook maps o200k_base to the GPT-4o family and adds 3 tokens of overhead per message plus 3 to prime the reply. It also warns that tool definitions consume extra tokens on top of that estimate. In Python, the equivalent is tiktoken.get_encoding("o200k_base").
Is the 4-characters-per-token rule good enough?
No. It is close for English prose and wrong for the content that fills agent contexts. We ran a small test in our sandbox (Node v22.23.4, js-tiktoken 1.0.21, o200k_base) on five realistic strings:
import { Tiktoken } from "js-tiktoken/lite";
import o200k from "js-tiktoken/ranks/o200k_base";
const enc = new Tiktoken(o200k);
for (const [name, text] of Object.entries(samples)) {
const chars = [...text].length, tokens = enc.encode(text).length;
console.log(name, chars, tokens, (chars / tokens).toFixed(2), Math.ceil(chars / 4));
}
| Content | Chars | Real tokens | Chars per token | chars/4 estimate | Estimate error |
|---|---|---|---|---|---|
| English prose | 310 | 63 | 4.92 | 78 | +24% |
| TypeScript code | 315 | 91 | 3.46 | 79 | −13% |
| Hindi text | 127 | 45 | 2.82 | 32 | −29% |
| JSON tool result | 298 | 124 | 2.40 | 75 | −40% |
| Base64 blob | 528 | 360 | 1.47 | 132 | −63% |
A negative error means the heuristic under-counts, which is the dangerous direction. JSON tool results came out 40% heavier than the rule predicts, and base64 63% heavier. That fits a real forum case where a developer’s tiktoken count was 73,878 tokens but the API reported 249,114. The reply points to base64 image data pasted into a text field as a common cause.
Don’t reuse counts across Claude model generations
Anthropic notes that Claude 4.7 and later models use a newer tokenizer that produces approximately 30 percent more tokens for the same text. A trimming budget tuned on an older model can overflow after an upgrade. Recount against the model ID you actually call.
How do you trim chat history to fix the LLM context length exceeded error?
Trim whole turns from the oldest end, keep the original task pinned, and never separate a tool call from its result. Dropping single messages is the classic second bug. You delete an assistant tool_use block but keep its tool_result, and the provider rejects the request for a different reason. Here’s the fix we ran:
// FIXED: budgeted, turn-aware trimming
function toTurns(messages) {
const turns = [];
for (const m of messages) {
const isToolResult = m.role === "tool" ||
(Array.isArray(m.content) && m.content.some((b) => b.type === "tool_result"));
if (isToolResult && turns.length) turns.at(-1).push(m); // glue to its call
else turns.push([m]);
}
return turns;
}
export function fitToWindow(messages, { contextWindow, maxOutput, safety = 0.1, count }) {
const budget = Math.floor((contextWindow - maxOutput) * (1 - safety));
const turns = toTurns(messages);
const pinned = turns.shift(); // first user turn = the task
const cost = (t) => t.reduce((s, m) => s + count(m), 0);
let used = cost(pinned);
const kept = [];
for (let i = turns.length - 1; i >= 0; i--) { // newest first
const c = cost(turns[i]);
if (used + c > budget) break;
kept.unshift(turns[i]);
used += c;
}
const dropped = turns.length - kept.length;
const out = [pinned, ...kept].flat();
if (dropped > 0 && kept[0]?.[0].role === "user") // keep roles alternating
out.splice(pinned.length, 0, { role: "assistant", content: `[${dropped} earlier turns omitted]` });
return { messages: out, used, budget, dropped };
}
We tested it on a synthetic agent transcript of 1,601 messages with 400 tool call/result pairs, estimated at 57,219 tokens. With a 32,000-token window and 4,000 reserved for output, the budget was 25,200. The function kept 707 messages using 25,197 tokens and dropped 671 old turns. It left zero orphaned tool results and zero consecutive same-role messages. Those numbers come from our own run of the code above, not a published benchmark.
The 10% safety margin covers the gap between a local estimate and the provider’s real count. If you call countTokens for an exact figure, you can shrink it.
Should you use provider truncation, compaction or summarization instead?
Use them when they fit, but know what they drop. OpenAI’s Responses API has a truncation parameter. Its default, disabled, fails with a 400; auto drops items from the beginning of the conversation. The same reference warns that trimming can start earlier than expected because the API subtracts the model’s maximum output allowance, even if you set a lower max_output_tokens. The reference also marks the parameter as deprecated, so don’t build new code around it.
Dropping from the beginning is exactly what removes the user’s original task. That’s why the fix above pins the first turn. For Claude, Anthropic recommends server-side compaction, in beta for Claude 4.6 and later models, which summarizes earlier turns on the server. Context editing can also clear old tool results, which are usually the heaviest items, as our table shows.
Summarization is the third option. When trimming would drop many turns, ask a cheap model to summarize the dropped span and insert it after the pinned turn. It costs one extra call, so trigger it only when dropped crosses a threshold. If you also need a hard cost ceiling, our guide to AI agent budget guards pairs well with this.
How do you handle an LLM context length exceeded error as a last resort?
Wrap the call, catch the overflow, trim harder, and retry once. Never resend the same payload: a context overflow is deterministic, so a blind retry loop just burns requests.
- Before each call, run
fitToWindowwith your normal budget. - Send the request.
- If
isContextOverflow(err)is true, rerunfitToWindowwithsafety: 0.3and retry once. - If it fails again, the pinned turn itself is too large: chunk the document or move it to retrieval.
- For Claude, also check
wasTruncatedByWindow(res)and treat it as a failure, not a short answer.
export async function safeChat(send, messages, opts) {
let { messages: fitted } = fitToWindow(messages, opts);
try {
return await send(fitted);
} catch (err) {
if (!isContextOverflow(err)) throw err;
({ messages: fitted } = fitToWindow(messages, { ...opts, safety: 0.3 }));
return await send(fitted); // one retry, then let it throw
}
}
If the overflow comes from oversized tool output, cap it at the source. Our post on the hidden token tax of Claude agent skills covers how instructions and tool payloads quietly eat the window. If a truncated reply is also breaking your JSON, see LLM structured outputs in Node.js.
| Behaviour | OpenAI | Anthropic Claude |
|---|---|---|
| Input alone too large | 400, context_length_exceeded (example) |
400, “prompt is too long” (docs) |
| Input + output too large | Input cap can sit below the window, e.g. GPT-5 272K of 400K | Claude 4.5+: accepted, stops with model_context_window_exceeded |
| Pre-flight counting | Local tokenizer (tiktoken), an estimate | countTokens, free, 5,000–20,000 RPM by tier |
| Built-in mitigation | truncation: "auto" drops oldest items (deprecated) |
Server-side compaction (beta), context editing |
| Window on current models | Model-specific; check each model page | 1M on current models, 200K on older ones |
The NexGismo verdict: Treat the context window as a budget you manage, not a limit you discover from an error. I’d use the provider’s counter where it exists, a local tokenizer with a 10% margin elsewhere, and turn-aware trimming that pins the task. Provider truncation is a convenience, not a strategy, because it deletes the very turn your user cares about most.
- OpenAI returns HTTP 400 with
code: 'context_length_exceeded'when a request is too large for the model’s window. - Claude returns 400 “prompt is too long” when input alone overflows, but Claude 4.5+ stops with
model_context_window_exceededwhen input plus output overflows. - GPT-5’s 400,000-token window includes a 272,000-token input cap, so budget against the input limit.
- In our o200k_base test, the chars/4 rule under-counted JSON by 40% and base64 by 63%.
- Claude 4.7 and later tokenizers produce about 30% more tokens than earlier models for the same text.
- Trim whole turns from the oldest end and keep every tool call with its tool result.
Frequently Asked Questions
How do I fix “This model’s maximum context length is” errors?
Count the tokens in your full request before sending it, reserve space for the reply, and remove the oldest complete conversation turns until it fits. Keep the system prompt and the original task. For very large single inputs, such as documents, split them into chunks or retrieve only the relevant parts instead of sending everything.
Does max_tokens count toward the context window?
Yes, on most providers the output shares the window with the input. OpenAI’s Responses API can start trimming early because it subtracts the model’s maximum output allowance. On Claude 4.5 and newer, a request whose input plus max_tokens exceeds the window is accepted but may stop with model_context_window_exceeded.
How many characters are in a token?
It depends on the content. In our o200k_base test, English prose averaged 4.92 characters per token, TypeScript 3.46, JSON 2.40 and base64 just 1.47. The common four-characters rule works only for plain English and under-counts structured data, so use a real tokenizer for budgeting.
Is the Claude token counting API free?
Yes. Anthropic’s token counting endpoint is free to use but rate-limited by usage tier, starting at 5,000 requests per minute. Its limits are separate from message creation, so counting does not reduce your messaging quota. The returned count is an estimate and can differ slightly from actual usage.
Does prompt caching increase the context window?
No. Anthropic’s documentation states that cached prompt prefixes still occupy the context window. Prompt caching lowers the cost and latency of repeated prefixes, but every cached token still counts toward the model’s limit, so you need the same trimming or compaction strategy with or without caching.
The LLM context length exceeded error is predictable, which makes it fixable before users see it. Count tokens with the right tokenizer, trim whole turns while pinning the task, check Claude’s stop reason, and keep one bounded retry as a safety net. Run the trimming function against your longest real transcripts before you ship it; the token totals will quickly show whether your budget and margin are right. If you’re tuning repeated prefixes too, our walkthrough of Bedrock prompt caching in Node.js is the natural next read.
Sources
- Anthropic — Claude Platform Docs: Context windows
- Anthropic — Claude Platform Docs: Token counting
- OpenAI — GPT-5 model page
- OpenAI — API reference: Create a response (truncation)
- OpenAI Cookbook — How to count tokens with tiktoken
- npm — js-tiktoken package
- OpenAI Developer Community — “Your input exceeds the context window of this model”
- OpenAI Developer Community — GPT-4o context length issue: tiktoken vs API count