LLM Structured Outputs in Node.js: Fix Broken JSON From AI

LLM structured outputs exist because “Return ONLY valid JSON” in a system prompt is a request, not a contract. Every developer who has wired a model into a Node.js backend has seen the failures: a ```json fence around the payload, a friendly “Sure! Here’s the data:” before it, a missing key, or "Enterprise" where your code expected "enterprise". Each one throws in JSON.parse() or, worse, passes parsing and writes bad data to MongoDB.

OpenAI’s launch post describes the old workaround honestly: developers relied on open source tooling, prompting, and “retrying requests repeatedly” to get usable output. Anthropic’s documentation lists the same symptoms: parsing errors, missing required fields, inconsistent types and schema violations that need retries.

This guide shows the broken pattern, the fix for OpenAI and Claude in Node.js with Zod, and the edge cases that still break “guaranteed” output in production. The examples use an Express and MongoDB endpoint, so they drop straight into a MEAN or MERN backend.

Why does prompting an LLM for JSON keep breaking your parser?

Prompting fails because an unconstrained model can sample any token at any time, including ones that make the JSON invalid. OpenAI explains that by default models “can select any token from the vocabulary,” so they are “free to sample a curly brace token at any time, even when that would not produce valid JSON.”

Better models reduce the error rate but don’t remove it. OpenAI trained gpt-4o-2024-08-06 specifically to follow complex schemas, and it still reached only 93% on their benchmark without constraints. That’s roughly one failure in fourteen calls on hard schemas, which is fine for a demo and unacceptable for a queue processing thousands of documents.

Here’s the pattern most codebases start with:

// ❌ The "prompt and pray" pattern
import OpenAI from "openai";
const openai = new OpenAI();

const res = await openai.chat.completions.create({
  model: process.env.MODEL,
  messages: [
    { role: "system", content: "Return ONLY valid JSON: {name, email, plan}. No prose." },
    { role: "user", content: emailBody },
  ],
});

// Breaks on: ```json fences, "Sure! Here's the JSON:", trailing commas,
// missing keys, "plan": "enterprise" vs "Enterprise", truncated output...
const lead = JSON.parse(res.choices[0].message.content);
await db.collection("leads").insertOne(lead);   // garbage in, garbage stored

Nothing in this code is unusual, and that is the problem. Each failure mode needs its own regex, retry or try/catch, and none of them is ever complete.

How do LLM structured outputs actually guarantee valid JSON?

LLM structured outputs guarantee valid JSON by masking every token that would break the schema before the model samples. OpenAI converts the supplied JSON Schema into a context-free grammar. After each token, the inference engine computes which next tokens are valid and lowers the probability of everything else to zero.

The result is a different reliability class. On OpenAI’s evals of complex JSON schema following, gpt-4o-2024-08-06 with Structured Outputs scored 100%, while gpt-4-0613 scored less than 40%. Anthropic uses the same approach; its docs describe constrained decoding with compiled grammar artifacts for both JSON outputs and strict tool use.

Structured outputs vs JSON mode

JSON mode is the older, weaker feature. OpenAI’s guide states that while both produce valid JSON, “only Structured Outputs ensure schema adherence,” and recommends Structured Outputs over JSON mode whenever possible. If your code still sends response_format: { type: "json_object" }, you are getting valid syntax with no guarantee about keys or types.

How do you use LLM structured outputs in Node.js with Zod?

You use LLM structured outputs in Node.js by defining one Zod schema and passing it to the provider’s format option. The same schema then types your result and validates it again in your own code.

OpenAI: zodTextFormat with the Responses API

The OpenAI structured outputs guide uses zodTextFormat from openai/helpers/zod inside text.format, and returns the parsed object on output_parsed.

// ✅ OpenAI Responses API + Zod: schema enforced by constrained decoding
import OpenAI from "openai";
import { zodTextFormat } from "openai/helpers/zod";
import { z } from "zod";

const Lead = z.object({
  name: z.string(),
  email: z.string(),
  plan: z.enum(["starter", "team", "enterprise"]),
  demoRequested: z.boolean(),
  notes: z.string().nullable(),          // "optional" = nullable + required
});

const openai = new OpenAI();
const response = await openai.responses.parse({
  model: process.env.MODEL,
  input: [
    { role: "system", content: "Extract the sales lead from the email." },
    { role: "user", content: emailBody },
  ],
  text: { format: zodTextFormat(Lead, "lead") },
});

const lead = response.output_parsed;     // typed as z.infer<typeof Lead>

Claude: output_config.format

Anthropic’s Messages API takes the schema in output_config.format with type: "json_schema", and returns valid JSON in the text content block. Zod 4’s built-in z.toJSONSchema() produces the raw schema. Note that types with no JSON equivalent, such as z.date() and z.transform(), throw by default.

// ✅ Claude Messages API: JSON outputs via output_config.format
import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";

const Lead = z.object({
  name: z.string(),
  email: z.string(),
  plan: z.enum(["starter", "team", "enterprise"]),
  demoRequested: z.boolean(),
});

const client = new Anthropic();
const msg = await client.messages.create({
  model: process.env.CLAUDE_MODEL,
  max_tokens: 1024,
  messages: [{ role: "user", content: `Extract the sales lead:\n\n${emailBody}` }],
  output_config: {
    format: {
      type: "json_schema",
      schema: z.toJSONSchema(Lead),   // Zod 4 built-in converter
    },
  },
});

Gemini supports the same idea. Google’s November 2025 update expanded JSON Schema support so libraries like Zod and Pydantic work directly, and keeps key order from your schema on Gemini 2.5 and later.

My recommendation: keep one Zod schema per output type in a shared module, and never hand-write the JSON Schema separately. Two sources of truth drift within a sprint.

When do LLM structured outputs still return invalid or wrong data?

Structured outputs can still fail in four documented ways, and each needs explicit handling. This is the section most tutorials skip, and it is where production incidents come from.

1. Refusals. Both providers let the model refuse unsafe requests. OpenAI adds a refusal field to the response. Anthropic returns stop_reason: refusal with a 200 status, bills the tokens, and the output may not match your schema.
2. Truncation. If generation hits max_tokens, the JSON can be incomplete. Anthropic’s docs say to retry with a higher max_tokens; OpenAI flags it via finish_reason.
3. Enum casing drift. Anthropic warns that structured outputs “don’t guarantee the capitalization of string enum and const values.” The response completes normally with no error. Compare enums case-insensitively.
4. Correct shape, wrong values. OpenAI is explicit that structured outputs “doesn’t prevent all kinds of model mistakes,” such as a wrong step in a math equation. Schema-valid is not the same as true.

Here’s the guard I’d put around every structured call:

// ✅ The guard you still need, whichever provider you use
function readStructured(msg, schema) {
  // Claude: stop_reason; OpenAI Chat Completions: finish_reason / message.refusal
  if (msg.stop_reason === "refusal") {
    throw new LlmRefusalError("Model refused; output may not match schema");
  }
  if (msg.stop_reason === "max_tokens") {
    throw new LlmTruncatedError("Output cut off; raise max_tokens and retry");
  }
  const text = msg.content.find((b) => b.type === "text")?.text ?? "";
  const data = JSON.parse(text);

  // Re-validate in your code: catches enum casing drift and constraints
  // (min/max/length) that the API stripped from the schema
  const result = schema.safeParse(normaliseEnums(data));
  if (!result.success) throw new LlmSchemaError(result.error);
  return result.data;
}

const normaliseEnums = (d) => ({ ...d, plan: String(d.plan).toLowerCase() });

Constraints that silently disappear

The Claude SDK helpers remove unsupported constraints such as minimum, maximum, minLength and maxLength before sending the schema, and move them into field descriptions. The model sees a hint, not a rule. If you call the raw API instead, those constraints return a 400 error. Either way, the safeParse step above is what actually enforces them.

What schema limits and latency costs should you plan for?

Plan for a one-time compile cost per schema and hard limits on schema complexity. Both providers compile your schema into a grammar on first use, so the first request is slower.

OpenAI says typical schemas take under 10 seconds to process on the first request, while complex ones may take up to a minute, and later requests have no penalty. Anthropic caches compiled grammars for 24 hours from last use; changing the schema structure, or the tool set when combined with tools, invalidates the cache. Changing only names or descriptions does not.

Anthropic’s explicit complexity limits

Per the Claude structured outputs docs, a single request allows at most 20 strict tools, 24 optional parameters across all strict schemas, and 16 parameters using union types such as ["string", "null"]. Exceeding internal grammar limits returns “Schema is too complex for compilation,” and compilation times out at 180 seconds. The docs advise making parameters required where possible, because each optional parameter “roughly doubles a portion of the grammar’s state space.”

Two more gotchas worth knowing. Anthropic outputs required properties first, then optional ones, regardless of schema order. And OpenAI notes that structured outputs is not compatible with parallel function calls; set parallel_tool_calls: false when you need strict tool schemas. For agent designs that juggle many tools, our guide to single vs multi-agent orchestration covers how to split tool sets across agents.

How do you wire structured outputs into an Express and MongoDB API?

You wire it in by treating the model as an untrusted upstream service: validate, map failures to HTTP statuses, and only then write to the database. The schema guarantee removes parsing failures, not the need for error handling.

// MEAN-stack style endpoint: Express + MongoDB, validated before insert
app.post("/api/leads/extract", async (req, res) => {
  try {
    const lead = await extractLead(req.body.emailBody);   // one of the calls above
    const { insertedId } = await db.collection("leads").insertOne({
      ...lead,
      source: "email",
      extractedAt: new Date(),
    });
    res.status(201).json({ id: insertedId, lead });
  } catch (err) {
    if (err instanceof LlmRefusalError) return res.status(422).json({ error: "refused" });
    if (err instanceof LlmTruncatedError) return res.status(502).json({ error: "truncated" });
    if (err instanceof LlmSchemaError) return res.status(502).json({ error: "invalid_output" });
    throw err;
  }
});

Map each failure to a distinct status so your dashboards separate “model refused” from “model output was truncated.” With MongoDB’s flexible documents, it is tempting to insert whatever comes back; the Zod check is what keeps the collection consistent for the Angular or React front end reading it.

Token cost is the other operational detail. Anthropic injects an extra system prompt describing the format, which slightly increases input tokens, and changing output_config.format invalidates prompt caching for that conversation. If you also load agent skills, our breakdown of the hidden token tax of Claude Agent Skills shows how fixed prompt overhead adds up. Teams running Claude through AWS can check our Claude on Bedrock setup guide for platform-specific setup.

LLM output approaches compared (from provider documentation)

Approach	Valid JSON?	Schema adherence?	Known failure cases	Source
Prompt-only (“return JSON”)	No guarantee	No guarantee (gpt-4-0613: under 40% on complex schemas)	Fences, prose, missing keys	OpenAI
Trained model, no constraints	Usually	93% on OpenAI’s complex-schema benchmark	Occasional schema violations	OpenAI
JSON mode	Yes	No	Wrong keys or types	OpenAI guide
OpenAI Structured Outputs (strict)	Yes	100% in OpenAI’s evals	Refusal, max_tokens, parallel tool calls	OpenAI
Claude JSON outputs / strict tools	Yes	Yes, with documented exceptions	Refusal, max_tokens, enum casing	Anthropic

• LLM structured outputs use constrained decoding to stop a model from emitting tokens that break the supplied JSON Schema.
• OpenAI reported 100% schema adherence on complex schemas with strict structured outputs, versus under 40% for gpt-4-0613.
• JSON mode guarantees valid syntax only; structured outputs also guarantee keys, types and enum values.
• Refusals and max_tokens truncation can still produce output that does not match the schema on both OpenAI and Claude.
• Claude’s structured outputs may change the capitalization of enum values without raising an error.
• Re-validating responses with the same Zod schema enforces constraints that providers strip from the sent schema.

Frequently Asked Questions

What is the difference between JSON mode and structured outputs?

JSON mode only guarantees syntactically valid JSON. Structured outputs also guarantee the JSON matches the schema you supply, including required keys, types and enum values. OpenAI's guide says only Structured Outputs ensure schema adherence and recommends using it instead of JSON mode wherever the model supports it.

Are LLM structured outputs 100% reliable?

The format is reliable, but not unconditionally. OpenAI and Anthropic both document two cases where output may not match the schema: a safety refusal and hitting the max_tokens limit. Values can also be wrong even when the shape is right, so validate business rules in your own code.

Does Claude support structured outputs?

Yes. Anthropic's Messages API supports JSON outputs via output_config.format with type json_schema, and strict tool use via strict: true on a tool definition. The docs list it as available on all models. The older output_format parameter is deprecated and needs a beta header.

Why does my structured outputs request return a 400 error?

Usually the schema uses an unsupported JSON Schema feature. Anthropic, for example, rejects numerical constraints like minimum, string length constraints, recursive schemas, and additionalProperties set to anything other than false. Very complex schemas can also fail with a 'Schema is too complex for compilation' error.

Can I use Zod schemas for LLM structured outputs in Node.js?

Yes. The OpenAI Node SDK ships zodTextFormat and zodResponseFormat helpers that convert a Zod object and parse the response into a typed value. For other APIs, Zod 4's built-in z.toJSONSchema() produces a standard JSON Schema you can pass directly, though types like z.date() throw by default.

What should you change in your AI integration today?

Search your codebase for JSON.parse calls on model output and for prompts containing “only JSON.” Replace each with a Zod schema passed through the provider’s structured output option. Then add the guard: refusal, truncation, case-insensitive enums and a final safeParse. Finally, log each failure type separately so you can see which one actually happens in your traffic.

If you’re building AI features on a MEAN or MERN stack, this pattern is the foundation for everything else: tool calling, agents and RAG pipelines all depend on machine-readable output. Tell us which provider or framework you’d like covered next, and we’ll test it in a follow-up.

Sources

• OpenAI — Introducing Structured Outputs in the API
• OpenAI Docs — Structured Outputs guide
• Anthropic Claude Docs — Structured outputs
• Zod Docs — JSON Schema (z.toJSONSchema)
• Google — Improving Structured Outputs in the Gemini API