- What: Amazon engineers moved a 20-label ticket classifier from Claude Sonnet 4.6 to Strands Decider, a small decision model.
- Why it matters: If the output is a pick from a fixed list, you may be paying for text generation you never use.
- What to do: Run the cheap model in shadow (DUAL) mode first, log every disagreement, and only then cut over.
- Key stat: Cost per ticket fell from about $0.0222 to about $0.0005 at list prices, per the vendor-reported figures.
In the decision model vs LLM choice, a decision model wins when every answer is one option from a fixed list and volume is high. It scores the options in one forward pass instead of generating text. An LLM still wins for open-ended output, changing label sets, low volume, or long context.
A decision model is a language model whose text-generating head has been removed and replaced with a small scoring head. It takes a state and a question with predefined options, and returns a probability for each option plus a confidence score. It cannot write free text, so it suits routing, labeling and policy checks rather than chat or summarization.
The decision model vs LLM question landed on the Hacker News front page this week after the Strands Agents team published how two Amazon engineers replaced Sonnet 4.6 on an internal ticket-labeling job. The pain point is familiar to anyone running AI in production. You write a long system prompt, ask a frontier model for JSON, validate the output, and pay per token for every ticket, even though the answer is one word from a list of 20.
The reported result is large: roughly 44 times cheaper and 4,215 ms down to 897 ms per call. But it is one internal workload, reported by the vendor, with no public evaluation set. This post breaks down what was actually measured, recomputes the arithmetic, and adds the part most write-ups skip: the break-even volume below which a dedicated GPU endpoint costs more than the LLM it replaces. You also get a TypeScript shadow-mode router that I ran on Node.js 22, so you can copy the rollout pattern without copying the hype.
What did Amazon actually change when it swapped Sonnet for a decision model?
Amazon replaced the model behind one closed-choice step, not an agent. The ticket system sorted escalations, capacity requests, alarms and security findings into 20 categories. Before the change, Sonnet 4.6 did this with a long system prompt and JSON output. After it, the E4B build of Strands Decider, a Gemma 4-based decision model, ran on a SageMaker ml.g5.xlarge instance, according to DevelopersDigest’s summary of the Strands post.
The vendor-reported numbers, from the Strands Agents write-up as summarized by DevelopersDigest, are:
- Average call time fell from 4,215 ms to 897 ms.
- Input tokens per ticket fell from 6,554 to 2,009.
- Output tokens per ticket fell from 170 to 1.
- Production Lambda duration fell from an average of 3,312 ms to 748 ms.
Notice the output column. Sonnet spent 170 tokens per ticket to emit a label wrapped in JSON. A decision model emits one choice. That gap is the whole thesis.
Decision model vs LLM: how do they differ as classifiers?
A decision model scores options you supply; an LLM classifier generates text you then have to parse. The Strands Decider launch post (October 1, 2026) describes the design: take a pretrained model, remove the language-model head, and add a small pointer head that scores each offered option, per DevelopersDigest’s reading of it.
That buys you three practical things. You get a probability per option instead of a string. You get a confidence score you can threshold. And you cannot get malformed JSON, because there is no JSON. If you have fought broken model output before, our guide on LLM structured outputs in Node.js shows how much code exists only to repair it.
The trade-off is stated plainly by its authors: the model is significantly worse than reasoning models at complex problems and is unsuited to coding, chatbots and summarization. The 2B model’s median decision latency was around 115 ms on an RTX 3090, which is also vendor-reported.
Trying the open 2B model from Python
The launch post documents a pip install and a CLI call. This is the documented example, not something I ran (the sandbox has no GPU):
pip install strands-decider
strands-decider ask StrandsAgents/strands-decider-2B-hobson-v19 \
--state "Help! My payouts have been failing for 3 days! " \
--choice "Which team should handle this?=billing,sales,retail"
The documented output ranks billing highest. Amazon’s production result used the larger E4B build, so treat the 2B model as a starting point for your own tests. Code, recipe and evaluations live in the strands-labs/strands-decider repository.
Is the 98% cost cut in the decision model vs LLM story real?
The arithmetic holds at list prices, but only at high, steady volume. Sonnet 4.6 was priced at $3 per million input and $15 per million output tokens, and the decision model at $1.408 per instance-hour, per the summary of the Strands post. Independent pricing write-ups give the same $1.408 per hour for ml.g5.xlarge in us-east-1.
We re-ran the numbers in Node.js 22.23.4 rather than trusting the headline:
// cost.mjs — run with: node cost.mjs
const perTicket = (inTok, outTok, inPrice, outPrice) =>
(inTok * inPrice + outTok * outPrice) / 1e6;
const sonnet46 = perTicket(6554, 170, 3, 15); // $ per ticket
const hourly = 1.408; // ml.g5.xlarge, us-east-1
const monthlyAlwaysOn = hourly * 730;
const ticketsPerHour = 3600 / 0.640; // ~640 ms per request
console.log(sonnet46.toFixed(5)); // 0.02221
console.log((hourly / ticketsPerHour * 1e6).toFixed(0)); // 250 ($ per 1M tickets, full use)
console.log(monthlyAlwaysOn.toFixed(2)); // 1027.84
console.log(Math.round(monthlyAlwaysOn / sonnet46)); // 46274 (break-even tickets/month)
Output from our run: $0.02221 per Sonnet ticket, 5,625 tickets per instance-hour, $250 per million tickets at full use, and $1,027.84 per month for one always-on instance. The Strands team budgets about $500 per million for burst headroom, which gives $0.0005 per ticket and the roughly 44x ratio. That is a 97.7% reduction, which rounds to the reported 98%.
The break-even most write-ups skip
Here is the insight the coverage I read leaves out. A token-priced LLM costs nothing when idle. A dedicated endpoint costs $1,027.84 a month even at zero traffic. At Sonnet 4.6’s per-ticket cost, you need about 46,300 tickets a month just to break even on one instance.
The bar rises as LLM prices fall. Anthropic lists Sonnet 5.5 at $2 input and $10 output per million tokens. With the same token counts, that is about $0.0148 per ticket, and break-even climbs to roughly 69,400 tickets a month. Anthropic also advertises up to 90% savings with prompt caching and 50% with batch processing. A 6,500-token prompt that is mostly a static label guide is exactly what caching targets, so price that option before buying GPUs. Our Bedrock prompt caching guide for Node.js walks through the setup.
My recommendation: below tens of thousands of decisions a month, optimize the LLM call first. Above that, a decision model or fine-tuned classifier starts to pay for itself.
How do you roll out a decision model safely in Node.js?
Copy Amazon’s three-mode rollout, not its model. A flag-driven rollout turns the decision model vs LLM debate into data from your own traffic. The team read a setting from AWS AppConfig on every message: BEDROCK (LLM only), DUAL (both run, LLM result applied, outputs compared) and DECIDER (new model only). Rolling back was a 45-second config change. AppConfig supports this natively through feature flag configuration profiles.
- Create a flag with three values:
BEDROCK,DUAL,DECIDER. - Ship the router below with the flag set to
BEDROCK. - Switch to
DUALand log every disagreement with the decision model’s confidence. - Review disagreements by hand and fix label criteria, not just thresholds.
- Switch to
DECIDERwith a low-confidence fallback to the LLM.
This router uses mock classifiers so it runs anywhere. We executed it on Node.js 22.23.4:
type Mode = 'BEDROCK' | 'DUAL' | 'DECIDER';
type Ticket = { id: number; text: string };
type Disagreement = { id: number; llm: string; decider: string; conf: number };
declare function llmClassify(t: Ticket): Promise<{ label: string }>;
declare function deciderClassify(t: Ticket): Promise<{ label: string; confidence: number }>;
export async function classify(t: Ticket, mode: Mode, log: Disagreement[]): Promise<string> {
if (mode === 'BEDROCK') return (await llmClassify(t)).label;
if (mode === 'DECIDER') {
const d = await deciderClassify(t);
// Low confidence? Fall back to the LLM instead of guessing.
return d.confidence < 0.5 ? (await llmClassify(t)).label : d.label;
}
// DUAL: run both in parallel, apply the LLM, record disagreements.
const [l, d] = await Promise.allSettled([llmClassify(t), deciderClassify(t)]);
if (l.status !== 'fulfilled') throw l.reason;
if (d.status === 'fulfilled' && d.value.label !== l.value.label) {
log.push({ id: t.id, llm: l.value.label, decider: d.value.label, conf: d.value.confidence });
}
return l.value.label;
}
Raw output from our mock run, where the fake decider fails on long tickets:
BEDROCK ["billing","capacity","alarm","security"] disagreements: [] 181ms
DUAL ["billing","capacity","alarm","security"] disagreements: [{"id":2,"llm":"capacity","decider":"none","conf":0.41}] 178ms
DECIDER ["billing","capacity","alarm","security"] disagreements: [] 80ms
Two details matter. Promise.allSettled means a decider outage never breaks DUAL mode. And the confidence fallback in DECIDER mode caught the long ticket our mock got wrong. If the fallback path calls a paid model, wrap it in a spend cap like the ones in our post on AI agent budget guards.
What gotchas did the Amazon team hit with the decision model?
Three, and each will bite you too. First, context. The label descriptions were longer than the model’s 4,096-token window, so each of the 20 labels was rewritten as a one- or two-sentence criterion and sent as options, plus a “none” option.
Second, truncation. On long tickets the model drops the oldest comments to fit. If your signal sits in the first message, that silently hurts accuracy. Put the original description first and trim the middle instead.
Third, serialization. The authors found that JSON key order changed output probabilities by up to 0.19, so they now send sorted keys. In Node.js, build the state string from a sorted copy of the object, and do the same in Python with json.dumps(state, sort_keys=True). Deterministic input is cheap insurance.
When should you keep the LLM instead of a decision model?
Keep the LLM when the label set changes often, when volume is low, when the input will not fit in a few thousand tokens, or when you need a written reason. Those are the conditions where the decision model vs LLM trade-off flips. In the Hacker News discussion, several commenters noted that classification never needed an LLM, and that TF-IDF plus a linear model is fast and cheap. Others countered that LLMs are useful for proofs of concept and for generating training data.
Research backs the cheap side for stable tasks. A February 2026 arXiv study found fine-tuned BERT-family encoders matched or beat zero- and few-shot LLM prompting on four benchmarks at one to two orders of magnitude lower cost and latency. So the real ladder has three rungs: LLM for discovery, decision model for zero-training closed choices, and a fine-tuned classifier once you have labels. For where this step sits inside a larger agent, see our single vs multi-agent orchestration guide.
| Factor | Frontier LLM (Sonnet 4.6) | Decision model (Strands Decider E4B) | Fine-tuned encoder (BERT family) |
|---|---|---|---|
| Output | Generated text/JSON, 170 tokens per ticket | One option + confidence, 1 token | Class probabilities |
| Avg call time (Amazon workload) | 4,215 ms | 897 ms | Not measured in this case |
| Cost basis | $3 / $15 per 1M tokens | $1.408 per instance-hour (us-east-1) | Your hosting + labeling effort |
| Cost per ticket | ~$0.0222 | ~$0.0005 at high volume | 1–2 orders of magnitude below LLM prompting (arXiv 2602.06370) |
| Idle cost | $0 | $1,027.84/month per always-on instance | Depends on hosting |
| Needs labeled data | No | No (options written as criteria) | Yes |
| Label set changes | Edit the prompt | Edit the options | Retrain |
| Context limit pain | Low | 4,096 tokens; oldest text dropped | Typically short inputs |
The NexGismo verdict: The decision model vs LLM result is real but narrow. It proves that closed-choice steps are overpaying when they run on a generative model at high volume. It does not prove a 2B or 4B model can replace your agent’s main model. I’d price prompt caching first, then run a decision model in DUAL mode on a week of real traffic before trusting any vendor number.
- Amazon engineers replaced Sonnet 4.6 with Strands Decider E4B on a 20-label internal ticket classifier.
- Vendor-reported cost per ticket fell from about $0.0222 to about $0.0005, roughly 44 times cheaper.
- Average call time fell from 4,215 ms to 897 ms, and output dropped from 170 tokens to 1.
- One always-on ml.g5.xlarge instance costs $1,027.84 a month in us-east-1, so low-volume workloads may not break even.
- A three-mode flag (LLM only, dual, decider only) lets you compare models on live traffic and roll back in seconds.
- Decision models fail on long inputs, changing label sets and anything that needs written output.
Frequently Asked Questions
What is a decision model in AI?
A decision model is a language model with its text-generation head replaced by a scoring head. You give it a state and a question with fixed options, and it returns a probability for each option plus a confidence score. It cannot write free text, so it suits routing, labeling, tool selection and guardrail checks.
Should I use an LLM for text classification?
Use an LLM when you have no labeled data, low volume, or labels that change often, because editing a prompt is faster than retraining. For high-volume, stable label sets, fine-tuned encoders or decision models are usually far cheaper and faster. Measure accuracy on your own tickets before switching.
How much cheaper was Strands Decider than Sonnet 4.6?
The Strands team reported about $0.0005 per ticket versus about $0.0222 for Sonnet 4.6 at list prices, roughly 44 times cheaper. That assumes steady high volume on an ml.g5.xlarge instance at $1.408 per hour. At low volume, the always-on instance cost can exceed the LLM bill.
Can a decision model replace my AI agent’s main model?
No. A decision model only picks from options you supply, and its authors say it is unsuited to coding, chat and summarization. It fits the rote choice points inside an agent, such as routing, tool selection and policy checks, while the main LLM still handles open-ended reasoning and writing.
How do I test a cheaper model without risking production?
Run both models in parallel behind a feature flag, apply only the current model’s answer, and log every disagreement with the new model’s confidence. Review the disagreements by hand, then switch to the new model with a low-confidence fallback. Amazon used AWS AppConfig and could roll back in about 45 seconds.
Amazon’s swap is a useful signal, not a template. The lesson is that a closed-choice step should not pay for generation it never uses, and that the safest way to prove a cheaper model is to run it beside the old one on live traffic. Start by measuring your own decision volume, then compare the idle cost of a dedicated endpoint with a cached LLM call. If the numbers favor a decision model, ship the three-mode router and let the disagreement log make the call. Have you moved a classification step off a frontier model yet? Tell me what broke in the comments.
Sources
- Strands Agents — Amazon saves millions by replacing Sonnet 4.6 with Strands Decider
- Strands Agents — Introducing Strands Decider 2B
- GitHub — strands-labs/strands-decider repository
- DevelopersDigest — Amazon Swapped Sonnet 4.6 for a Decision Model
- Anthropic — Claude Sonnet model page and pricing
- CloudBurn — Amazon SageMaker AI pricing (ml.g5.xlarge)
- AWS Documentation — Creating a feature flag configuration profile in AWS AppConfig
- arXiv — Cost-Aware Model Selection for Text Classification (2602.06370)
- Hacker News — discussion thread
