How to Choose an LLM for Your Product – A Decision Framework
Stop starting from benchmarks. A practical sequence for picking a model based on your task, your latency budget, and what failure actually costs you.
There is no best LLM, only the best fit — which is true, unhelpful, and repeated everywhere. So here is the actual sequence, in the order the decisions matter.
The framing that saves the most time: you are not picking a model, you are designing a routing policy. Almost every production system worth building ends up using more than one.
Step 1: Write the eval before you look at models
Twenty real examples from your domain with known-good outputs. Not a benchmark, not a vibe check — twenty cases where you can say unambiguously whether the output is acceptable.
This feels like a detour and is the highest-leverage thing you can do. Without it you cannot tell whether a cheaper model is adequate, which means you will default to the most expensive one and never find out. With it, every subsequent decision becomes measurable.
Public leaderboards will not do this job for you. Single-shot benchmark scores systematically overestimate reliability, because they measure whether a model can do something once rather than whether it does so consistently across varied attempts. Leadership also rotates every few weeks.
Step 2: Establish the task is possible at all
Start with the strongest model you can afford and get it working. Do not optimise cost yet.
The reason is diagnostic clarity: debugging a capability problem and a cost problem at the same time is miserable, and you will not know which one you are looking at. Prove the task is achievable, then make it cheap. That order is much faster in practice.
Step 3: Classify what kind of problem this is
The answer determines everything downstream.
Is it multi-step? Reasoning models help substantially on problems requiring several dependent inferences — debugging, planning, maths, long-horizon agent work. They actively hurt on extraction, classification, and moderation, where you pay latency and tokens for deliberation that adds nothing.
Is latency user-visible? Anything with always-on thinking is disqualified from autocomplete, live suggestions, or interactive chat, regardless of price. Some current models cannot turn reasoning off at all.
What does a wrong answer cost? This is the question that justifies premium tiers. For legal analysis, medical research, or code that ships unreviewed, error cost dwarfs token cost and you should buy capability. For draft generation with a human in the loop, it does not.
How much context do you genuinely need? Million-token windows are now standard, which makes this less of a constraint than it was — but long prompts get expensive, cross repricing thresholds, and recall degrades. Retrieval usually beats stuffing.
Step 4: Route downward against your eval
Now go back to the eval and try the cheaper tiers. Run the mid-range, then the budget tier.
You will usually find the cheap tier handles more than expected. The 2026 price cuts pushed genuinely capable models into brackets where they outperform the previous generation’s flagships — one budget tier now beats last generation’s top-end model on coding evals at roughly a fiftieth of the frontier price.
The pattern that falls out of this:
- Classify difficulty cheaply — heuristics on length and task type get you far.
- Send the easy majority to a budget tier.
- Verify cheaply — schema validation, a test run, a confidence threshold.
- Escalate to the expensive tier only on failure.
Because escalation only fires on the hard minority, blended cost lands near the cheap tier while your quality ceiling stays near the flagship. The price spread within a single provider’s family is often 20x or more, which is what makes this worth the added complexity.
Step 5: Decide how much lock-in you accept
A question people postpone and regret postponing.
Staying within one provider’s family buys you a single integration, consistent behaviour across tiers, and simple credential management. It also means a price change or capacity problem hits your whole stack at once.
Spreading across providers costs engineering effort and gains you leverage plus a fallback. Open-weight models go further — portability between inference vendors, or self-hosting if your volume is steady enough to justify the GPU. Note that self-hosting wins on sustained throughput and loses badly on spiky traffic, since you pay for the GPU idle or busy.
Step 6: Then optimise the bill
Only once behaviour is stable. In rough order of impact: prompt caching (40–80% of input cost), difficulty routing, cutting output tokens, batching async work. Details here.
What to ignore
Benchmark leadership. It changes constantly and rarely predicts performance on your specific task.
Parameter counts. Providers mostly stopped publishing them, and they were never a good proxy for capability.
Context window as a headline. Capacity and economical capacity are different claims.
“Best model for X” listicles. Including ours, honestly — our coding page leads with a benchmark table and immediately tells you to validate on your own repository, because that is the truthful answer.
The one-line version
Build the eval, prove it works with an expensive model, then route downward until quality breaks. Everything else is detail.