Choosing an LLM or model provider for enterprise use

Most teams arrive asking “which model is best?” The honest answer is that the leaderboard changes every few months, and the model that tops it is rarely the one you should standardise on. The better question is: which provider can carry this specific workload, at this data sensitivity, at this volume, for the next two years — and how expensive is it to change our mind later?

That reframing matters because model choice is mostly reversible and provider choice mostly is not. Swapping a model behind a well-designed abstraction is a week of prompt and evaluation work. Unwinding a data-residency commitment, a bespoke fine-tune, or an application built directly against one vendor's proprietary agent framework is a quarter.

What follows is the evaluation framework we use when advising on LLM development and infrastructure. It is deliberately weighted toward operational reality rather than benchmark scores.

Score each candidate provider from 1 to 5 on six axes, then weight the axes according to your own constraints rather than accepting a generic total. A regulated finance workload and an internal marketing drafting tool should not use the same weights.

• Cost structure. Not headline price per million tokens — the shape of the bill at your expected volume, including output tokens (usually priced higher than input), context re-sending, caching discounts, and whether batch or off-peak tiers exist.
• Data handling. Where data is processed, what retention applies, whether your inputs can be used for training, what contractual commitments exist in writing, and whether a regional deployment is available.
• Latency and throughput. Time to first token and tokens per second at your concurrency, plus published rate limits and how quickly they can be raised.
• Task accuracy. Measured on your own tasks with your own data, never on a public benchmark.
• Portability. How much of your application would survive a provider change — measured in files, not in feelings.
• Operational maturity. Status history, deprecation notice periods, support responsiveness, and whether model versions are pinnable rather than silently updated beneath you.

How to weight it

• Regulated or personal data in the prompts: data handling at double weight, and treat any score below 4 as disqualifying regardless of the total.
• Customer-facing and interactive: latency at double weight — a technically better answer arriving eight seconds late is a worse product.
• High-volume background processing: cost structure at double weight, latency at half.
• Long-lived platform investment: portability and operational maturity at double weight.

Per-token pricing is published and easy to compare, which is exactly why it misleads. The bill is driven by architecture far more than by unit price. Three effects dominate.

Context is the real cost driver. A retrieval-augmented request that stuffs twenty documents into the prompt to answer one question pays for those twenty documents on every single call. Tightening retrieval so it returns four relevant passages instead of twenty is usually a larger saving than any provider switch.

Output tokens cost more than input tokens. Systems that ask models to restate their reasoning at length, or to return whole documents rather than the changed fields, spend disproportionately on the expensive side of the meter.

Model tiering beats model choice. Most workloads contain a mix: classification and routing that a small cheap model handles perfectly, and a minority of genuinely hard reasoning that warrants a frontier model. Routing by difficulty typically moves the bill more than negotiating rates.

Before comparing providers, build a small cost model: expected requests per day, average input and output tokens per request measured from a real prototype, and a peak-month multiplier. Then compare. A provider that is more expensive per token but supports prompt caching for your large fixed system context can be cheaper in practice.

Everything else on the scorecard is a trade-off. This one is a gate. If your prompts will contain customer records, health information, financial detail or unreleased commercial material, get specific written answers before technical evaluation begins.

1. In which countries or regions is the request processed, and can that be pinned rather than routed dynamically?
2. How long are prompts and completions retained, for what purpose, and can retention be reduced to zero for your account?
3. Is there an unambiguous contractual commitment that your data is not used to train or improve the provider's models?
4. What sub-processors sit behind the API, and are you notified before that list changes?
5. What logging exists on the provider side, who inside the provider can access it, and under what process?
6. What certifications or audit reports can be produced on request, and what is their current date?

Where the answers do not clear your bar, the options are a regional or private deployment through your existing cloud provider, self-hosted open-weight models, or redaction and tokenisation of sensitive fields before the call. All three are workable; all three cost more than the default path, and that cost belongs in the business case from the start rather than as a surprise during security review. Teams in finance and healthcare should assume this gate shapes the whole architecture.

Public benchmarks measure general capability on problems that are not yours. They are useful for excluding obviously unsuitable models and useless for choosing between the top few. Build a private evaluation set instead — it is the single highest-value artefact of the whole selection exercise, and it keeps paying off long after the decision.

Building a usable evaluation set

• Collect 50 to 150 real examples of the task, drawn from actual work rather than written for the test.
• Include the awkward ones deliberately: ambiguous inputs, missing fields, mixed languages, the cases your experts argue about.
• Have a domain expert record the correct or acceptable answer, and note where more than one answer is acceptable.
• Define what failure means for this task — a wrong extracted figure is not the same severity as a slightly clumsy summary.
• Run every candidate model against the same set, with the same prompts, and record cost and latency alongside the score.

Two things fall out of this that a benchmark never gives you. First, the shape of the failures — one model may be less accurate overall but fail safely by declining, while another fails confidently, which is far worse in a system without human review. Second, a regression suite: when the provider ships a new version or you change a prompt, you can tell in an hour whether anything got worse. Our guide to enterprise RAG goes further into evaluating retrieval quality specifically.

This is usually framed as an ideological question and answered as an economic one. The practical distinction: proprietary API models offload the infrastructure, capacity planning and upgrade path onto the vendor; open-weight models hand you control over data residency, versioning and unit economics along with the responsibility for all three.

Open-weight tends to win when

• Data cannot leave your infrastructure or a specific jurisdiction.
• Volume is high and steady, so reserved GPU capacity beats per-token pricing at scale.
• The task is narrow and a smaller fine-tuned model matches frontier quality on it.
• You need a model version frozen for years for reproducibility, audit or validation reasons.

Proprietary APIs tend to win when

• Volume is low, spiky or unpredictable — idle GPUs are the most expensive kind.
• The task needs the strongest available reasoning and that frontier keeps moving.
• You have no team to own inference infrastructure, and hiring one is not the plan.
• Time to first working version matters more than long-run unit cost.

The honest middle path, and the one most organisations end up at, is both: an open-weight model self-hosted for the high-volume sensitive path, and a proprietary API for the small share of hard cases, behind one internal interface. That requires designing the interface first — which brings us to lock-in.

Total provider neutrality is a myth that costs real money; every abstraction layer built to achieve it gives up the features you are paying for. The goal is not zero lock-in but known, bounded, priced lock-in.

• Keep the call site thin. Application code should ask for a capability — classify, extract, summarise — not construct provider-specific payloads inline across forty files.
• Own your prompts and evaluations. They are your intellectual property and the bulk of the real work. Version them in your own repository.
• Own your data layer. Embeddings, chunking and the vector store should be yours and re-computable. Re-embedding a corpus on a new model is a job; recovering documents from a vendor-managed store you cannot export is a crisis.
• Adopt proprietary orchestration deliberately. Vendor agent frameworks and hosted tool-calling can be excellent. Use them where they earn their keep, and record that decision as a deliberate trade rather than letting it accumulate by default.

A useful exercise: write down, in a paragraph, what a switch would actually involve today. If you cannot, the lock-in is already larger than you think.

Selection paralysis is the most common failure here — months of comparison producing no running system. This sequence is deliberately short.

1. Days 1–2. Write down the workload: the tasks, the data sensitivity, expected daily volume, latency requirement and who is accountable for the outcome.
2. Days 3–4. Apply the data-handling gate. Exclude providers that cannot meet it in writing. This usually shortens the list more than anything else.
3. Days 5–7. Build the evaluation set with a domain expert. Do not skip this to save time; everything downstream depends on it.
4. Days 8–10. Run two or three surviving candidates against it, recording accuracy, failure shape, latency and measured cost per request.
5. Days 11–12. Score the six axes with your weights, and model the annual bill at realistic volume including a growth multiple.
6. Days 13–14. Decide, write down the reasons and the assumptions that would invalidate them, and set a review date six months out.

That last step is the one that makes the decision safe. A choice with recorded assumptions and a scheduled review is a manageable commitment. A choice made once and never revisited becomes the constraint everyone complains about three years later.