Deploying LLMs in Production

Deploying an LLM in production is a different discipline from calling one in a notebook: you have to balance latency, throughput, cost, and reliability while serving real traffic that never stops, and the gap between a working demo and a dependable service is wide. The core decisions are whether to use a hosted API or self-host, which model to run, how to serve inference efficiently, how to scale and monitor it, and how to keep it safe and affordable. This guide maps that whole journey—from the first architectural choice to operating a reliable, observable, cost-controlled service.

The first decision: managed API or self-host

Everything starts with one fork: call a model through a managed API (OpenAI, Anthropic, Google, and others) or self-host an open model on your own or rented GPUs.

A managed API is the fastest path to shipping. There's no infrastructure to run, you pay per token, it scales instantly, and you always have access to frontier models. The costs are ongoing per-token spend that grows linearly with usage, some vendor lock-in, and the fact that your data passes through a third party—a real concern for sensitive workloads. Self-hosting flips the tradeoffs: you gain full control, data privacy, no per-token vendor bill, and—at sufficient scale—lower cost per request. But you take on the GPUs, the serving stack, and the operational burden of keeping it all alive.

The practical rule almost every experienced team follows: start with a managed API while you're validating, and self-host only once volume, economics, or privacy requirements justify it. Building your first version on an API lets you focus on the product—the actual work of building applications with LLMs—rather than on infrastructure. When you do consider bringing inference in-house, weigh it honestly: the deep dive on self-hosting AI models and the mechanics of running LLMs locally both matter, and so does a cost that surprises people—the MLOps engineer needed to operate a self-hosted stack often costs more than the hardware itself. Self-hosting isn't cheaper by default; it's cheaper at scale, if you have the team to run it.

Choosing and right-sizing your model

Which model you deploy is one of the biggest levers you have over cost, latency, and quality all at once—and the instinct to reach for the largest, most capable model is usually wrong.

Bigger models cost more per token, respond more slowly, and need more expensive hardware. For many production tasks—classification, extraction, summarization, routing, structured generation—a smaller, cheaper, faster model matches a large one's quality on that specific job. Understanding the tradeoffs between small and large language models is central to deploying economically, because right-sizing the model is one of the largest cost savings available, often bigger than any infrastructure optimization.

The way to choose well is to match the model to the actual use case rather than to benchmarks or hype—a process worth doing deliberately, as covered in choosing an LLM for your use case—and then to test your shortlisted candidates on your tasks before committing. A model that tops public leaderboards may underperform on your specific data. One more technique expands your options: quantization, which compresses a model's weights to lower precision (such as 4-bit), lets you run a larger model on smaller, cheaper GPUs with modest quality loss—often the difference between needing an expensive card and a mid-range one.

How LLM inference works in production

Serving a model to real users efficiently is a specialized problem, and the piece of software that does it is the inference engine—it loads the model weights, manages GPU memory, and handles many concurrent requests.

As of 2026, vLLM has become the de facto standard for self-hosted production inference. Developed at UC Berkeley and used in production by companies including Meta, Mistral AI, IBM, and Stripe, it delivers roughly 14–24× the throughput of naive serving on the same hardware through two key innovations. PagedAttention manages the KV cache (the memory of the tokens generated so far) like an operating system manages virtual memory—in non-contiguous pages—eliminating 60–80% of the memory waste that plagued older approaches. Continuous batching schedules work at the token level, dynamically swapping finished requests out and new ones in rather than waiting for the slowest request in a batch to complete. vLLM also exposes an OpenAI-compatible API, which means your application code doesn't need to know whether it's talking to vLLM or OpenAI—a huge practical convenience. Alternatives exist for specific needs: Hugging Face TGI for familiar integration, NVIDIA TensorRT-LLM for maximum GPU optimization, SGLang for certain workloads, and Ollama for simple local use.

Two caching ideas matter for cost and speed. The KV cache avoids recomputing earlier tokens on every step, trading compute for memory. Prompt (prefix) caching reuses the computation for a shared prefix across requests—so a long system prompt sent with every call is processed once, not repeatedly, which can dramatically cut cost.

To run any of this well, you monitor a specific set of metrics:

Scaling, reliability, and observability

Getting a model to respond once is easy; keeping it responsive under real, variable traffic is the actual work.

Scaling means adding capacity as load rises. In practice this looks like running multiple replicas of your inference service behind a load balancer, with autoscaling driven by a signal like queue depth, on an orchestrator such as Kubernetes. GPUs are expensive, so you want enough to meet demand without paying for idle hardware—which is why predictable scaling on real signals matters.

Reliability means the service degrades gracefully instead of falling over. Build in timeouts, automatic retries, and fallbacks—routing to a smaller model or a backup provider when your primary is overloaded or down—plus sensible rate limiting so one client can't starve the rest.

Observability is non-negotiable, because you cannot operate what you cannot see. Log and trace every request, monitor the metrics above along with error rates and cost, build dashboards (Prometheus and Grafana are the common pairing for infrastructure, with tools like Langfuse for request tracing and quality scoring), and alert on the signals that predict user pain, such as rising P95 latency or error rate.

A typical production stack ties these together:

1. A request layer (often FastAPI or similar) handling authentication, rate limiting, prompt templating, and streaming responses to clients.
2. An inference engine (vLLM) serving the model behind an OpenAI-compatible API, so the application stays engine-agnostic.
3. GPUs sized to the model—for example an L40S 48GB (roughly $1.20–1.80/hour rented) for 7B–34B models, or an A100 80GB for a 70B model.
4. Autoscaling and load balancing on Kubernetes, scaling on queue depth to match traffic.
5. An observability layer capturing metrics, traces, and quality scores, with alerting.
6. Reliability layers—retries, timeouts, and fallbacks—wrapping the whole thing.

Load-test this stack under realistic traffic before routing real users to it; the demo-to-production gap is where most deployments stumble.

Managing cost

LLM inference cost is the thing that surprises teams most, and it can quietly ruin a product's economics if ignored. The good news is that a handful of levers control most of it.

The biggest is model right-sizing—running the smallest model that does the job well, as covered above, since a smaller model is cheaper and faster on every single request. Next is token discipline: shorter prompts and responses cost less, and trimming a bloated system prompt or capping output length pays off on every call. Prompt caching slashes the cost of repeated prefixes. On self-hosted setups, continuous batching raises the number of requests each GPU serves, and reserved GPU instances cut cloud costs by roughly 30–40% versus on-demand. And the API-versus-self-host choice is itself a cost decision: API pricing scales linearly with usage, so a high-volume workload can reach a break-even point where self-hosting is dramatically cheaper—Stripe, for instance, reported a 73% cost reduction after moving inference to vLLM.

Because cost optimization has so many interacting levers, it deserves focused attention; the dedicated guide to AI inference cost optimization goes deeper. The headline, though, is simple: measure cost per request from day one, and treat model choice and token count as your primary dials.

Safety, guardrails, and ongoing evaluation

A model in production faces untrusted users and real consequences, so it needs guardrails around it.

On the input side, treat everything users send as untrusted—defend against prompt injection (malicious instructions hidden in user input or retrieved data), filter abusive or out-of-scope requests, and handle personal data carefully. On the output side, screen for harmful, off-brand, or incorrect responses before they reach users, especially in high-stakes domains, and keep a human in the loop where the cost of a mistake is high. Because LLM output is probabilistic, guardrails and monitoring aren't optional extras—they're how you make a non-deterministic system safe to ship.

Deployment also isn't finished at launch. Prompts and models need versioning and regression testing, because a model upgrade or a prompt tweak can silently break behavior that worked yesterday. Evaluate continuously: score quality in production, watch for drift and degradation, and connect real failures back into your test suite. Doing this rigorously depends on tracking the right AI model evaluation metrics, so you can tell whether a change actually improved things or quietly made them worse. Production LLM systems are living systems, and the teams that succeed treat evaluation as a continuous practice rather than a launch-day checkbox.

Common mistakes to avoid

• Self-hosting too early. Standing up GPUs and a serving stack before you have the volume to justify it burns time and money. Start with an API.
• Ignoring cost until the bill explodes. Inference cost compounds silently. Measure cost per request from the start and right-size your model.
• Deploying an oversized model. Paying for a frontier model on a task a small one handles wastes money and adds latency on every request.
• Running blind. No observability means you learn about outages from users. Instrument latency, errors, and cost before launch.
• Skipping load testing. A stack that works in a demo can collapse under concurrent traffic. Test under realistic load first.
• No fallbacks. A single provider or model with no backup is a single point of failure. Build in retries and graceful degradation.
• Neglecting guardrails. Shipping without input/output safety invites prompt injection and harmful output. Treat user input as untrusted.
• Not evaluating changes. Model and prompt updates can silently regress quality. Test before and after every change.

Frequently asked questions

Should I use an LLM API or self-host? Start with a managed API (OpenAI, Anthropic, and others) while validating your product—it's faster to ship, requires no infrastructure, and scales instantly, at the cost of per-token pricing and sending data to a third party. Move to self-hosting once your volume, cost, or privacy needs justify it. Remember that self-hosting is cheaper only at scale, and the engineering effort to run it often exceeds the hardware cost.

What is vLLM and why is it used for LLM deployment? vLLM is an open-source inference engine, developed at UC Berkeley, that has become the standard for self-hosted production LLM serving. It delivers roughly 14–24× the throughput of naive serving through PagedAttention (efficient KV-cache memory management) and continuous batching (token-level scheduling), and it exposes an OpenAI-compatible API so application code stays portable. Companies like Meta, Mistral, IBM, and Stripe use it in production.

What metrics matter when deploying LLMs in production? The key ones are time to first token (TTFT), which drives perceived responsiveness; tokens per second (TPS) for throughput; P99 latency for the worst-case experiences; queue depth as an overload and autoscaling signal; and cost per request or per million tokens, which determines economic viability. Monitor these alongside error rates, and alert on the ones that predict user-facing problems, like rising P95 latency.

How do I reduce LLM inference costs? The biggest lever is running the smallest model that does the job well, since it's cheaper and faster on every request. Then reduce token counts with shorter prompts and capped outputs, use prompt caching for repeated prefixes, and—if self-hosting—use continuous batching and reserved GPU instances (roughly 30–40% cheaper than on-demand). At high volume, self-hosting open models can cut costs dramatically compared to per-token API pricing.

How do I keep a production LLM safe and reliable? Wrap the model in guardrails: treat user input as untrusted to defend against prompt injection, filter harmful or off-scope requests, screen outputs before they reach users, and keep humans in the loop for high-stakes decisions. For reliability, add timeouts, retries, and fallbacks to a backup model or provider. Because output is probabilistic, pair these with continuous monitoring and evaluation rather than assuming launch-day quality holds.

The takeaway

Deploying LLMs in production is fundamentally about balancing four forces—latency, throughput, cost, and reliability—while serving traffic that never stops. Get the sequence right: start on a managed API, right-size your model as your biggest cost lever, serve efficiently with an engine like vLLM when you self-host, and wrap it all in observability, reliability layers, guardrails, and continuous evaluation. The recurring lesson is that the gap between a demo and a dependable service is wide, and it's closed by measuring everything—cost, latency, and quality—rather than assuming. Your next step is to instrument cost per request and TTFT on whatever you're running today, because you can't optimize a production LLM you aren't measuring, and those two numbers will tell you exactly where to focus first.