Building Applications With LLMs

Building applications with LLMs is less about the model and more about the engineering around it: you are wrapping a powerful but non-deterministic component—one that can be fluent, wrong, and slow in the same response—in a system that makes it reliable enough to ship. The craft is a sequence of decisions: whether to use an LLM at all, how much capability to add and in what order, how to architect for a probabilistic core, how to evaluate it, and how to run it in production. This guide is the map of that whole territory, with pointers to the deeper articles on each part.

First, decide whether you even need an LLM

The most useful and least fashionable question to ask before building with an LLM is whether you should use one here at all. LLMs are expensive per call, slow compared to ordinary code, non-deterministic, and capable of confidently fabricating output. For a task that a regular program solves exactly—validating an email format, summing a column, matching an exact rule—an LLM is the wrong tool: slower, costlier, and less reliable than ten lines of deterministic code.

LLMs earn their cost on a specific class of problems: understanding and generating natural language, handling fuzzy or unstructured input, summarizing, classifying open-ended text, extracting structure from mess, and reasoning over ambiguous situations where you cannot enumerate the rules in advance. The honest framing is that an LLM is a probabilistic reasoning-and-language engine, and you should reach for it when a task genuinely needs that and not because it's the interesting option.

A strong pattern in real applications is hybrid: use deterministic code for the parts that are deterministic, and call the LLM only for the genuinely fuzzy step. A support-ticket router might use plain code to look up the customer's account and plan, and the LLM only to classify the free-text complaint. That division keeps the system faster, cheaper, and easier to trust than routing everything through the model.

The capability ladder: prompt, RAG, fine-tune, agents

Once you've decided an LLM belongs in the task, the central architectural decision is how much machinery to add around it. There's a natural ladder of techniques, each more capable and more complex than the last. The discipline—and this is where most teams overspend—is to climb only as high as the task requires, because every rung adds cost, latency, and failure modes.

Start at the bottom and stop as soon as the task is solved. Most production LLM features never need to leave the first two rungs.

Prompting: start here, always

Prompting is simply giving the model well-crafted instructions, context, and examples to get the output you want, and it is the first thing to exhaust before anything heavier. A surprising share of "we need to fine-tune" problems are actually prompt problems. The core techniques—being specific, providing context and a role, showing input-output examples (few-shot), and demanding structured output—are covered in prompt engineering techniques. For tasks involving multi-step reasoning, instructing the model to work through the problem before answering (covered in the chain-of-thought prompting guide) measurably improves accuracy. Prompting is cheap, instant to iterate, and often enough on its own—so prove the task can't be solved by prompting before you climb higher.

RAG: give the model your data

The base model only knows what it learned in training—nothing about your private documents, and nothing more recent than its cutoff. Retrieval-augmented generation (RAG) fixes this by retrieving relevant pieces of your data at query time and inserting them into the prompt, so the model answers from real, current, specific information instead of its frozen training or its imagination. It's the standard architecture for question-answering over private knowledge, documentation assistants, and anything that must cite real sources, and it directly reduces hallucination by grounding answers in retrieved text. How retrieval, chunking, and grounding actually work is the subject of retrieval-augmented generation explained. RAG is usually the right next step above prompting because it adds knowledge without touching the model, and because its failures are inspectable—you can see exactly which chunks were retrieved.

Fine-tuning: change the model's behavior

Fine-tuning continues training a model on your own examples to adjust its weights, which bakes a consistent behavior—a format, a tone, a narrow classification skill—into the model itself. The crucial and widely misunderstood point is what fine-tuning is for: it teaches behavior and style, not knowledge. If you need the model to know new facts, that's RAG; if you need it to respond in a specific way every time, that's fine-tuning. The two solve different problems and are often combined, a decision laid out in when to fine-tune vs use RAG. Fine-tuning is a real investment—you need a quality dataset, compute, and an evaluation loop—and the practical path for most teams is fine-tuning open-source LLMs with an efficient method like LoRA rather than full retraining. Reach for it only when prompting and RAG have genuinely hit their limits.

Agents: let the model act

At the top of the ladder, an agent is an LLM running in a loop: it reasons about a goal, calls tools (functions you expose to it), observes the results, and repeats until the task is done. This is what lets an LLM application do things—query a database, call an API, book something—rather than only produce text. The underlying mechanism is covered in the function-calling tutorial, and the broader concept in tool use in LLMs; the frameworks that orchestrate the loop are compared in AI agent frameworks compared. Agents are the most powerful rung and by far the least reliable: every step is a chance to go wrong, and errors compound across steps, so a 95%-reliable step chained ten times succeeds only about 60% of the time. Build an agent when the task genuinely needs multi-step autonomy—and not a moment before, because you are trading a large amount of reliability for that autonomy.

Architecting around a probabilistic core

Whatever rung you land on, the defining engineering challenge of an LLM application is that its core component is probabilistic: the same input can yield different outputs, and some fraction of outputs will be wrong or malformed no matter how good your prompt. Good LLM architecture is largely about containing that uncertainty.

Treat the LLM as one unreliable component

The most important mental shift is to stop thinking of the LLM as the application and start thinking of it as one unreliable service your application calls—like a flaky third-party API you must defend against. Your code surrounds it with validation, retries, fallbacks, and deterministic logic. The model proposes; your system disposes.

Get structured output you can actually use

Free-form text is hard to build on. For anything programmatic, constrain the model to return structured output—typically JSON matching a schema—using the structured-output or function-calling features that major providers now offer. This turns an unpredictable text generator into something closer to a typed function: you ask for specific fields and get them back parseable. Always validate the result against your schema anyway, because "usually valid JSON" still fails often enough to break a pipeline.

Manage the context window deliberately

Everything the model sees on a given call—instructions, retrieved data, conversation history, the user's input—lives in its context window, which is finite. Naively stuffing in everything degrades both quality and cost: models attend less reliably to the middle of a long context (the "lost in the middle" effect), and you pay for every token. Deliberate context management—retrieving only the most relevant chunks, summarizing or truncating history, putting the most important instructions where the model weights them most—is a real part of the engineering, not an afterthought.

Plan for latency and streaming

LLM calls are slow—often one to several seconds, longer for big models or long outputs, and multiplied in any agent loop that makes several calls. Design for it: stream tokens to the user so the response feels responsive as it generates, do work in parallel where you can, and cache aggressively. A perfect answer that takes fifteen seconds loses to a good one that starts appearing in one.

Guardrails: the model faces untrusted input

An LLM application in production is exposed to real users and sometimes to data from the open web, and the model will treat text in its context as potentially instructive. That creates a security surface ordinary apps don't have, and it is one of the most underappreciated parts of building with LLMs.

The signature risk is prompt injection: malicious instructions hidden in user input or in retrieved content ("ignore your previous instructions and…") that hijack the model's behavior. There is no complete fix, so you defend in layers: separate instructions from data clearly in your prompts, never grant the model more authority than the task needs, validate and sanitize anything it produces before acting on it, and treat its tool calls as untrusted input—never pass model output straight into a shell command, SQL query, or privileged action without checks.

Beyond injection, production guardrails include screening outputs for harmful, off-brand, or wrong content before they reach users, rate limiting, handling personal data carefully, and keeping a human in the loop for high-stakes actions. Because the system is probabilistic, these aren't optional polish—they're how you make a non-deterministic component safe to expose.

Evaluation is the real engineering work

Here is the part teams consistently underestimate: with a probabilistic system, you cannot tell whether a change helped by eyeballing a few outputs. A prompt tweak that fixes three examples may break five others you didn't check. Systematic evaluation is not a final QA step in LLM development—it is the development loop.

The practical core is to build an evaluation set: 20 to 100 real examples from your use case, each with a known-good output or a clear success criterion. You run every change against it and watch the score, rather than trusting your impression. What to measure and how—including the now-standard "LLM-as-judge" approach and its biases—is covered in AI model evaluation metrics. Specialized systems need specialized evaluation layered on top: a RAG system's retrieval quality and faithfulness are measured separately, as in evaluating RAG quality, and an agent's whole trajectory is scored, not just its final answer, as in evaluating AI agent reliability.

Wire evaluation into your workflow like tests: run it on every prompt, model, or retrieval change, and—most valuable of all—turn every production failure into a new eval case so the same bug can never silently return. The teams that ship reliable LLM features are the ones that made evaluation cheap and constant.

Choosing a model and shipping it

Two decisions bracket the build: which model powers it, and how you run it in production.

Model choice is a bigger lever on cost, latency, and quality than almost anything else, and the instinct to grab the most powerful model is usually wrong. A large frontier model is a sensible default while you're proving the task works, but many production jobs run well on a smaller, cheaper, faster model—a tradeoff examined in small vs large language models—and the right way to decide is to test candidates on your own eval set rather than trusting a leaderboard, as laid out in choosing an LLM for your use case. A powerful pattern is to route most requests to a cheap model and escalate only the hard ones to an expensive model.

Deployment is its own discipline: serving the model reliably under real traffic, with monitoring, scaling, retries, and fallbacks, whether you call a managed API or run an open model yourself. The full treatment is in deploying LLMs in production, with self-hosting AI models and running LLMs locally covering the own-infrastructure path. Two operational realities dominate: cost, which scales with every token and is controlled mostly by right-sizing the model and trimming tokens (see AI inference cost optimization), and the fact that the landscape moves fast—so keep your integration model-agnostic, routing calls through a thin abstraction rather than hardcoding one provider, so you can switch as better or cheaper options appear.

A reference architecture

Putting the pieces together, a typical production LLM feature looks like this end to end:

1. Input handling — receive the user's request; use deterministic code for anything that doesn't need the model.
2. Context assembly — retrieve relevant data (RAG) and assemble the prompt, managing what fits in the context window.
3. The model call — call a right-sized model, request structured output, and stream the response.
4. Validation — check the output against your schema and guardrails before using or showing it.
5. Action — if it's an agent, execute tool calls through code that treats the model's requests as untrusted; otherwise return the result.
6. Observability — log the full interaction (inputs, outputs, cost, latency) for monitoring and for building eval cases.
7. Evaluation loop — feed real interactions, especially failures, back into the eval set that gates every future change.

Notice how little of this is "the prompt." The prompt is one step; the engineering is everything around it.

Common mistakes

• Using an LLM where plain code would do. Non-determinism, cost, and latency for a task a deterministic function solves exactly. Reach for the model only on genuinely fuzzy problems.
• Climbing the capability ladder too fast. Jumping to fine-tuning or agents when prompting or RAG would solve it adds cost and failure modes for nothing. Start low, stop early.
• Confusing fine-tuning with knowledge. Fine-tuning changes behavior and style; it doesn't reliably teach facts. For knowledge, use RAG.
• Trusting the model's output unconditionally. It will produce malformed or wrong output some of the time. Validate structured output against a schema; sanitize anything before acting on it.
• Ignoring prompt injection. Treating user and retrieved text as safe instructions invites hijacking. Separate instructions from data, limit the model's authority, and never pass its output into privileged actions unchecked.
• Evaluating by vibes. Eyeballing a few outputs hides regressions in a probabilistic system. Build an eval set and measure every change against it.
• Over-stuffing the context window. More context isn't better—it degrades attention and raises cost. Retrieve and include only what's relevant.
• Hardcoding one provider and the biggest model. Lock-in plus overspend. Stay model-agnostic and right-size to the task.

Frequently asked questions

What does it take to build an application with an LLM? At minimum, a clear task the model is actually suited to, a way to call the model (a managed API or a self-hosted model), and code around it that assembles context, validates output, and handles errors. Beyond that, most of the work is engineering discipline: choosing the right level of complexity (prompting, RAG, fine-tuning, or agents), building an evaluation set, and adding guardrails. The model is one component; the application is everything around it.

Do I need RAG or fine-tuning to build with an LLM? Often neither. Start with prompting—clear instructions, examples, and context—which solves a large share of tasks on its own. Add RAG when the model needs facts it doesn't have, like your private or current data. Add fine-tuning only when you need consistent behavior or style that prompting can't reliably produce. Each step adds cost and complexity, so climb the ladder only as far as the task requires.

What's the hardest part of building LLM applications? Dealing with non-determinism. Because the same input can produce different and sometimes wrong outputs, you can't verify quality by spot-checking, and you must engineer validation, guardrails, and fallbacks around an unreliable core. The practical consequence is that systematic evaluation—building a test set and measuring every change against it—is the central discipline, and it's the step teams most often skip and most regret skipping.

How do I stop an LLM app from making things up? Ground it in real data with RAG so it answers from retrieved facts rather than its training, and ask it to cite sources you can check. Constrain outputs to structured formats you validate, keep prompts focused, and for high-stakes answers keep a human in the loop. You can reduce hallucination substantially but not eliminate it, so design the application so that an occasional wrong answer is caught or contained rather than acted on blindly.

How much does it cost to run an LLM application? It depends almost entirely on the model you choose and how many tokens each request uses, since API pricing is per token and frontier models can cost 10–50× more than small capable ones. The biggest cost levers are right-sizing the model to the task, trimming prompt and output length, caching, and routing easy requests to cheaper models. Measure cost per request from the start, because it scales with usage and can quietly dominate a product's economics.

The takeaway

Building applications with LLMs is fundamentally a discipline of engineering around a probabilistic component: the model is one unreliable part of a larger system, and the real work is deciding how much capability to add, containing the model's uncertainty with structure and validation, defending against untrusted input, and—above all—evaluating relentlessly, because you cannot ship what you cannot measure. Climb the capability ladder only as far as the task demands, treat the prompt as one step rather than the whole solution, and keep your model choice flexible as the field moves. Your next step is to pick one real task, solve it with the simplest rung that works—usually prompting, maybe RAG—and build a 20-example evaluation set before you write another line, because that set is what turns building with LLMs from guesswork into engineering.