Chain of thought prompting is the technique of asking a language model to reason through a problem step by step before giving its final answer, which measurably improves accuracy on tasks involving math, logic, and multi-step reasoning. Instead of forcing the model to commit to an answer immediately, you give it room to work through the intermediate steps—much as a person solves a hard problem on paper rather than in their head. This chain of thought prompting guide covers what the technique is, why it works, the main variants, exactly when to use it (and when not to), and the caveats that matter in production.
What chain of thought prompting is
By default, a language model generates its answer token by token, and for a hard question it can lock onto a wrong answer before it has "worked anything out." Chain of thought (CoT) prompting changes that by instructing the model to produce its reasoning first—the intermediate steps—and only then state the final answer.
The classic contrast makes it concrete. Ask a model a multi-step word problem directly and it may blurt out a wrong number:
Without CoT: "A shop has 23 apples. They use 20 for pies and buy 6 more. How many apples now?" → "3."
Prompt it to reason step by step, and it works through the arithmetic correctly:
With CoT: "…Let's think step by step." → "Start with 23. Use 20 for pies, leaving 3. Buy 6 more, giving 9. The answer is 9."
The technique emerged from research (notably Wei and colleagues in 2022) showing that prompting large models to generate reasoning chains dramatically improved their performance on arithmetic, commonsense, and symbolic reasoning tasks—often turning near-random results into reliable ones. It's now one of the most important techniques in the broader toolkit of prompt engineering.
Why it works
Chain of thought works because it gives the model more computation to spend on the problem. A model does a fixed, limited amount of work per token it generates, so demanding an immediate answer caps how much "thinking" can happen. By generating reasoning steps first, the model spreads the problem across many tokens, and each step builds on the last—so it can break a hard problem into manageable pieces rather than solving it in one leap.
There's a useful analogy: asking a model to answer a complex question instantly is like asking a person to compute 47 × 89 in their head and shout the answer. Give them room to work it out step by step—on paper, or out loud—and they're far more accurate. The reasoning tokens are the model's scratch paper.
This also explains why CoT helps on some tasks and not others. It shines when a problem genuinely requires multiple dependent steps, because that's exactly what the extra computation buys you. For a simple factual lookup or a direct classification, there's no multi-step reasoning to unfold, so CoT adds tokens and latency without improving the answer.
The main variants
Chain of thought comes in several flavors, worth knowing so you reach for the right one.
Zero-shot CoT
The simplest and most famous version: just add a phrase like "Let's think step by step" to your prompt. Remarkably, this single instruction triggers reasoning behavior with no examples needed, and it reliably improves results on reasoning tasks. It's the first thing to try because it costs almost nothing to add.
Few-shot CoT
Here you include a few worked examples in the prompt that show the reasoning, not just the answer—demonstrating the step-by-step process you want the model to imitate. This is more powerful than zero-shot for complex or domain-specific reasoning, because the examples teach the model both how to reason and the format to follow. The tradeoff is that the examples consume context and add cost, and they must be correct—a flawed example teaches flawed reasoning.
Self-consistency
A refinement that trades cost for accuracy: instead of generating one reasoning chain, you sample several (using a higher temperature for variety), then take the majority answer across them. Because different reasoning paths often converge on the correct answer even when individual chains err, voting across multiple attempts measurably boosts reliability on hard problems. It's more expensive—several model calls instead of one—so reserve it for high-stakes tasks where accuracy justifies the cost.
| Variant | How it works | Best for |
|---|---|---|
| Zero-shot CoT | Add "let's think step by step" | A cheap first improvement on any reasoning task |
| Few-shot CoT | Show worked reasoning examples | Complex or domain-specific reasoning |
| Self-consistency | Sample several chains, take the majority answer | High-stakes tasks where accuracy justifies extra cost |
There are more elaborate extensions—breaking a problem into ordered sub-problems, or exploring multiple reasoning branches like a tree—but the three above cover the vast majority of practical use.
When to use chain of thought (and when not to)
CoT is powerful but not free, so apply it deliberately.
Use it when the task requires genuine multi-step reasoning: arithmetic and math word problems, logic puzzles, multi-hop questions that chain several facts, planning, code debugging, and any task where a person would need to work things out rather than answer instantly. It's also valuable inside agents—when an agent decides which action to take next, reasoning about the situation first leads to better decisions, which is why it improves both tool use (choosing the right tool for the moment) and the reliability of multi-agent systems coordinating complex work. Most agent frameworks build step-by-step reasoning into their planning loops for exactly this reason.
Skip it when the task is simple: direct factual lookups, straightforward classification, or format conversions gain nothing from step-by-step reasoning and just pay the cost. And be aware of the cost: CoT generates far more tokens, which means higher latency and expense—a real consideration when a prompt runs at scale or inside an agent making many calls.
A note on modern models: newer "reasoning models" are trained to perform chain-of-thought reasoning internally before answering, so they often do this automatically without being asked. When using such a model, explicit "think step by step" instructions may be redundant. Knowing whether your model reasons by default tells you how much CoT prompting it still needs.
Common mistakes and caveats
- Using CoT on tasks that don't need it. Adding step-by-step reasoning to simple lookups or classifications wastes tokens and time for no benefit. Match the technique to the task.
- Trusting the reasoning as a true explanation. This is the deepest caveat: the model's stated reasoning is not guaranteed to be faithful—it can reach a correct answer while the written steps don't reflect how it actually got there, or produce plausible-sounding but flawed logic. Treat the reasoning as a performance aid, not a verified audit trail of the model's internal process.
- Providing incorrect few-shot examples. The model imitates your examples, so a mistake in a worked example propagates into its reasoning. Verify your examples.
- Ignoring the cost at scale. In production and in agents, the extra tokens from CoT multiply into real latency and expense. Budget for it and confirm the accuracy gain justifies it.
- Not measuring whether it actually helps. CoT usually improves reasoning tasks but not always. Test it against a no-CoT baseline on your task rather than assuming, applying the same evaluation discipline you'd use for any prompt change.
- Forgetting to extract the final answer. When the model reasons then answers, your code needs to reliably parse out the answer from the reasoning—ask for a clear final-answer marker, or pair CoT with function calling to return a structured result.
Frequently asked questions
What is chain of thought prompting? It's a technique where you instruct a language model to reason through a problem step by step before giving its final answer, rather than answering immediately. Producing the intermediate reasoning first measurably improves accuracy on tasks involving math, logic, and multi-step reasoning, because it lets the model break a hard problem into manageable pieces instead of solving it in one leap.
Why does chain of thought improve LLM accuracy? Because it gives the model more computation to spend on the problem. A model does limited work per token, so demanding an instant answer caps how much reasoning can happen. Generating step-by-step reasoning spreads the problem across many tokens, with each step building on the last—like working a hard calculation out on paper instead of in your head. This mainly helps genuinely multi-step problems.
What's the difference between zero-shot and few-shot chain of thought? Zero-shot CoT simply adds an instruction like "let's think step by step" with no examples, triggering reasoning behavior on its own. Few-shot CoT includes worked examples that demonstrate the step-by-step reasoning you want the model to imitate. Few-shot is stronger for complex or domain-specific reasoning because it teaches both the method and the format, but it costs more context and the examples must be correct.
When should I not use chain of thought prompting? Skip it for simple tasks—direct factual lookups, straightforward classification, or format conversions—where there's no multi-step reasoning to unfold, so it only adds tokens, latency, and cost. Also reconsider explicit CoT with modern reasoning models that already reason internally by default. Always weigh the accuracy benefit against the extra expense, especially when a prompt runs at scale or inside an agent.
Is the model's chain of thought a true explanation of its reasoning? Not necessarily. The stated reasoning is not guaranteed to be faithful to how the model actually reached its answer—it can produce a correct answer with reasoning that doesn't reflect its real process, or plausible-sounding steps that are flawed. Chain of thought is a technique to improve answers, not a verified audit trail, so don't treat the written steps as a reliable window into the model's internals.
The takeaway
Chain of thought prompting improves a model's answers on hard problems by giving it room to reason step by step before committing—whether through a simple "let's think step by step," worked few-shot examples, or self-consistency voting across multiple chains for high-stakes accuracy. The craft is applying it deliberately: reach for it on genuine multi-step reasoning, skip it on simple tasks where it only adds cost, remember the reasoning isn't a guaranteed audit trail, and measure whether it actually helps your task. Your next step is to take a reasoning task your model gets wrong, add "let's think step by step," and compare the results—because seeing the accuracy jump on a problem it previously failed is the fastest way to understand when this technique earns its cost.