Quick answer: Chain of thought prompting is a technique where you ask a language model to work through a problem step by step before giving its final answer, instead of jumping straight to a conclusion. Writing out the intermediate reasoning improves accuracy on math, logic, and multi-step tasks. The catch is that it costs extra tokens and does little for simple questions, and newer reasoning models now do it internally.
The core idea is almost too simple. If you ask a model a hard question and add "think step by step," it tends to get more answers right. Making the reasoning explicit gives the model room to work, the same way showing your working on a math test catches errors you would miss doing it in your head.
Why "Think Step by Step" Works
A language model generates one token at a time, and each token it writes becomes part of the context for the next one. When a model blurts out a final answer immediately, it has to compress all the reasoning into a single leap. On a multi-step problem, that leap fails often.
When you ask it to reason out loud, each step it writes becomes scaffolding for the next. The model computes an intermediate result, writes it down, and then reads its own writing back as it produces the following step. The reasoning is not just for you to read; it is working memory the model uses on itself.
Here is the difference in practice.
Prompt: A store has 23 apples. They sell 7 in the morning and receive a
delivery of 15. In the afternoon they sell 9. How many apples remain?
Without chain of thought:
"22 apples." (wrong, and no way to see where it went off the rails)
With chain of thought:
"Start with 23. Sell 7 in the morning: 23 - 7 = 16.
Delivery of 15: 16 + 15 = 31. Sell 9 in the afternoon: 31 - 9 = 22.
Answer: 22 apples."
In this small case the visible steps also let you check the work. On genuinely hard problems, the steps are often what makes the right answer reachable at all.
How to Actually Prompt for It
There are a few reliable ways to trigger chain of thought, from simplest to most controlled.
| Technique | What you do | Best for |
|---|---|---|
| Zero-shot | Add "Let us reason step by step" | Quick wins, no examples needed |
| Few-shot | Show 1-3 worked examples with reasoning | Steering the style of reasoning |
| Structured | Ask for numbered steps, then a final line | When you need to parse the answer out |
The structured version is the one I reach for in production, because it keeps the final answer separate from the reasoning so you can extract it cleanly.
Solve this problem. Show your reasoning as numbered steps.
Then, on the final line, write "ANSWER:" followed by only the result.
Problem: {your problem here}
Now you can pull whatever follows "ANSWER:" and ignore the rest. This pattern sits alongside the other techniques in our broader guide to prompt engineering.
When It Helps and When It Wastes Tokens
Chain of thought is not free and it is not always worth it. Every reasoning step is generated text, and generated text costs money and time. Spending 300 tokens of reasoning to answer "What is the capital of France" is pure waste.
Use chain of thought when the task has these traits:
- Multiple steps that build on each other, like word problems or multi-hop logic.
- A right answer you can be wrong about, where a single skipped step flips the result.
- Structured extraction from messy input, where the model needs to work through the parsing.
Skip it when the task is:
- A direct lookup or recall, where the answer is a single fact.
- A short classification with obvious labels.
- Latency-sensitive, where the extra generation time hurts the user experience.
A rough test: if a competent person would answer instantly without pausing, chain of thought is probably overhead. If they would grab a scratchpad, it probably helps.
How Reasoning Models Changed the Picture
For a few years, chain of thought was something you added by hand in the prompt. That has shifted. A newer class of models, often called reasoning models, is trained to do this kind of extended thinking internally before they answer. You ask a normal question and the model works through the steps on its own, sometimes for many seconds, then returns a final response.
This changes your job in two ways.
First, on these models you often do not need to say "think step by step" at all. The reasoning happens whether you ask for it or not, and adding your own manual chain of thought instructions can be redundant or even get in the way.
Second, you now choose a model based on the task instead of only patching a weaker model with prompting. A fast, cheaper model with an explicit chain of thought prompt is great for moderate tasks. A dedicated reasoning model is worth its higher cost and latency for genuinely hard problems. For the exact model names, context windows, and current pricing on the Claude family, check the claude-api skill rather than trusting a number you read somewhere, since these change often.
The old manual technique has not disappeared. It is still the right tool on fast, non-reasoning models, and it is still how you steer the shape of the reasoning when you need output in a particular form. But "always paste think step by step" is no longer a universal rule.
A Few Practical Cautions
Chain of thought improves accuracy on average, but it comes with sharp edges worth knowing.
- The reasoning is not a guaranteed audit trail. A model can produce plausible-looking steps and still reach a wrong answer, or reach the right answer for reasons that do not match the steps it wrote. Do not treat the visible reasoning as proof.
- Longer output means more surface for error. More generated tokens is more chances to drift off track, especially on very long chains.
- It raises cost and latency. Budget for it. If you are running chain of thought across thousands of requests, the token bill is real.
None of these are reasons to avoid the technique. They are reasons to apply it where it pays off and measure whether it actually improved your results rather than assuming it did.
The Short Version
Chain of thought prompting asks a model to reason step by step so each step supports the next, which lifts accuracy on multi-step and logic problems. It costs extra tokens, so save it for tasks that genuinely need working memory and skip it on simple lookups. Newer reasoning models fold this behavior in automatically, which means your lever is increasingly model choice as much as prompt wording. Use it deliberately, verify the answers, and do not confuse pretty reasoning with correct reasoning.
Building prompts that reason over live web data? link.sc fetches and searches the web so your model has real facts to reason about. Start free.