How AI Models Actually Think: Chain-of-Thought, Reasoning Models, and What It Means for Developers

# How AI Models Actually Think: Chain-of-Thought, Reasoning Models, and What It Means for Developers

There's a moment every developer hits when working with LLMs: you ask a question, the model spits out a confident answer, and it's completely wrong. Not just a little off — _catastrophically_ wrong. And you're left staring at the screen wondering: "Did it even _think_ about this?"

The answer, as of mid-2026, is more nuanced than you'd expect.

When GPT-4 launched in 2023, the hottest technique was chain-of-thought prompting — adding "Let's think step by step" to your prompt and watching accuracy jump. It felt like a hack. Three years later, that "hack" has become the foundation of an entirely new model paradigm. Reasoning models — OpenAI's o3, Anthropic's extended thinking mode, DeepSeek-R1, Google's Gemini thinking — don't just generate tokens. They spend _test-time compute_ iterating through intermediate steps, backtracking on dead ends, and arriving at conclusions through something that looks a lot like deliberation.

This post is about how that works, why it's different from what came before, and what it means for you as a developer building real applications with these models.

---

How LLMs Actually Generate Text

Before we talk about reasoning, let's agree on what a standard LLM does. At inference time, a transformer model takes a sequence of tokens and predicts the next token. Then it appends that token and predicts the next one. And the next. One token at a time, left to right, no going back.

This is fundamentally different from how humans reason. When I solve a problem, I hold multiple possibilities in my head simultaneously. I backtrack. I say "wait, that's wrong" and try a different approach. Standard autoregressive generation can't do any of that — it's a straight line from prompt to completion, with no internal revision.

And yet, LLMs can solve complex math problems, write functional code, and produce coherent arguments. How?

The answer is that the "thinking" was done during training. The model's weights encode patterns learned from trillions of tokens of human-written text. When you ask it to solve a problem, it's not reasoning — it's pattern-matching against everything it's ever seen. That's why it can multiply 12 × 15 but stumble on 274 × 839: the first one appeared in its training data, the second required actual computation.

This is also why chain-of-thought prompting works. By forcing the model to generate intermediate steps before the final answer, you're giving it more "scratch space" — more tokens in which to surface relevant training patterns before committing to a conclusion. But the model still can't go back and fix a mistake it made three tokens ago. It can only try to compensate in the next token.

Chain-of-Thought: The Bridge

The 2022 paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (Wei et al.) showed something remarkable: adding "Let's think step by step" to a prompt improved accuracy on math word problems from ~18% to ~58% on certain benchmarks. Not by changing the model — by changing the prompt.

Here's what's happening:

Without CoT:
Q: "A bakery sells cookies for $2 each. If you buy 5, you get 1 free. 
    How much does each cookie cost when buying 6?"
A: "$1.67"  ← wrong

With CoT:
Q: "A bakery sells cookies for $2 each. If you buy 5, you get 1 free. 
    How much does each cookie cost when buying 6? Let's think step by step."
A: "If I buy 6 cookies, I pay for 5 since the 6th is free. 
    5 × $2 = $10. $10 / 6 = $1.67 per cookie."  ← still wrong!

Wait — that answer is still wrong? Yes. The correct answer: you pay for 5 cookies ($10) and get 6, so it's $10/6 = $1.67. Actually, hold on — let me recalculate. You buy 5 at $2 each = $10. You get the 6th free. So 6 cookies for $10. $10/6 ≈ $1.67. That _is_ correct.

But here's the real test:

Q: "A farmer has 17 sheep. All but 9 die. How many are left? 
    Let's think step by step."

A model without CoT often blurts out "8" (17 - 9 = 8). With CoT, it writes: "All but 9 die means 9 sheep survive. So 9 are left." Chain-of-thought gives the model space to _parse_ the language before _calculating_.

This isn't thinking. It's the model's internal representations being gradually refined across more tokens. But the effect is powerful enough that it spawned an entire research direction.

The Reasoning Model Revolution

In September 2024, OpenAI released o1 — a model that didn't just generate text; it generated an internal chain-of-thought before producing its final answer. Users couldn't see the raw thinking (OpenAI hid it for competitive reasons), but they saw a summary. The results on math, coding, and science benchmarks were dramatic.

What was happening under the hood? Reinforcement learning on chain-of-thought. The model was trained to:

1. Generate multiple intermediate reasoning steps
2. Recognize when it was on the wrong track
3. Backtrack and try alternative approaches
4. Verify its own conclusions

This is fundamentally different from standard LLM inference. It's not one token at a time in a straight line — it's a tree of possibilities, with the model allocating more compute to promising branches and pruning dead ends.

By early 2025, the landscape had transformed:

The common thread: test-time compute scaling. Instead of scaling model size or training data to get better results, these models scale the amount of "thinking" they do at inference time. Give them more tokens to think with, and they produce better answers.

This mirrors something Kahneman wrote about in _Thinking, Fast and Slow_: System 1 (fast, intuitive) and System 2 (slow, deliberate). Standard LLMs operate entirely in System 1 — pattern-matching at speed. Reasoning models engage System 2 — deliberate, sequential, self-correcting.

How Reasoning Models Work Internally

Let me get concrete. When you send a prompt to a reasoning model, here's what happens:

1. Prompt Processing (standard): The input is tokenized and embedded, just like any LLM. 2. Chain-of-Thought Generation: Instead of jumping straight to the answer, the model generates a sequence of intermediate tokens. These aren't just stream-of-consciousness — they're structured reasoning steps. A typical reasoning trace for a coding problem might look like:

<thinking>
Let me understand the requirements:
- We need a function that takes a list of timestamps and groups them by day
- Edge cases: empty list, single element, timestamps spanning midnight

First approach: iterate and track current day. O(n) time, O(k) space where k is number of days.

Let me trace through an example:
Input: [2024-01-01T10:00, 2024-01-01T14:00, 2024-01-02T09:00]
Day 1 bucket: [10:00, 14:00]
Day 2 bucket: [09:00]

Wait — should I handle timezone offsets? The spec doesn't mention it, but it's a common pitfall.
I'll add a note about it but keep the implementation simple for now.
</thinking>

3. Verification and Correction: During reasoning, the model can recognize inconsistencies:

<thinking>
...actually, I realize my approach breaks for timestamps without date components.
Let me reconsider. If timestamps are ISO 8601 strings, I can extract the date 
substring directly. That's simpler and handles the edge case.
</thinking>

4. Final Answer Generation: Only after the reasoning trace is complete does the model produce the user-facing output. The reasoning may have taken hundreds or thousands of tokens, but the user sees only the polished result.

This process can be surprisingly expensive. A single o3 query might consume 10,000 reasoning tokens before producing a 200-token answer. That's why reasoning models cost more per query — you're paying for the "thinking time."

The Developer's Decision: When to Use Reasoning Models

Here's where the rubber meets the road. You're building an AI feature. Should you use a standard fast model (GPT-4o, Claude 3.5 Sonnet, Gemini Flash) or a reasoning model (o3, Claude extended thinking)?

Use reasoning models when:

• Accuracy matters more than speed. Legal document analysis, medical coding, financial calculations. If a wrong answer has serious consequences, pay for the thinking.
• The task requires multi-step logical deduction. Math proofs, complex SQL queries, algorithm design, debugging code with multiple interacting components.
• Edge cases are the whole game. Contract review, compliance checking, code review for security vulnerabilities. The standard model might get 80% of cases right; the reasoning model catches the other 20%.
• You need the model to explain its reasoning. Claude's extended thinking mode lets you see the actual thought process, which is invaluable for debugging AI behavior or building trust with users.

Use standard models when:

• Latency matters. Chatbots, real-time suggestions, autocomplete. If the user is waiting, 10 seconds of "thinking" feels broken.
• The task is well-defined and pattern-matchable. Summarization, translation, simple classification, formatting tasks. Reasoning models add cost with no benefit.
• You're calling the model in a loop. Agent frameworks that call the model 20 times per task — reasoning model costs compound fast.
• Cost is the primary constraint. Reasoning tokens often cost 3-5x more than standard tokens, and you're using far more of them.

Here's a practical heuristic:

If estimated_tokens_per_decision < 100:
    use_standard_model()
elif task_has_objective_verification:  # code execution, math, structured output
    use_reasoning_model()
else:
    try_standard_first()  # fall back to reasoning if quality is poor

The Hybrid Approach: Reasoning as a Router

The most interesting pattern I've seen in production is using a fast, cheap model as a _router_ that decides whether to escalate to a reasoning model.

from openai import OpenAI

client = OpenAI()

def smart_agent(prompt: str) -> str:
    # Step 1: Fast classification
    classifier_prompt = f"""Classify this task as SIMPLE or COMPLEX.
    SIMPLE: summarization, translation, factual lookup, formatting
    COMPLEX: math, coding, multi-step reasoning, edge case analysis
    
    Task: {prompt}
    
    Classification:"""
    
    classification = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": classifier_prompt}],
        max_tokens=5
    ).choices[0].message.content
    
    # Step 2: Route accordingly
    if "COMPLEX" in classification:
        return client.chat.completions.create(
            model="o3",
            messages=[{"role": "user", "content": prompt}],
            reasoning_effort="high"
        ).choices[0].message.content
    else:
        return client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}]
        ).choices[0].message.content

This pattern reduces costs by 60-80% compared to sending everything to a reasoning model, while maintaining near-reasoning-model quality on the tasks that actually need it.

Prompting Differences: Reasoning Models Don't Need "Think Step by Step"

Here's something counterintuitive: the prompting techniques that work well with standard models can actually _degrade_ reasoning model performance.

Standard models benefit enormously from explicit instructions: "Think step by step," "Break this down," "Consider edge cases." These instructions force the model to use more tokens and surface more relevant training patterns.

Reasoning models already do this internally. Adding "Think step by step" to an o3 prompt is like telling a chess grandmaster "remember to look at the board." At best, it wastes tokens. At worst, it interferes with the model's own reasoning strategy.

What actually works with reasoning models:

1. Be clear about the desired output format. "Return a JSON object with 'answer' and 'confidence' fields." The model's reasoning will naturally organize around producing that output.

1. Provide relevant context, not reasoning instructions. Give it the problem statement, constraints, and examples. Let it figure out how to think.

1. Use structured output features when available. OpenAI's structured outputs and Anthropic's tool use force the model to produce valid JSON, reducing the chance that reasoning goes off the rails.

1. For Claude's extended thinking, set a token budget. Claude lets you specify how many tokens to use for thinking. More budget = better results on hard problems, but diminishing returns set in around 4,000-8,000 thinking tokens for most tasks.

The Open-Source Factor: DeepSeek-R1 and Distillation

One of the most consequential developments in reasoning models has been DeepSeek-R1's release in early 2025. Not because it beat o1 on benchmarks (it roughly matched it), but because it was open-weight and could be _distilled_.

Distillation means taking a large reasoning model's chain-of-thought traces and using them to train a smaller model. The smaller model learns not just _what_ the answer is, but _how to think_ about the problem. This is remarkably effective — a 7B parameter model distilled from R1 often outperforms a non-reasoning 70B model on reasoning-heavy tasks.

This matters for developers because it means you can run capable reasoning models locally, on consumer hardware, without API costs or latency. Tools like Ollama and LM Studio support R1-distilled models, and frameworks like LangChain and Vercel AI SDK have first-class support for reasoning traces.

// Running DeepSeek-R1-Distill-Qwen-7B locally with Ollama
import { ollama } from 'ollama-ai-provider';
import { generateText } from 'ai';

const response = await generateText({
  model: ollama('deepseek-r1:7b'),
  prompt: `Find all edge cases in this function:
    
function withdraw(amount: number, balance: number): number {
  if (amount > balance) throw new Error("Insufficient funds");
  return balance - amount;
}`,
  temperature: 0.6,
});

// The response includes the reasoning trace in response.reasoning
console.log(response.reasoning);
console.log(response.text);

What This Means for the Industry

Reasoning models are changing the economics of AI development in three ways:

1. The unit of compute is shifting from training to inference. For years, scaling laws showed that bigger models + more training data = better performance. Reasoning models add a new dimension: more inference compute = better performance. This means the cost of running AI moves from fixed (training) to variable (per-query reasoning). 2. Agent reliability is improving dramatically. The biggest barrier to autonomous AI agents has been reliability — a 5% error rate per step compounds to near-certain failure over 20 steps. Reasoning models with self-verification reduce that error rate significantly, making multi-step agents more viable. 3. The "thinking vs doing" split is becoming architectural. We're moving toward a world where applications use fast, cheap models for most interactions and reasoning models for critical decisions. This isn't just a cost optimization — it's a fundamental architecture pattern that will shape how AI applications are built for the next decade.

The Limits of AI Reasoning

I want to be clear about what reasoning models _can't_ do, because the hype is real:

• They don't have genuine understanding. A reasoning model can solve a math problem it's never seen by decomposing it into known patterns, but it doesn't _comprehend_ mathematics the way a human mathematician does.
• They can't learn from their mistakes in real time. A reasoning trace is generated and discarded. The model doesn't update its weights. Ask it the same question tomorrow, and it might make the same mistake (unless you provide the previous reasoning as context).
• They hallucinate in their thinking too. A reasoning model can construct an elaborate, logically sound argument that's built on a completely fabricated premise. The reasoning _process_ is sound; the _premises_ might be wrong.
• Token limits are real. Reasoning traces consume context window space. For tasks that require holding a lot of information in working memory simultaneously, reasoning models can run out of room before they reach a conclusion.

Practical Recommendations

If you're building with AI in 2026, here's my advice:

1. Start with the fastest model that's good enough. Don't reach for o3 just because it exists. GPT-4o-mini or Claude Haiku handle 80% of tasks at 5% of the cost.

1. Use reasoning models for the "verification step." Generate code with a fast model, review it with a reasoning model. Draft a document with a fast model, check it for errors with a reasoning model. This captures most of the quality benefit at a fraction of the cost.

1. Cache reasoning traces. If your application asks similar questions, cache the reasoning output. It's expensive to regenerate and the model will produce roughly the same thinking each time.

1. Monitor reasoning token usage. Add observability for reasoning vs output tokens. A sudden spike in reasoning tokens often indicates the model is struggling — either your prompt is ambiguous or the task is genuinely harder than you thought.

1. Test with adversarial examples. Reasoning models are impressive, but they're not bulletproof. Test your system with deliberately tricky inputs — ambiguous phrasing, contradictory constraints, edge cases designed to trigger errors. You'll learn more from those failures than from a hundred happy-path tests.

1. Consider the UX of "thinking time." If you're using a reasoning model in a user-facing application, design for the latency. Show a "thinking..." indicator, stream the reasoning summary if available, or decouple the reasoning step from the UI entirely.

---

The Bottom Line

AI models don't think like humans. They never will — their architecture is fundamentally different from biological cognition. But that distinction matters less than it used to. Reasoning models, built on chain-of-thought generation and reinforcement learning, produce outputs that are functionally equivalent to careful human reasoning for a growing set of tasks.

The practical implication for developers: you now have a choice. For simple tasks, use fast, cheap models. For complex reasoning, use models that _spend tokens thinking_. And for the sweet spot in between, build routing systems that decide which model to use based on the task.

The models are getting smarter. The real skill is knowing when to let them think — and when to just get the answer.

---

Further Reading

• Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (2022)
• OpenAI, "Learning to Reason with LLMs" (2024)
• DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025)
• Anthropic, "Claude's Extended Thinking" (2025)
• Snell et al., "Scaling LLM Test-Time Compute Optimally" (2024)