Chain of Thought Reasoning

1. What Intermediate Steps Add

Consider an inventory question: a store has four packs of six pens and sells five pens. How many remain? The final answer is 19, but reaching it requires accounting for both the pack size and the sale. A short worked solution first calculates 4 × 6 = 24, then subtracts 5 to obtain 19.

A direct-answer response might simply say "19 pens." A chain-of-thought response includes the intermediate quantities before giving that answer. Both formats can be correct. The useful difference is that the written quantity 24 becomes available in the response before the subtraction and final answer are generated.

The original chain-of-thought prompting study demonstrated this approach using worked examples in the prompt, with improvements on several arithmetic, commonsense, and symbolic reasoning tasks. Those results established a useful technique, not a guarantee that every model or every question benefits from a longer response.

2. Asking for a Worked Solution

A prompt is the input supplied to the model, including the task, instructions, and any examples. In zero-shot prompting, we provide no solved examples of the task. An instruction such as "Show the relevant calculation, then give the final answer" asks for intermediate work directly. The zero-shot CoT study investigated how a short step-by-step cue could improve reasoning on its evaluated models and tasks.

In few-shot prompting, we also provide a small number of solved examples. Each example shows the question, a useful sequence of intermediate steps, and the answer. This gives the model a concrete pattern to continue. For our inventory problem, examples should preserve the distinction between packs and individual pens rather than merely demonstrate a writing style.

Solve each inventory problem with a short calculation.
Put the final integer answer on its own line as FINAL: <answer>.

Question: There are 3 packs of 5 pens. A customer buys 4 pens.
How many remain?
Solution: 3 × 5 = 15 pens. After the sale, 15 − 4 = 11 pens.
FINAL: 11

Question: There are 2 packs of 8 pens. A customer buys 3 pens.
How many remain?
Solution: 2 × 8 = 16 pens. After the sale, 16 − 3 = 13 pens.
FINAL: 13

Question: There are 4 packs of 6 pens. A customer buys 5 pens.
How many remain?
Solution:

The examples change the context for this request. They do not update the model's weights. Their arithmetic, assumptions, and answer format still need to be correct, because an incorrect example can supply a misleading pattern. The wording is also model-dependent: a system already trained to solve reasoning tasks may need only a clear problem statement, so an explicit step-by-step instruction should be tested rather than assumed to help.

3. What Happens Inside the Transformer

A language model reads and generates tokens, which are pieces of text. During standard autoregressive generation, it predicts one next token using the prompt and the tokens already produced. The chosen token is appended to that prefix, and the process repeats. A word, number, or equation may occupy several tokens.

Intermediate calculations follow this same generation process. After the response contains "4 × 6 = 24 pens," later predictions can use that text along with the original question. Causal attention gives a new position access to earlier positions; it does not give the model access to a final answer that has not yet been generated.

CoT prompting does not add a special reasoning layer to the Transformer. The existing network continues predicting tokens, with a prefix that now includes intermediate work. A direct-answer model also performs computation through its layers, so the absence of a written solution does not imply an absence of internal computation.

4. More Tokens Mean More Computation

Writing intermediate results gives the model a text-based workspace. It can establish a quantity, refer to that quantity later, and split a longer problem into smaller calculations. Earlier scratchpad research studied training language models to emit intermediate computations for this purpose. The important resource is the sequence of useful intermediate states, not the length of the prose by itself.

In ordinary sequential decoding, every additional generated token requires another decoding step. Longer solutions also extend the context that later positions can attend to. A KV cache reuses earlier attention keys and values, reducing repeated computation on the prefix, but it does not make those extra output tokens free. Their cost depends on sequence length, the model, and the serving system.

Extra steps can also introduce errors. If the model writes the wrong starting quantity, subsequent calculations may consistently use that incorrect value. A longer token budget creates room for useful work without ensuring that the work is useful. Simple factual questions may gain little from a lengthy derivation, while missing facts may require evidence from retrieval or tools before a sound solution is possible.

5. Prompting and Training Change Different Things

Asking for intermediate work at inference time is different from training a model to produce useful intermediate work. The first changes the input to a fixed model; the second changes the learned parameters. These approaches can be combined, but they operate at different stages.

With supervised fine-tuning, a training example can include a question and a worked solution as the target sequence. Next-token training encourages the model to reproduce that pattern, using the same prediction principle introduced in GPT pretraining. With methods such as GRPO, generated attempts receive rewards that influence policy updates. A reward for the final answer alone does not independently certify every intermediate step.

6. Compare Several Attempts with Self-Consistency

A single generated solution may make an avoidable mistake. Self-consistency samples several solution attempts for the same question, extracts each final answer, and selects the most frequently occurring answer. The intermediate wording can differ across attempts; the comparison is between their conclusions. The self-consistency paper showed gains over a single greedy CoT generation on its evaluated reasoning benchmarks.

For example, answers of 19, 21, 19, 19, and 18 give three votes to 19. This is often described as majority voting, although the usual rule selects the most frequent answer even when it has less than half the votes. A tie needs an explicit policy. The demonstration below leaves the answer unselected when the leading counts are equal.

Repeating deterministic decoding under identical conditions can simply repeat the same output. Self-consistency relies on sampling varied attempts, which increases generation work. Even independently sampled attempts can share a systematic mistake because they use the same model and prompt. Agreement is therefore a useful selection signal, not independent proof that the selected answer is correct.

7. Check the Answer and the Steps

A verifier checks a candidate against a rule or source of evidence. In the inventory example, ordinary arithmetic establishes that 4 × 6 − 5 = 19. A generated answer of 21 fails that check even if several attempts agree on it. For code, executing relevant tests can provide evidence; for a factual claim, checking a reliable source addresses a different requirement from producing a plausible explanation.

We can distinguish checking the outcome from checking the process. Outcome checking evaluates the final result. Process checking evaluates intermediate steps, such as whether the initial quantity was calculated correctly and whether the sale was subtracted. The process-supervision study investigated training reward models with step-level feedback for mathematical solutions.

Verification has limits of its own. A small test suite may miss bugs, and a learned verifier can make mistakes. Asking the same model to review its response may catch some errors, but it does not automatically supply independent evidence. The strength of the check depends on what it actually tests and whether those tests cover the original task.

8. A Readable Explanation Has Limits

A worked response makes claims that a reader can inspect. That can help locate arithmetic errors, unsupported assumptions, or missed conditions. However, the written explanation is still generated text. It does not expose every activation, attention operation, or influence that contributed to the answer.

Correctness asks whether the answer or derivation is valid. Faithfulness asks whether the explanation accurately reflects the factors that drove the model's behavior. Research on unfaithful CoT explanations found cases where biasing information changed answers without being acknowledged in the accompanying explanation. A persuasive rationale can therefore omit an important influence.

Presentation adds another distinction: an application might display generated intermediate text, a shorter explanation, or only a final answer. The text a user sees is not sufficient to reconstruct the computation performed by the system. A concise, checkable explanation can still be useful, provided its claims are evaluated on their merits rather than treated as a complete record of internal reasoning.

9. A Small Answer-Voting Example in Python

This example implements only the final-answer aggregation stage. It accepts already generated responses to the same integer-answer question and expects the last nonempty line to have the form FINAL: 19. The parser uses that dedicated line so an intermediate number such as 24 is not accidentally counted as the answer.

import re
from collections import Counter


def vote_integer_answers(completions: list[str]):
    counts = Counter()
    invalid = 0
    for text in completions:
        lines = text.strip().splitlines()
        final_line = lines[-1].strip() if lines else ""
        match = re.fullmatch(r"FINAL: ([+-]?[0-9]+)", final_line)
        if match is None:
            invalid += 1
            continue
        try:
            answer = int(match.group(1))
        except ValueError:
            invalid += 1
            continue
        counts[answer] += 1

    top_votes = max(counts.values(), default=0)
    leaders = [answer for answer, votes in counts.items() if votes == top_votes]
    return {
        "answer": leaders[0] if len(leaders) == 1 else None,
        "top_votes": top_votes,
        "valid": sum(counts.values()),
        "invalid": invalid,
    }


samples = ["FINAL: 19", "FINAL: 21", "FINAL: 19", "FINAL: 19", "FINAL: 18"]
result = vote_integer_answers(samples)
# {"answer": 19, "top_votes": 3, "valid": 5, "invalid": 0}

Converting the extracted value to an integer makes 19, +19, and 019 count together. Responses without a valid final line are counted separately and do not vote. A tie or a batch with no valid answers returns None. The top vote count and the number of valid responses describe the sample set; they are not a calibrated confidence score.

The parser deliberately handles only integer answers. Fractions, units, symbolic expressions, and open-ended text need task-appropriate equivalence rules. Generation, answer verification, and decisions about malformed responses remain outside this function. The five sample strings are illustrative data, so running the code tests aggregation without calling a language model.

10. Follow One Request from Prompt to Answer

For the inventory task, the application first prepares the question and an answer-format instruction. It can include worked examples when they are useful for the chosen model. The model then generates a solution and final answer, with each new token conditioned on the available prefix. Throughout this inference request, the model's weights stay fixed.

If self-consistency is enabled, the application collects several sampled attempts and extracts their final answers. It applies the chosen tie and malformed-output rules, then checks the selected result against the task. Here, the independent arithmetic check confirms 19, and the user can receive that answer with a short supporting calculation.

Each part has a specific role: intermediate text carries forward useful information, sampling explores additional attempts, voting aggregates their conclusions, and verification tests the result. Their value comes from helping the system solve the task correctly. The length or fluency of a solution is not a substitute for that final check.