Chain-of-Thought Prompting: Making LLMs Show Their Work

Make the Model Show Its Work

Chain-of-thought prompting increases accuracy on reasoning tasks by asking the model to produce intermediate steps before reaching a conclusion. The insight: LLMs are next-token predictors, and a token representing “the answer” is more likely to be correct when preceded by tokens that represent valid reasoning. CoT trades output length for answer accuracy — a trade-off worth making for any task where correctness matters.

Zero-Shot CoT

The simplest form: append “Let's think step by step” to any prompt. This instruction activates reasoning traces in models trained with CoT examples (which includes most instruction-tuned models today). No examples needed.

flowchart LR
    direct["Direct answer
(no reasoning)
Prompt → Answer
Fast, less accurate"]
    zeroCoT["Zero-shot CoT
Append 'step by step'
Prompt → Reasoning → Answer
Slower, more accurate"]
    fewCoT["Few-shot CoT
Examples with reasoning
Examples + Prompt → Reasoning → Answer
Best for complex tasks"]

    direct -->|add 'step by step'| zeroCoT
    zeroCoT -->|add examples| fewCoT

// Zero-shot CoT: one phrase changes accuracy significantly

// ❌ Direct — model jumps to conclusion, more error-prone
const directPrompt = `
An API processes 240 requests per minute. 
During a traffic spike, the rate triples for 5 minutes then drops to 180 req/min.
If each request uses 2KB of memory and you have 512MB available,
do you have enough memory during the spike?
`;

// ✅ Zero-shot CoT — model reasons through intermediate steps
const cotPrompt = `
An API processes 240 requests per minute.
During a traffic spike, the rate triples for 5 minutes then drops to 180 req/min.
If each request uses 2KB of memory and you have 512MB available,
do you have enough memory during the spike?

Think through this step by step.
`;

// Model output with CoT:
// Step 1: Normal rate = 240 req/min, spiked rate = 240 × 3 = 720 req/min
// Step 2: Memory during spike = 720 × 2KB = 1,440KB = 1.44MB per minute of requests
// Step 3: But requests are concurrent — how many at once? Need avg response time...
// [model identifies the ambiguity and addresses it]
// Final answer: [more accurate than direct]

Few-Shot CoT

// Few-shot CoT: examples with full reasoning traces
// Most effective for tasks with a consistent reasoning pattern

const fewShotCoTPrompt = `
You are a code review assistant. For each change, reason through its correctness before deciding.

Example 1:
Code change:
  const user = await db.users.findOne(userId);
  return user.email;

Reasoning:
- findOne could return null if userId doesn't exist
- Accessing .email on null will throw a TypeError at runtime
- This is missing a null check

Decision: REJECT — add null check: if (!user) throw new Error('User not found')

Example 2:
Code change:
  const users = await db.users.findMany({ limit: 100 });
  return users.map(u => u.email);

Reasoning:
- findMany returns an empty array if no results, never null
- limit: 100 prevents unbounded queries
- map on empty array returns empty array safely

Decision: APPROVE — looks safe

Now review this change:
Code change:
  const config = JSON.parse(process.env.APP_CONFIG);
  return config.apiKey;
`;
// Model reasons: JSON.parse throws on invalid JSON, env var could be undefined...
// Decision: REJECT — missing error handling

Self-Consistency

// Self-consistency: generate N reasoning paths, take majority answer
// Improves accuracy further — like an ensemble for reasoning tasks

async function selfConsistentAnswer(
  prompt: string,
  n = 5
): Promise<string> {
  // Generate N independent reasoning paths with temperature > 0
  const responses = await Promise.all(
    Array.from({ length: n }, () =>
      openai.chat.completions.create({
        model: 'gpt-4o',
        messages: [{ role: 'user', content: `${prompt}

Think step by step.` }],
        temperature: 0.7,  // Non-zero for diversity
      })
    )
  );

  const answers = responses.map(r => {
    const text = r.choices[0].message.content ?? '';
    // Extract final answer from CoT output
    // Common patterns: "Therefore: X", "Final answer: X", last sentence
    return extractFinalAnswer(text);
  });

  // Majority vote
  const counts = new Map<string, number>();
  for (const answer of answers) {
    counts.set(answer, (counts.get(answer) ?? 0) + 1);
  }

  return [...counts.entries()].sort((a, b) => b[1] - a[1])[0][0];
}

// When to use self-consistency:
// - High-stakes decisions where accuracy outweighs cost (N × API calls)
// - Tasks with discrete answers (classification, calculation, yes/no)
// - When you see high variance in single-pass answers

CoT for Code Tasks

// Ask the model to explain before implementing
// Catches design errors before they become code errors

const codingPrompt = `
Task: Write a function that deduplicates an array of objects by a given key,
keeping the last occurrence of each duplicate.

Before writing code:
1. State your approach and data structure choice
2. Identify edge cases you'll handle
3. Note any performance considerations

Then write the implementation.
`;

// Model output structure:
// Approach: Use a Map, iterate forward, overwrite on duplicate key (last-wins)
// Edge cases: empty array, key not present on some objects, undefined values
// Performance: O(n) time, O(n) space
//
// function dedupeByKey<T>(arr: T[], key: keyof T): T[] {
//   const seen = new Map<unknown, T>();
//   for (const item of arr) {
//     seen.set(item[key], item);  // overwrite = last wins
//   }
//   return [...seen.values()];
// }

// The pre-code reasoning step surfaces edge cases before they're embedded in code
// Significantly reduces bug rate compared to asking for direct implementation

Pitfalls

CoT on simple lookup tasks

Adding “think step by step” to “What is the capital of France?” adds tokens without adding accuracy. CoT helps reasoning tasks — multi-step math, logic, code analysis, decisions with trade-offs. For factual retrieval, direct prompts are faster and equally accurate.

Hallucinated reasoning steps

A model can produce confident, plausible-looking reasoning steps that are factually wrong — and then correctly derive the answer from the wrong premises. Always verify factual claims in the reasoning chain, especially for domain-specific knowledge. CoT makes the reasoning inspectable, but inspection is still your job.

Wrong final answer despite correct steps

The model can reason correctly up until the final step and then make a non sequitur conclusion. For high-stakes tasks, validate the final answer against the reasoning chain independently. Self-consistency helps here — if 4 of 5 paths reach the same answer through different reasoning, confidence is much higher.