Branching Reasoning Paths
Propose several next steps, judge the partial states, keep the best and expand them. How to build a Tree-of-Thought search, why the evaluator decides everything, and when a reasoning model makes it unnecessary.
Tree-of-Thought (ToT) extends chain-of-thought by exploring multiple reasoning branches simultaneously. Instead of one linear reasoning chain, ToT generates several candidate "thoughts" at each step, evaluates them, and expands the most promising branches — like a search tree. This enables backtracking from dead ends, which plain CoT cannot do. ToT is most effective for planning problems, creative writing with constraints, and mathematical reasoning where multiple approaches should be explored.
The pipeline
Each stage in order — click any step to read what it does.
Interactive Pipeline Flow
Architecture diagrams
How Tree-of-Thought differs from chain-of-thought and zero-shot.
Key design decisions
The trade-offs worth knowing before you build this.
The original ToT paper showed 74% success rate on the 24 Game vs 4% for CoT. For problems requiring multiple decision points and backtracking (puzzles, code debugging, route planning), ToT can improve from 5% to 50%+ pass rate.
ToT requires 10-50 LLM calls per problem vs 1-3 for CoT. Cost: $0.50-5.00 per problem vs $0.02-0.10 for CoT. Reserve ToT for high-value problems where a 10x quality improvement justifies the cost (e.g., complex code generation, strategic planning).
Full parallel BFS is complex to implement. A simpler approximation: 'Generate 3 approaches, evaluate them, pick the best, continue.' This sequential ToT captures most of the benefit with much less orchestration complexity.
“Build next week’s on-call rota: nobody twice in a row, Priya off Wednesday, at least one senior every night.”
The same request as in Read, with the values it has at each stage. Step through it, or click any stage. Values are illustrative for this scenario.
Propose
For Monday and Tuesday, the model proposes 3 different partial rotas
Evaluate
Code checks each partial rota against the hard constraints; one already puts Priya on twice in a row and is dropped
Expand
The two surviving branches are extended to Wednesday, again with 3 options each
Backtrack
One branch cannot cover Wednesday with a senior while Priya is off, so it is abandoned and the search continues from the other
Complete
After 4 levels a full valid rota is found, using 19 model calls
The same request, failing at “Evaluate”
An early version asked the model to rate each partial rota itself. It rated a rota with Priya on Wednesday as valid, and the search built on it for three levels. When the constraints can be checked in code, check them in code; model self-evaluation is the weak point of tree-of-thought.
Continue reading
All Prompt Engineering articles →Meta-Prompting: Letting a Model Tune Your Prompts
From a one-off rewrite to refinement loops on failures, candidate search and compiled DSPy programs. How to optimise prompts against a metric without overfitting, and how to review what the optimiser produces.
Red Teaming LLM Applications: Attack It Before Your Users Do
Threat model first, then manual sessions, then automated scans with garak, promptfoo and PyRIT. How to find jailbreaks, data leaks and unsafe tool calls, fix them in the architecture, and keep them fixed in CI.
Constrained Generation: Beyond Hope-Based Parsing
Enforce output structure at the token level with grammars and token masks — how it differs from native JSON mode, and when the strictness costs you reasoning quality.
Discussion
Sign in to share your feedback and join the discussion.