Start With One Chain

If chain-of-thought, or a reasoning model at higher effort, gets the task right, stop there. A tree costs ten to fifty model calls for one answer. It earns that on a narrow class of problems, where a wrong early choice can't be recovered inside a single answer.

Chain-of-thought gives the model room to work, but it still commits to one path. If the second step is a bad choice, every step after it is spent defending that choice. You see this on planning tasks: ask a model to order eleven database migrations so that no foreign key breaks, and it picks an order early, notices the conflict at migration seven, and then talks itself into a workaround instead of going back.

Tree-of-Thought (ToT) is the fix for that specific failure. Instead of one chain, you generate several candidate next steps, score each one, keep the promising ones, and expand them further. It is ordinary search, the same breadth-first or depth-first search you wrote in a data structures course, with the model doing two jobs: proposing moves and judging positions.

Reach for Tree-of-Thought when
  • The task has intermediate states you can name: a partial plan, a half-filled schedule, a draft patch
  • A wrong early choice is expensive to recover from inside a single answer
  • You can judge a partial state, either with the model or, better, with real code such as tests or a validator
  • You can afford ten to fifty model calls for one answer, because the quality of that answer matters more than its cost

If none of these hold, stop at chain-of-thought or a reasoning model with higher effort. ToT is a tool for a narrow class of problems, and it is easy to spend a lot of tokens on it for nothing.