When Is More Reasoning Worth It?
One effort setting per benchmark saves 17.8% of tokens within half an accuracy point of always thinking hard. Choosing per question with oracle-like knowledge would save 43.8%. Current forecasts close about 5% of that gap.
Abstract
The next generation of reasoning models will not be the ones that think longest, but the ones that know when to think. Adapting reasoning effort to each question is a central open problem, and solving it would move the frontier between cost and accuracy. This work is a step in this direction. We build a controlled evaluation of effort selection and release its data: repeated responses from open models at every effort setting on the same questions. We first measure the opportunity: within half an accuracy point of always using high effort, one setting per benchmark already saves 17.8% of generated tokens, whereas choosing per question with oracle-like knowledge could save 43.8%. We then compare two practical strategies, predicting from the question text and checking agreement among cheap-model answers, with every model call charged. We propose question-level cost forecasts that improve an existing selector without extra generation. Finally, we identify where further gains lie: knowing where extra reasoning helps matters beyond knowing where it is expensive, but this benefit is model specific and current forecasts capture little of it.
Key numbers
- 597,744
- scored responses, released as an open dataset
- 9 × 10
- open models × benchmarks, 8 or 16 responses per question and effort setting
- 17.8%
- tokens saved by one setting per benchmark, within 0.5 accuracy points of always-high
- 43.8%
- tokens saved by per-question choice with oracle-like knowledge
- +0.69
- accuracy points gained by adding question-level cost forecasts to RADAR at equal tokens
- 20.3%
- of the dollar bill saved by four cheap-model probes guiding Qwen3.5-27B, for a 0.42-point loss
Interactive
Where to spend thinking
A toy, not the paper’s data. Two made-up benchmarks, a token budget and three ways to spend it. The same effort everywhere, one setting per benchmark or a per-question oracle. The shape of the gap is the point. In the paper it is 17.8% against 43.8% of tokens saved, and the question is how much of it a forecast from the question text can recover.
A toy with 60 invented questions in two “benchmarks”: each has a chance of being right at low and at high effort, and a token cost for each. The curves sweep the budget; the dots are the current one. The oracle line is the ceiling the paper measures at 43.8% of tokens; the per-benchmark line is its 17.8% floor. Forecasts from question text would sit between them.
Oracle allocation at this budget · one cell per question · filled = high effort · left half benchmark 1, right half benchmark 2
Cite
@article{kaiser2026when,
title = {When Is More Reasoning Worth It?},
author = {Kaiser, Daniel and Ricaud, Benjamin},
year = {2026},
note = {Under review}
}