CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density
A benchmark where task length, intrinsic difficulty and distractor density move independently. Across 22 reasoning models, task length is the dominant constraint and distractor ratio produces a U-shaped response.
Abstract
Current benchmarks for long-context reasoning in Large Language Models (LLMs) often blur critical factors like intrinsic task complexity, distractor interference, and task length. To enable more precise failure analysis, we introduce CogniLoad, a novel synthetic benchmark grounded in Cognitive Load Theory (CLT). CogniLoad generates natural-language logic puzzles with independently tunable parameters that reflect CLT’s core dimensions: intrinsic difficulty (d) controls intrinsic load; distractor-to-signal ratio (ρ) regulates extraneous load; and task length (N) serves as an operational proxy for conditions demanding germane load. Evaluating 22 SotA reasoning LLMs, CogniLoad reveals distinct performance sensitivities, identifying task length as a dominant constraint and uncovering varied tolerances to intrinsic complexity and U-shaped responses to distractor ratios. By offering systematic, factorial control over these cognitive load dimensions, CogniLoad provides a reproducible, scalable, and diagnostically rich tool for dissecting LLM reasoning limitations and guiding future model development.
Key numbers
- 22
- state-of-the-art reasoning models evaluated
- 140
- configurations of d ∈ {1,3,5,7,10}, N ∈ {20,50,100,250}, ρ ∈ {5…95%}
- 383
- ECL₅₀ of GPT-5: statements it can process at 50% accuracy under mean load, the highest measured
- 17 / 22
- models with a significant negative d × N interaction: long and hard compound
Interactive
Generate a puzzle
Every CogniLoad puzzle follows a few people through a chain of state updates. Some statements change the answer, the rest are hay. The three dials set how many attributes a step can touch, how long the chain is and how much of it is hay. This is the reference generator itself, ported from cogniload.py with Python’s random number generator, so a puzzle id gives the same file here as the script writes. Download the puzzle, its state file or a whole run.
This is the reference generator, ported step for step from cogniload.py together with Python’s random number generator. The same id gives the same puzzle and the same state file as the script, byte for byte. The paper’s grid is N in 20, 50, 100, 250, ρ from 5% to 95% and d in 1, 3, 5, 7, 10. The ticks on the sliders mark it.
Reproduce a run
One file with a line per puzzle, the puzzles the script writes for this configuration, in its order. Keep the full state out unless you need it, it is large at N = 250.
Interactive
Score a model
The scoring pipeline from Appendix D of the paper, run in your browser. Drop model outputs in and get the accuracy and the error buckets. The scorer looks at three windows, the last sentence that names the person and the asked property, the last sentence that names the person, and the last sentence of the output. Each is checked for the right value and for a wrong value from the same category. Nothing is uploaded anywhere.
What it reads
- A .jsonl, .json or .csv with one record per answer. Each record needs the model output (response) and the puzzle name (name or file, like 20_3_10_102133_yellow).
- Or the model outputs as .txt files named like the puzzles, with the puzzle files and the _state.json files beside them if you have them.
- For puzzles this generator reproduces, the name is enough. For others, add poi, the question and category_defaults, or the state file. prompt_tokens and response_tokens switch on the context-limit check.
Check one answer
Interactive
Read a model’s capacity
The paper fits one logistic response model per LLM, with accuracy as a function of difficulty d, log task length N and needle ratio ρ, plus a ρ² term for the U-shape. Pick a model, choose which dimension to vary and read off ECL₅₀, ID₅₀ and NT₅₀, the loads at which it drops to 50%.
Fitted per-model logistic response from Table 3 of the paper (coefficients to two decimals). Grey curves are the other 21 models. The capacity points hold the other dimensions at the evaluation grid’s means, as in the paper.
Cite
@inproceedings{kaiser2026cogniload,
title = {CogniLoad: A Synthetic Natural Language Reasoning Benchmark With Tunable Length, Intrinsic Difficulty, and Distractor Density},
author = {Kaiser, Daniel and Frigessi, Arnoldo and Ramezani-Kebrya, Ali and Ricaud, Benjamin},
booktitle = {International Conference on Learning Representations (ICLR)},
year = {2026},
url = {https://openreview.net/forum?id=0Sex2H5Jnn}
}