Large language models have recently evolved to include specialized variants explicitly designed for reasoning tasks, known as large reasoning models (LRMs). These models, such as OpenAI's GPT-3, Deepseek R1, and Claude 3.7, have demonstrated promising results across various reasoning benchmarks.

However, the paper questions whether these models are truly capable of generalizable reasoning or if they are simply leveraging different forms of pattern matching. The authors argue that current evaluations predominantly focus on established mathematical and coding benchmarks, emphasizing final answer accuracy. This evaluation paradigm often suffers from data contamination, where the models were trained on the benchmarks themselves, and does not provide insights into the reasoning traces, structure, and quality of the models' thought processes.

The paper proposes the use of controllable puzzle environments as an alternative evaluation metric. These puzzles, such as Tower of Hanoi, Checkers Jumping, River Crossing, and Blocks World, can be systematically varied in complexity while preserving the core logic. This approach allows for fine-grain control over complexity, avoids contamination common in established benchmarks, and requires only the explicitly provided rules, emphasizing algorithmic reasoning.

The authors' experiments reveal that under equivalent inference token budgets, non-thinking language models can eventually reach performance comparable to thinking models on benchmarks like Math 500 and ARC. However, the performance gap widens on more complex benchmarks like ARC-25, suggesting that the thinking models may have genuine advantages for more complex problems or that the newer benchmarks are less contaminated.

Interestingly, the paper finds that as the complexity of the puzzles increases, the reasoning models initially spend more tokens while accuracy declines gradually until a critical point where reasoning collapses, and performance drops sharply. The models also exhibit a phenomenon called "overthinking," where they find the correct solution early in their thinking but continue exploring incorrect solutions, leading to wasted compute.

These findings highlight the limitations of current reasoning approaches and suggest that more sophisticated improvements may be necessary to advance toward more robust reasoning capabilities.

Large language models have recently evolved to include specialized variants explicitly designed for reasoning tasks, known as large reasoning models (LRMs). These models, such as OpenAI's GPT-3, Deepseek R1, and Claude 3.7, have demonstrated promising results across various reasoning benchmarks.

However, the paper questions whether these models are truly capable of generalizable reasoning or if they are simply leveraging different forms of pattern matching. To address this, the paper proposes a new evaluation approach using controllable puzzle environments.

Rather than relying on established mathematical and coding benchmarks, which often suffer from data contamination issues, the paper suggests using puzzles of varying complexity. These puzzles allow for systematic adjustments to the problem elements while preserving the core logic. The puzzles used in the evaluation include Tower of Hanoi, Checkers Jumping, River Crossing, and Blocks World.

The key advantages of this puzzle-based evaluation are:

1. Controlled Complexity: The complexity of the puzzles can be systematically adjusted, enabling a more nuanced understanding of the models' reasoning capabilities.

2. Avoidance of Contamination: The puzzles are less susceptible to data contamination, as they cannot be easily memorized like standard benchmarks.

3. Emphasis on Algorithmic Reasoning: The puzzles require explicit application of logical reasoning, rather than relying on pattern matching alone.

4. Rigorous Evaluation: The puzzle environments support simulator-based evaluation, allowing for precise solution checks and detailed failure analyses.

By adopting this puzzle-based evaluation approach, the paper aims to provide a more insightful and controlled assessment of the reasoning capabilities of large language models, shedding light on their strengths, limitations, and the potential scaling challenges they face as problem complexity increases.

The paper examines the capabilities of large reasoning models (LRMs) that are designed for thinking tasks, such as OpenAI's GPT-3, Deepseek R1, and Claude 3.7. It questions whether these models truly possess generalized reasoning abilities or are simply leveraging pattern matching.

The key findings from the paper's experiments using puzzle-based benchmarks are:

1. Performance Gap Widens with Complexity: At low complexity, thinking and non-thinking models perform similarly. However, as complexity increases, the thinking models outperform the non-thinking models. But at high complexity, both models fail.

2. Thinking Models Waste Compute: Thinking models initially spend more tokens on solving the puzzles, but as complexity increases, they start using fewer tokens, even as accuracy declines. This suggests the models are "giving up" rather than continuing to reason.

3. Overthinking and Inefficiency: For simpler problems, thinking models often find the correct solution early but continue exploring incorrect solutions, wasting compute. As complexity increases, the models first explore incorrect solutions and only later arrive at the correct ones.

4. Algorithmic Execution Fails: Even when provided the exact algorithm to solve the puzzles, the thinking models still exhibit the same performance collapse at high complexity, suggesting limitations in verification and logical reasoning.

The paper concludes that current LRMs, despite their sophisticated self-reflection mechanisms, fail to develop truly generalized reasoning capabilities beyond certain complexity thresholds. It highlights the need for further advancements in reasoning approaches to achieve more robust problem-solving abilities.

Large reasoning models (LRMs) such as OpenAI's GPT-3, Anthropic's Claude, and DeepSEE have demonstrated promising results on various reasoning benchmarks. However, this paper questions whether these models truly possess generalizable reasoning capabilities or are simply leveraging pattern matching.

The key findings regarding the scaling limits of reasoning effort are:

1. Performance Gap with Increasing Complexity: As the complexity of the puzzles increases, the performance gap between thinking (LRMs) and non-thinking (standard LLMs) models widens. This suggests that the thinking models may have genuine advantages for more complex problems, or it could be due to reduced data contamination in newer benchmarks.

2. Reasoning Effort Collapse: As the puzzle complexity increases, the reasoning models initially spend more tokens on the thinking process, but their accuracy gradually declines until a critical point where their reasoning collapses, and their performance drops sharply. At this point, the reasoning effort also decreases, indicating that the models may be "giving up" on the more complex problems.

3. Overthinking and Inefficient Exploration: For simpler problems, the reasoning models often find the correct solution early in their thinking process but continue exploring incorrect solutions, leading to wasted compute. As the problems become moderately more complex, the models first explore incorrect solutions and mostly arrive at the correct ones later in their thought process.

4. Algorithmic Execution Limitations: Even when provided with the algorithm to solve the problems, the reasoning models' performance does not improve, and the observed collapse in performance still occurs at roughly the same point. This highlights the models' limitations in verification and following logical steps to solve a problem.

These findings suggest that current reasoning models, despite their sophisticated self-reflection mechanisms, fail to develop truly generalizable reasoning capabilities beyond certain complexity thresholds. The paper argues that the limitations of these models may not be fully captured by standard benchmarks and that more controlled and systematic evaluation, such as the proposed puzzle environments, is necessary to uncover the inherent limitations of current reasoning approaches.

As the complexity of the puzzles increases, the reasoning models exhibit some surprising limitations in their problem-solving capabilities:

1. Overthinking: For simpler problems, the reasoning models often find the correct solution early in their thinking process, but then continue exploring incorrect solutions. This "overthinking" leads to a waste of computational resources.

2. Logical Execution Failures: Even when the reasoning models are provided with the exact algorithm to solve the puzzle, their performance does not improve. They still exhibit a collapse in performance at roughly the same level of complexity. This suggests that the models struggle with verifying and following logical steps to solve a problem, despite their self-reflection capabilities.

3. Divergent Behavior: The paper observes different behavior between the Claude 3.7 Sonnet Thinking model and the broader Claude family of models. For the Tower of Hanoi puzzle, the Claude 3.7 Sonnet Thinking model's first error in the proposed solution often occurs much later compared to the River Crossing environment.

These findings highlight the limitations of current reasoning models in developing truly generalizable problem-solving capabilities. While they may excel at certain benchmarks, their performance degrades sharply as the complexity of the tasks increases, even when provided with the exact algorithms to solve the problems.

Our findings reveal fundamental limitations in current models. Despite sophisticated self-reflection mechanisms, these models fail to develop generalizable reasoning capabilities beyond certain complexity thresholds.

While our experiments represent a narrow slice of reasoning tasks and may not capture the diversity of real-world or knowledge-intensive reasoning problems, they highlight important limitations in the current generation of large reasoning models.

We observe that as problem complexity increases, these models initially spend more tokens on reasoning, but their accuracy gradually declines until a critical point where performance collapses sharply, and reasoning effort decreases. This suggests inherent limitations in their ability to scale reasoning capabilities.

Interestingly, even when provided with the exact algorithm to solve the problems, the models still exhibit the same collapse in performance, underscoring their challenges in verification and logical step-by-step execution.

These results call into question the current evaluation paradigm that primarily focuses on final answer accuracy, and emphasize the need for more comprehensive assessments that examine the reasoning process and quality, not just the end result.

Ultimately, our findings suggest that while large reasoning models have made impressive strides, there remain significant hurdles to achieving truly generalizable and robust reasoning capabilities. Continued research and innovation will be necessary to advance towards more advanced and reliable reasoning in artificial intelligence.