Researchers at Mila have developed a technique aimed at enhancing the efficiency of large language models (LLMs) in complex reasoning tasks. This method, termed Markovian Thinking, enables LLMs to engage in extensive reasoning without the high computational costs usually associated with such tasks.
The team’s implementation, known as Delethink, organizes the reasoning process into fixed-size segments, addressing the challenges posed by the current standard methods. Initial assessments indicate that this approach can reduce training costs by over two-thirds for a model with 1.5 billion parameters compared to conventional methods.
Typically, LLMs must produce a series of intermediate reasoning tokens—often referred to as chain-of-thought (CoT)—to tackle complex problems. While using reinforcement learning (RL) to extend CoT has improved reasoning capabilities, the standard approach faces a significant challenge: the computational cost increases quadratically as more tokens are generated. Most strategies to manage this expense either limit the reasoning process or favor shorter solutions, which remains within the restrictive LongCoT framework.
In contrast, Mila’s approach restructures the RL setup, allowing the model to maintain a constant reasoning context while separating the duration of reasoning from the amount of context to process. The Markovian Thinker confines the computing effort to linear rates, making it more efficient for LLMs to engage in lengthy reasoning tasks.
Delethink requires the model to work within fixed token intervals, resetting the context after each segment while carrying over essential information. This allows the model to summarize prior reasoning, addressing concerns about memory retention of critical details.
In testing, the researchers trained R1-Distill-1.5B using Delethink on competitive math problems, yielding results that either matched or exceeded those of models trained with the standard LongCoT method. Notably, improvements were observed even beyond the model’s training constraints, with significant benefits for enterprise applications.
The ability of off-the-shelf reasoning models to perform in a Markovian manner without specialized training has practical implications for developers. Overall, the success of Markovian Thinking suggests potential advancements in AI capabilities, allowing models to engage in extensive reasoning tasks and potentially contributing to scientific discovery.
Source: https://venturebeat.com/ai/new-markovian-thinking-technique-unlocks-a-path-to-million-token-ai