A recent study conducted by Nous Research highlights that open-source artificial intelligence models consume significantly more computing resources compared to closed-source models when performing the same tasks. This finding raises questions about the cost-effectiveness of open-source AI in enterprise applications.
The research found that open-weight models utilize 1.5 to 4 times more tokens, the fundamental units of AI computation, than closed models, such as those developed by OpenAI and Anthropic. Notably, for simple knowledge questions, the disparity can increase to a staggering 10 times more tokens used by open models. This raises concerns about the economic viability of open-source models, as their lower per-token costs may be offset by their increased token usage.
This study examined 19 different AI models across three categories: basic knowledge questions, mathematical challenges, and logic puzzles. The concept of “token efficiency” was evaluated, which assesses how many computational units are employed in relation to the complexity of the solution provided. The findings suggest that the assumption that open-source models provide clear economic benefits may need reevaluation.
Closed-source models appear to be iteratively optimized for efficiency, leading to lower token usage and reduced inference costs. Conversely, some newer open-source models have increased their token consumption, focusing instead on reasoning performance. The researchers observed marked differences between model providers, with OpenAI’s models, particularly the o4-mini and gpt-oss variants, demonstrating superior token efficiency, especially for mathematical problems.
These results prompt enterprise leaders to reconsider their evaluations of AI models. While accuracy and per-token prices are often prioritized, the total computational requirements for real-world applications can lead to significant cost variations. The research is comprehensive and includes methodologies that tackle the challenge of measuring efficiency across different model architectures, providing insights for future improvements in AI model development. The complete dataset and evaluation protocols are accessible on GitHub for further research.
Source: https://venturebeat.com/ai/that-cheap-open-source-ai-model-is-actually-burning-through-your-compute-budget/