Google is currently testing a new AI-driven search tool called Scholar Labs, which is intended to provide answers to detailed research inquiries. However, the demonstration raised questions about the reliability of identifying high-quality scientific studies. The tool does not rely on traditional metrics, such as citation counts or impact factors, which are often used to evaluate the quality of a study. How will this shift in approach be perceived by the scientific community?
The tool leverages AI to understand a user’s query and identify relevant topics and relationships. Currently, it is available to a limited number of logged-in users. The initial demonstration included a query about brain-computer interfaces (BCIs), resulting in a review paper from 2024 published in Applied Sciences. The Scholar Labs tool provides insights into why certain results are chosen, highlighting the content’s relevance to the user’s inquiry.
Notably, Scholar Labs lacks filters commonly used to assess research quality. Traditionally, citation counts and journal impact factors serve as proxies for identifying reputable studies. For example, Applied Sciences reports an impact factor of 2.5, while Nature cites an impact factor of 48.5. The original Google Scholar platform offers options to rank studies based on relevancy and citation counts, while Scholar Labs focuses on the full text of papers to determine relevance.
Google’s rationale for omitting citation counts and impact factors is that these metrics can be misleading in specific research contexts. Critics, however, contend that these measurements often serve as essential indicators of trustworthiness in scientific literature. Some researchers, although recognizing the limitations of citation counts, admit to relying on them as a quick assessment tool.
The Scholar Labs feature allows users to specify the recency of papers in their queries. Google plans to incorporate user feedback into the tool’s development and has created a waitlist for access. While some experts believe that AI can enhance the discovery of relevant literature, they emphasize that scholars should remain engaged with the literature to make informed judgments about quality and rigor.
Source: https://www.theverge.com/news/823213/google-scholar-labs-ai-search