A BBC-led international study on the reliability of popular AI assistants has found that 45 percent of the bots’ responses to news and current event queries contained “at least one significant issue of any type,” according to the study text. These findings are troubling in light of the increasing use of these chatbot-powered tools by online users, and the sizable amount of trust put in the information being provided.

Titled “News Integrity in AI Assistants” and coordinated by the European Broadcasting Union (EBU), the study involved 22 major news broadcasters from 18 countries representing 14 different languages, evaluating the reliability of Microsoft’s Copilot, OpenAI’s ChatGPT, Google’s Gemini and the privately-owned Perplexity AI assistants.

While the results of the study found that the overall reliability of AI assistants had improved over those form an earlier BBC study, with overall responses containing “significant issues of any type” having dropped from 51 to 37 percent, the new study found that “45% of responses contained at least one significant issue of any type,” and that “errors remain at high levels, and that they are systemic, spanning all languages, assistants and organizations involved.”

Roughly one in three responses from Copilot, ChatGPT and Perplexity displayed a significant issue, but for Gemini it was significantly higher, with around half of the responses plagued with at least one issue. Overall, one in five responses “had significant issues with accuracy.”

However, it was sourcing that proved to be “the biggest cause of problems, with 31% of all responses having significant issues with sourcing,” a situation where the information provided in the AI-generated response was “not supported by the cited source, providing no sources at all, or making incorrect or unverifiable sourcing claims,” according to the study. Ironically, Google’s Gemini proved to be the worst offender in this regard, offering incorrect sourcing in 72 percent of its responses, more than triple the 30 to 37 percent rate displayed by ChatGPT’s 24 percent and Copilot and Perplexity’s 15 percent.

This degradation in reliability is important because of the increasing adoption by the public of these AI assistants over traditional search engine services, with seven percent of consumers of online news having already made the switch, a number that rises to 15 percent amongst users under 25 years of age.

“This research conclusively shows that these failings are not isolated incidents,” according to EBU Media Director and Deputy Director General Jean Philip De Tender. “They are systemic, cross-border, and multilingual, and we believe this endangers public trust. When people don’t know what to trust, they end up trusting nothing at all, and that can deter democratic participation.”

“We’re excited about AI and how it can help us bring even more value to audiences. But people must be able to trust what they read, watch and see,” BBC Programme Director Peter Archer stated in regards to the rise of generative AI services.

“Despite some improvements, it’s clear that there are still significant issues with these assistants,” he added. “We want these tools to succeed and are open to working with AI companies to deliver for audiences and wider society.”

Alongside the report, the study authors also published a “News Integrity in AI Assistants Toolkit” that offers guidance in developing solutions to the dilemma presented by the steady degradation of reliability in AI services, with suggestions ranging from the improvement of existing services, to increasing the level of media literacy amongst its users.

The EBU and participating members are also urging both European Union and national-level regulators to increase enforcement of existing information integrity laws, digital services and media pluralism; they also stress that it is essential that AI assistants be the subject of ongoing independent monitoring efforts, to parallel the continued development of publicly-available AI tools.

“If I were an intelligent machine, I would deceive you,” Whitley was told during an insightful encounter chronicled in his 2001 book, The Key. Pick up a copy and join this revealing conversation today!