There are growing concerns that generative AI is cannibalizing itself as it runs out of human-generated content to train on, and must increasingly rely on the AI-generated slop that has been inundating the internet since OpenAI’s ChatGPT 3 was released in 2023. This threatens to collapse the AI industry as studies looking into the problem show that the more these power-hungry programs ingest their own content, the more likely they are to go off the rails.

Referred to as “Garbage In/Garbage Out” (GIGO) by tech columnist Steven Vaughn-Nichols, GIGO is a phenomenon where generative AI such as large language models (LLMs) or image-generating diffusion models that are trained on AI-generated input tend to suffer “model collapse”, with biases and errors present in the pre-existing data being amplified in the new model’s learned experience, eventually resulting in the program producing nothing but useless gibberish.

This issue has been addressed by tech companies such as Anthropic, Google and OpenAI by plugging their LLMs into the internet to search for data that wasn’t in their original training data, previously confined to a closed database. However, this online method, called “retrieval-augmented generation” (RAG), is only delaying the inevitable, as the internet is steadily being flooded by user-prompted AI-generated content, referred to as “slop” in industry circles.

Numerous studies have been conducted in an effort to understand the GIGO model collapse effect, including a 2023 study that found that a model forced to learn from its own data can collapse in as few as nine generations. A more recent study that specifically targeted RAG-trained models found that “RAG can make models less safe and change their safety profile,” and “that even combinations of safe models with safe documents can cause unsafe generations.”

“Large language models (LLMs) can support many tasks but are susceptible to creating unsafe content,” the study, released at the 31st International Conference on Computational Linguistics in April, stated. “These safety concerns include harmful, illegal, offensive, and unethical content, such as spreading misinformation and jeopardizing personal safety and privacy. This can expose service providers to legal risks and undermine public trust in AI.”

“This counterintuitive finding has far-reaching implications given how ubiquitously RAG is used in gen AI applications such as customer support agents and question-answering systems,” warned Amanda Stent, the NLP Architect at Bloomberg’s Chief Technology Office. “The average internet user interacts with RAG-based systems daily. AI practitioners need to be thoughtful about how to use RAG responsibly, and what guardrails are in place to ensure outputs are appropriate.”

While some researchers believe that GIGO-fueled collapse “can be mitigated by mixing synthetic data with fresh human-generated content,” Vaughn-Nichols remarked “what a cute idea. Where is that human-generated content going to come from?”

To truly address the problem of impending model collapse—and the effect that such events have on real-life human users—greater investment needs to be put into the data being consumed by the AI models to prevent errors and biases from corrupting the data to begin with. But, “given a choice between good content that requires real work and study to produce and AI slop,” Vaughn-Nichols points out, “to gain operational efficiency” tech executives will always opt for the freely-available slop.

“We’re going to invest more and more in AI, right up to the point that model collapse hits hard and AI answers are so bad even a brain-dead CEO can’t ignore it,” Vaughn-Nichols concluded. In the meantime, remember to count your fingers before leaving your workstation.