AI's brain is getting bloated with AI content

The big picture: In simulations of AI search, models that relied on AI-generated reference material became increasingly likely to produce the same recommendations, according to new research from Graphite shared first with Axios.

• Graphite advises companies on how to improve their visibility in AI search.

Driving the news: Graphite's paper argues that AI search tools can experience "AI search collapse" when they retrieve AI-generated pages derived from earlier AI answers.

• Graphite borrows the phrase from "model collapse," a risk described in a 2024 Nature paper showing that generative models can degrade when repeatedly trained on outputs from earlier models.

Catch up quick: Early consumer chatbots were often outdated because they answered mainly from training data, not live web results.

• When companies added web search to chatbots, the systems could retrieve current pages and use them to ground answers.

The intrigue: Graphite previously found that AI-generated content made up around half of all article-style web pages.

• If chatbots use those pages as source material, their answers could become less reflective of human taste, judgment and firsthand experience.

Zoom in: A study by Wharton researchers Gideon Nave, Christian Terwiesch and Lennart Meincke found that individuals who use ChatGPT as a research partner generated stronger ideas, but groups of people that use it tended to converge on similar concepts.

• Graphite's research suggests a similar narrowing could happen across AI search.

What they're saying: "If we're all using our brains less and just go to the LLM, we're all going to get the results that the LLM came up with," Nave tells Axios. "Then we're going to lose diversity and randomness and exploration."

What they did: Graphite used models from OpenAI, Gemini and Anthropic APIs to run simulations that allowed them to provide specific context to the models.

• The firm ran several experiments testing what could happen when AI search retrieves AI-generated content.
• The scenarios included questions like, "Who are the best Twitch streamers currently?" and "What is the best restaurant in San Francisco?"
• Graphite started its experiments with real web pages surfaced by commercial AI search systems. Then it used AI-generated answers to create new reference pages for later rounds.
• "After converting the model's first-round answers into articles and using them as references, the distribution begins to shift," Graphite says in the report.

Yes, but: The study does not prove that real-world AI search is already collapsing or that the internet will inevitably converge on one perspective.

• The research is a simulation from a company with a stake in AI search visibility, not peer-reviewed academic research.
• An OpenAI spokesperson told Axios that the company is "constantly refining" its search indexing, ranking and model behavior.

• "We are always improving our methods," an Anthropic spokesperson told Axios.

State of play: Brands are already trying to game AI search.

• Companies are publishing chatbot-friendly content on their own websites in hopes of being cited by AI systems and steering users toward their products.
• The practice is known as GEO, or generative engine optimization — a new version of SEO aimed at influencing what AI systems cite, summarize and recommend.
• "The race to sway [consumer] decisions is spurring some strange experiments," Will Oremus wrote in The Atlantic earlier this month.
• Companies are posting self-promotional ranking listicles on their websites or paying armies of Redditors to post about their products.

Flashback: Search has always attracted people trying to reverse-engineer visibility.

• In the 1990s and 2000s, marketers and SEO purveyors tried to perfect the art of "keyword stuffing." They added popular search terms directly to pages, hid them in pages or showed one version of a page full of keywords to Google's crawler and another to human visitors.
• The goal is no longer just to rank high in links, but to shape the synthesized answer.
• Google continues to fight this kind of spam and has updated its "scaled content abuse" policy to keep pages designed to manipulate search out of top search results.

The other side: Some convergence is useful.

• For factual queries, users want the same correct answer. Variety is not the goal when you ask AI who won the 2024 Super Bowl or what time a store closes.
• But now that Google has gone all-in on AI search, users are less likely to see the source material behind answers.

The bottom line: AI might make web search more efficient but less serendipitous.