Understanding AI Overview Fails: Pitfalls and How to Avoid Them

Has Google told you to stick glue to your pizza lately? What about adding a rock per day to your diet?

Not the type of advice that I would follow.. But that hasn’t stopped Google’s new AI Overview feature from making these and other equally ridiculous recommendations over the past few months. 

It’s no wonder Google is now showing AI Overviews for far fewer results. Research by BrightEdge finds AI Overviews appear for just 7 percent of searches, down from 11 percent in June. 

Some of these results may be funny for the average searcher. But for brands, they’re no laughing matter. Google AI overview fails can damage brand reputation and alienate customers. 

That’s why I’m dedicating this article to helping you understand what’s wrong with AI Overviews. I’ll cover the most common types of AI Overview fails, why they occur, and the steps you can take to protect your brand. 

Key Takeaways

• AI Overview fails occur when Google’s new AI feature produces an incorrect, inaccurate, or irrelevant answer. 
• These failures occur because Google’s AI tools can’t actually understand the sources they cite. They have a particular problem with irony, which means they present satirical content as fact. 
• AI failures can be detrimental to brands, causing them to appear alongside inappropriate content or inadvertently mislead customers. 
• You can prevent your content from appearing in AI Overviews, but it will also remove your site from Google’s index. 
• Google has already made several changes to its new feature, including reducing its reliance on Reddit and other user-generated content and improving safeguards for health topics. 

Table of Contents

• Key Takeaways
• What AI Overview Fails Are Users Seeing?
  • Inaccurate answers
  • Outdated answers
  • Inappropriate content
  • Irrelevant content for the query
• Why AI Overviews Are Having Problems
• Avoiding AI Overview Fails: What You Can Do And What Google Is Doing
  • What can you do?
  • What’s Google doing to prevent these issues?
• FAQs
• Conclusion

What AI Overview Fails Are Users Seeing?

Google designed AI Overviews to do the Googling for you, summarizing the most relevant search results so you don’t have to waste time wading through dozens of links. 

But the feature often doesn’t work as intended. Below, I’ve curated some of the most common types of bad answers from Google AI. I’ve also included findings from our own research, in which we asked 623 U.S. digital marketers how often they’ve seen these errors. 

A word of warning, though: you probably won’t be able to recreate these results. Google has manually edited the results to prevent these fails (and ones like them).

Inaccurate answers

Inaccurate answers are probably the most common type of AI Overview fail. The feature has a penchant for drawing from inaccurate or ironic sources and presenting patently false information as fact. 

Take this example where Google suggests 13 US presidents have attended UW-Madison:

Anyone with an elementary school education knows it’s not possible for Andrew Jackson to have graduated in 2005, or for Andrew Johnson to study from 1947 through to 2012, earning 14 degrees in the process. 

This was the most common type of AI fail in our study, encountered by 67.35 percent of respondents. 

Outdated answers

AI Overviews don’t always use the most up-to-date sources, which can lead to outdated and inaccurate answers.

The most widely shared example of this is Google suggesting the U.S. has had one Muslim president, Barack Hussein Obama.

Here, it’s citing an academic study that makes reference to a now-debunked theory that President Obama is a practicing Muslim. 

This was the third most common issue in our survey, encountered by 58.02 percent of respondents.

Inappropriate content

Sometimes AIO content isn’t just inaccurate, it’s inappropriate. 

Take this example from Joe Youngblood, where Google details the health benefits of running with scissors:

Another more serious error is the recommendation to drink urine to flush kidney stones:

Inappropriate content was the second most common fail in our survey, seen by 60.06 percent of respondents. This is especially important, knowing the potentially serious impact these recommendations could have.

Irrelevant content for the query

Sometimes, AI Overviews don’t even provide a relevant answer. That’s what happened to Jake Kleinman, the executive editor of Inverse, who wanted to find the biggest box office flops of 2024. 

Instead of actual figures, AI Overviews provided a speculative list with made-up box office numbers. 

This was the least common failure in our survey, but over half (53.06 percent) of respondents still experienced it.

Why AI Overviews Are Having Problems

AI Overviews use a custom version of the Gemini large language model (LLM) that incorporates Google’s Search systems. This means they work differently from chatbots and other AI tools like the ones that let you write marketing copy using artificial intelligence.

They don’t just use training data to generate an answer. They also parse Google’s index, identifying and summarizing the best, most relevant results for the query in question. 

This means that while it may not hallucinate like ChatGPT, it has a tendency to misinterpret questions and provide inaccurate answers. 

AI Overviews seem to have a particular issue with humor and irony. Take AIO’s suggestion to add glue to pizza. Internet sleuths traced the origins of this recommendation back to an ironic comment in a 12-year-old Reddit thread. 

The presidential example I listed above came from this clearly satirical article:

So these answers are Google’s fault, right? It’s a problem with AI Overviews technology. 

Well, Google isn’t taking all of the blame. In an article published in the wake of poor publicity, head of search Liz Reid also blamed the internet for causing problems:

“There’s nothing quite like having millions of people using the feature with many novel searches. We’ve also seen nonsensical new searches, seemingly aimed at producing erroneous results.”

Reid also claims some of the widely shared screenshots were fake—a claim that has since been proven.  

It’s important to note that the majority of respondents in our survey were positive about the quality of AI Overviews. Over 80% rated it between 8 and 10 (out of 10) when asked about their confidence in its ability to answer questions about their industry.

Avoiding AI Overview Fails: What You Can Do And What Google Is Doing

Google AI’s bad answers can be detrimental to brands in several ways:

• Associating their company with false or misleading content
• Incorrectly summarizing their work
• Reducing SERP clickthrough rates 

In fact, almost two-thirds (60 percent) of our survey respondents said they were concerned or very concerned about the potential of AI overviews to spread misinformation or biased content, with 61.50 percent saying they felt inaccurate or inappropriate answers would turn people from Google to ChatGPT or other LLMs for answers.

You can’t fix AI Overview’s inaccuracies (Google is working on that), but there are a couple of ways you can better control the narrative.  

What can you do?

Removing all or part of your site from Google’s index is the only way to guarantee AIOs don’t feature your content. This will only make sense in very rare circumstances (if you have private content on your site, for example), but here are two ways to do it:

• Use disallow directives in robots.txt to stop Googlebot from crawling all or part of your site.
• Use the data-nosnippet HTML tag to prevent parts of your page from appearing in search snippets

Finally, spare a thought about your Google Ads strategy. Given that Google is experimenting with ads inside AI Overviews, you may want to think twice about these placements lest your brand is associated with inaccurate, inappropriate, or misleading content. 

What’s Google doing to prevent these issues?

Google has already taken action to reduce and prevent AI Overview failures like the ones above.  Writing in a blog post published on May 30, 2024, Head of Google Search Liz Reid said:

“From looking at examples from the past couple of weeks, we were able to determine patterns where we didn’t get it right, and we made more than a dozen technical improvements to our systems”

These changes include:

• Better detection of nonsensical queries
• Less reliance on user-generated sites like Reddit and Quora 
• Restricting instances when AI Overviews are not helpful
• Strengthening guardrails on health-related topics

As I highlighted in the introduction to this article, there’s evidence Google has slashed instances of AI Overviews, too.   

FAQs

Conclusion

Unless you work for Google or want to forgo search traffic completely, you can’t do much to stop these kinds of AI fails. But you can try to optimize for them to better control the narrative and improve the results. 

Start by reading my introductory guide to AI Overviews. It will teach you what they are and how you can rank for them. Then, spend your time on creating relevant, high-quality content, so that when your site is featured in an AI Overview, it’s the best result possible. 

See How My Agency Can Drive More Traffic to Your Website

• SEO - unlock more SEO traffic. See real results.
• Content Marketing - our team creates epic content that will get shared, get links, and attract traffic.
• Paid Media - effective paid strategies with clear ROI.

Book a Call