Google's AI Is Sending Searchers to 404s: 1,017 Invented URLs Across 10 Sites

A few weeks ago we shipped a new layer in our platform. It merges our technical SEO agents and page monitoring with Search Console and AI visibility data, so when something changes on a client's site that needs fixing, we hear about it as it happens rather than in next month's audit. Within days of going live, it caught something none of those sources would have caught on its own.

In mid-September, the status column started turning red for pages nobody had touched. The monitoring checks the live HTTP status of every URL a client ranks for or gets cited on, and puts it next to that URL's Search Console impressions and AI citations. Across several unrelated clients, the same thing appeared at once: URLs with a handful of impressions each, returning 404, that none of them had ever published.

My first guess was that a client had changed something without telling us: a new SEO plugin, an AI content experiment, a slug cleanup. But the same pattern hit unrelated sites in the same week, and none of them had touched a thing. So I pulled the data.

1,052

URLs in Google AI Features reports that return 404

94%

of them get every impression from AI features

68%

of the real pages behind them are cited in AI answers we track, vs 29% of other pages

Aloha first-party data. Share of 404 URLs in each site's AI Features report whose Web impressions come 100% from AI features.

It Started on a Specific Day

We keep daily, page-level Search Console history for our clients, so I can date the first day Google recorded an impression for any URL. I took every URL that first appeared from late August on, checked each one live, and counted the ones that return 404 by the day they first showed up.

There's a small rise on September 1 and a sharp step on September 11. Total AI Features impressions moved by a few tens of percent at most that week, nothing like a fivefold step. Google didn't start showing these sites much more. It started showing them at different addresses.

Proving Where the Impressions Come From

Search Console now has a Generative AI Features view, and its page-level export lists impressions from Google's AI surfaces. I exported it for all ten sites, then did two things:

1. Requested every URL in those exports live, plus every other page with Web impressions on the same sites: 10,891 URLs in all. (One site rate-limits bulk checks, so for it I checked only the AI Features URLs.)
2. Pulled each URL's total Web impressions from the Search Console API for the same dates, and compared them URL by URL with its AI Features impressions.

A page a site really deleted keeps showing up in regular results for a while. The invented URLs have never appeared in a regular result at all. They exist only inside AI answers. For comparison, live pages on the same sites get 4% to 27% of their impressions from AI features, and the rest from regular search.

Invented, Not Deleted

To separate invented URLs from real removals, I matched every dead URL against the pages on the same site that currently return 200, and checked where its impressions come from.

The largest cluster belongs to a single article on Site 1: 30 different invented spellings of one URL, all returning 404.

How the AI Gets URLs Wrong

The mistakes look like what a language model does when it writes a link from memory instead of copying it: it rewrites the slug the way it would phrase the title. I tagged every invented variant by how it differs from the closest live URL. One URL can carry several changes.

Some go further than a typo. On one site the model wrapped the word "article" around a real slug, producing things like /guides/artisourdough-starter-guidecle. It also invented hostnames, www.www. and www.com. in front of the real domain, which fail before any server answers.

Per-Site Numbers

The share of URLs looks alarming; the share of impressions does not. Each invented URL is shown a handful of times, and the real page keeps the bulk of the exposure.

Still, an impression on an AI answer that cites your page is the moment someone decides to click through, and for these URLs the click lands on a 404.

Caught Inside an AI Overview

Search Console tells you an invented URL was shown. It doesn't show you the answer it was shown in. For that I went back to the AI answers our visibility platform stores for every tracked query: every AI Overview and AI Mode response since June 1 for seven of these sites, over 3,500 distinct client URLs in all.

Almost all of them are correct. But four invented URLs from the Search Console reports turn up inside AI Overviews we captured, in 36 answers across nine queries, all in September. In every capture where the link type is recorded, they sit in the same place: an inline link in the answer text, never the source card.

Here's the shape of one:

Query: "what do cats eat?" · Google AI Overview

"Cats are obligate carnivores and need animal protein in every meal."

Source cards under the answer: example.com/what-do-cats-eat/ (the real page) and rival.com/blog/pets/what-do-cats-eat (a competitor).

The inline link: the site's domain with the competitor's path, /blog/pets/. That URL has never existed. It returns 404, and it's the one Search Console reports impressions for.

The source card and the sentence cite the same page, but only the card gets the address right. In the real case, the model borrowed the URL structure of the other site it was quoting. And the source cards are what AI citation trackers record, which is why they show the correct URL while Search Console shows the broken one.

Captured AI Overview · where the invented URL comes from

AI Visibility · Captured answer · Sep 13

what do cats eat?

AI Overview

AI Overview

Cats are obligate carnivores and need animal protein in every meal. Most adult cats do well on two portioned meals a day, with fresh water always available.

Inline linkexample.com/blog/pets/what-do-cats-eat/404path copied from rival.com

Source card Correct URL

What Do Cats Eat? A Vet-Approved Guide

example.com/what-do-cats-eat/ 200

Source card · competitor

What Should You Feed Your Cat?

rival.com/blog/pets/what-do-cats-eat

The AI Garbles the Pages It Cites Most

These aren't random dead links. For every invented URL I know the real page it copies, so I checked those real pages against the same AI answer archive.

The pages that get garbled are the pages Google's AI leans on. The better a page does in AI answers, the more chances the model gets to write its URL wrong.

Pro tip

Don't catch-all redirect these 404s to your homepage. Google treats that as a soft 404, and it throws away the one useful thing each variant tells you: which page the AI meant. Map each one to the page it copies, or leave it as a 404.

What to Do About It

• Export the AI pages report: In Search Console's Performance report, open the Generative AI Features view, switch to the Pages tab and export it.
• Check every URL's status: A 404 that closely resembles one of your live slugs is an invented variant, not a broken page on your side.
• Redirect what carries impressions: Map each variant to the live page it copies with a 301. A near-duplicate slug almost always has one obvious target.
• Skip the long tail: Most variants get one or two impressions and may never recur. Start with the ones that carry the most.

Do It Yourself With Claude Code

You don't need our platform to run this check. Put three things in one folder, open Claude Code there, and paste the prompt below.

• AI export: The Generative AI Features export from Search Console (the zip or its Pages.csv).
• Web export: The regular Performance → Pages export for the same date range. It's optional, but it's how you tell invented URLs apart from pages you really removed.
• Sitemap: Your sitemap URL, so Claude knows which pages are live.

The mapping works in tiers, and most URLs never need the clever ones. Formatting fixes (case, trailing slash, doubled folders) resolve a few exactly. String similarity handles the bulk, since most invented URLs are one word away from the real one. Only the reworded slugs, about one in ten in our data, need judgment, and Claude can make that call by reading the candidate pages' titles.

I want to find and fix URLs that Google's AI features are inventing for my site.

Files in this folder:
- AI export: the Search Console "Generative AI Features" Performance export
  (zip or Pages.csv). Required.
- Web export (optional): the regular Search Console Performance Pages export
  for the SAME date range.
- Sitemap: https://YOURSITE.com/sitemap.xml (follow sitemap indexes).

Write a small Python script (standard library only) so I can re-run it, and do this:

1. Parse the AI export into (url, ai_impressions). Keep only URLs on my domain.
2. Check every AI-export URL live: GET without following redirects, an honest
   user agent, at most 2 requests at a time, back off and retry on 429/503.
   Record the status code, and the Location header for any 3xx.
3. Build the list of live URLs: every sitemap URL that returns 200 (checked the
   same way), plus AI-export URLs that returned 200.
4. For every URL that returns 404 or 410:
   a. If I gave you the Web export, compute non_ai = web impressions minus AI
      impressions. If most impressions are non-AI, label it "removed page"
      (a real page I deleted), not invented.
   b. Normalize it (lowercase, strip trailing slash and www, collapse doubled
      path segments, underscores to hyphens). An exact match with a live URL
      is a "formatting variant".
   c. Otherwise score path similarity against every live URL (difflib ratio).
      0.85 or higher = "invented variant", high confidence. 0.70 to 0.85 = medium.
   d. Below 0.70, fetch the titles of the top 3 candidates and pick a target
      only if it clearly answers the same question. Otherwise leave it unmapped.
5. Rules: never map to the homepage or a category page, never map across a
   different topic, and leave a URL unmapped rather than guess.
6. Give me:
   - redirects.csv: from_url, to_url, class, confidence, similarity,
     ai_impressions, non_ai_impressions
   - a summary: count per class, total impressions on 404s, top 20 by impressions
   - 301 rules for HIGH-confidence rows only, sorted by impressions, in the
     format for my stack: [nginx / Apache / Vercel / Netlify / Next.js / Cloudflare]
Don't change anything on my server. Just give me the files.

Review the medium-confidence rows by hand before you ship them. A slug that's 75% similar can still be a different article, and a wrong redirect is worse than an honest 404.

What We Don't Know Yet

• AI Overviews vs AI Mode: Every invented URL we caught was inside an AI Overview. Our AI Mode captures show none, but the Search Console AI report doesn't split the two, and the API has no AI-features filter at all.
• Behavior or reporting change: The step on September 11 sits one day after the multimodal data begins. Google may have started logging links it was already showing.
• What the searcher saw: Impressions say the URL was displayed. Nobody outside Google can see the rendered answer behind them.

What This Means for How We Monitor Sites

We found this because our platform already puts three things on the same row: what Search Console says, what the AI engines cite, and what the URL returns right now. None of those alone would have surfaced it. Search Console shows impressions but not status. Citation trackers show the source cards, which carry the correct URL. A crawler only visits URLs that exist on your site. The invented URLs only become visible where the three meet.

AI search is now a referral source you can't audit by looking at the answer, because the answer you can scrape isn't necessarily the answer that was served. We're adding this check to how we monitor every client, so invented variants get flagged and redirected the week they appear.