AI Search Optimization: What Google Actually Says Works
Google published its generative AI optimization guide in May 2026 and told site owners to skip llms.txt. Here is what actually decides whether AI assistants can cite your business.
The pitch shows up in your inbox about once a week now. Search is being replaced by AI, your website is invisible to ChatGPT, and for a monthly retainer someone will make your business "AI ready." It usually arrives with two deliverables: a new file to put on your server, and a new kind of content to write.
Google published its own guidance on this in May 2026. It is free, it takes fifteen minutes to read, and it contradicts most of that pitch directly. Worth reading before you sign anything.
What Google actually published
On May 15, 2026, Google Search Central released a guide to optimizing for generative AI features. The companion documentation states the position plainly: "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."
The technical bar is the one you already know. To be eligible, a page must be "indexed and eligible to be shown in Google Search with a snippet." That is it. If you have blocked snippets with a nosnippet tag, you have opted out. Otherwise you are eligible today, with no new file and no new markup.
Two specific recommendations in that guide are worth quoting to whoever is selling you the alternative:
On llms.txt, the file that a good deal of the industry spent last year insisting was mandatory, Google's guidance is not to bother creating one, because Google Search itself does not use them. On structured data, the guide says it "isn't required for generative AI search, and there's no special schema.org markup you need to add." Schema still earns you rich results in classic search, which is a fine reason to keep it. It is not the AI lever it gets sold as.
The guide also tells you not to chop your pages into fragments for machine consumption, and not to rewrite existing content specifically for AI systems. The recommendations that remain are unglamorous and familiar: unique expert or experienced takes that go beyond common knowledge, clear headings and sections, crawlable pages, decent page speed, fewer duplicates.
Then on June 3, 2026, Google added Search Generative AI performance reports in Search Console, giving a dedicated view of impressions inside AI Overviews, AI Mode, and Discover, broken out by page, country, device, and date. It launched to a subset of sites, so check whether yours has it before you pay anyone to estimate the number for you.
The one thing that genuinely changed
There is a real shift underneath all this, and it is not a file format.
Google's documentation describes how the answers get built: AI Overviews and AI Mode may use a "query fan-out" technique, "issuing multiple related searches across subtopics and data sources," to develop a response. Google's own framing is that AI Mode does roughly a dozen searches in the time a person does one.
You are no longer competing for a single query. You are being retrieved for the sub-questions that sit inside somebody's larger question, most of which you never chose as keywords.
That changes what a useful page looks like. A page that answers one narrow question completely, with the specifics intact, is now retrievable inside dozens of larger questions. A page that covers eight topics at surface depth to chase a broad head term is retrievable inside none of them, because it never fully answers anything. As an illustration of the pattern, a landscaping company's honest page on when to winterize sprinklers along the Wasatch Front, with actual dates and actual frost behavior, can get pulled into fall maintenance answers, new homeowner answers, and freeze damage answers. The generic "our lawn care services" page competes for nothing.
That is not a new discipline. It is the same advice good writers have given for twenty years, with a mechanical reason attached to it now.
The part that is actually technical, and often broken
Here is where the real work is, and almost nobody sells it because it does not recur monthly.
The assistants reach your site through named crawlers, and those crawlers do different jobs. Blocking them is easy to do by accident.
OpenAI documents three agents. GPTBot "is used to crawl content that may be used in training our generative AI foundation models." OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features." ChatGPT-User covers fetches triggered by a person asking ChatGPT something, and because a human initiated them, ordinary robots.txt rules may not apply.
Those first two are the ones that matter to you, and they are not the same decision. Blocking GPTBot means your content will not be used to train models, which is a legitimate business position to take. Blocking OAI-SearchBot means you are not eligible to be cited in ChatGPT's search answers. Most site owners want the first and not the second. Most one-click "block AI bots" presets in security plugins and CDN dashboards do both.
Perplexity documents two. PerplexityBot is "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." Perplexity-User handles user-initiated visits and generally ignores robots.txt for the same reason ChatGPT-User does.
If you want search visibility while declining training use, the file is short:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Robots.txt is only half of it. A web application firewall can block these agents before robots.txt is ever consulted, and that failure is silent. Both companies publish their IP ranges so you can allow them properly: Perplexity at perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. If your site sits behind Cloudflare or similar and you have never looked at the bot rules, that is the first place to check. Changes on both platforms take roughly 24 hours to register.
What this will not do for your traffic
Being cited and being visited are different outcomes, and the honest version of this conversation says so.
Cloudflare's bot traffic report, published July 1, 2026, found that 52% of crawler requests are now for AI training, up from 22% in spring 2025, and that more than half of all internet traffic is now non-human. It also reports that Google still accounts for roughly 88% of referral traffic, and that some heavily crawled sectors have seen human traffic fall by as much as 40% in under a year.
Read those two facts together. The assistants are reading far more than they send back, and the channel that still delivers visitors is the one you were already working on. That is an argument for keeping your foundations in good order and making sure the assistants can reach you, not for redirecting the budget into a separate AI product.
I would treat anyone quoting you a precise percentage of traffic you will gain from AI search with real suspicion. The credible public data on that is thin, and most of the circulating numbers come from companies selling the remedy.
What to actually do
- Confirm your pages are indexed and not carrying
nosnippet. That is the whole eligibility requirement for Google's AI features. - Read your robots.txt and your firewall bot rules. Decide GPTBot and OAI-SearchBot separately, on purpose.
- Check whether Search Console has given you the generative AI performance report yet, and use your own numbers rather than an estimate.
- Pick your narrow questions and answer them completely on their own pages, with specifics that only you would know.
- Keep your business facts consistent everywhere. When your address, hours, and services disagree across sources, a confident answer about you is harder to assemble.
- Delete nothing that works in classic search to chase this. Google's own guide says the foundations are the same.
Point five overlaps with local search, and if you are a Utah business that is where the leverage usually is. Our post on Salt Lake City local SEO covers the map pack side of that, meaning how Google ranks you in local results and the address formatting trap specific to Utah's grid. This post is the other half, which is whether an assistant can reach you and cite you at all. Both come out of the same digital marketing practice, usually starting with an audit, because there is no sense buying visibility to a site that quietly blocks the crawlers.
Frequently asked questions
Should I create an llms.txt file? Google's optimization guide says not to, on the grounds that Google Search does not use them. No major assistant currently requires one. It is a small file and it will not hurt anything, but if it is a line item on a proposal you are being charged for a file that Google has publicly told you it ignores.
If I block GPTBot, will I disappear from ChatGPT? No. GPTBot is the training crawler. OAI-SearchBot is the one that surfaces sites in ChatGPT's search features, and they are configured separately in robots.txt. Blocking training while allowing search is a supported and common choice. The mistake to avoid is a blanket rule that catches both.
Do I need to write differently for AI assistants? Google's guidance is explicitly not to rewrite content for AI systems or break it into fragments. The one structural habit worth adopting is answering a single question completely per page, because fan-out retrieval pulls pages in to answer sub-questions inside larger ones. That is good writing practice regardless.
If you are not sure whether your site is reachable by these crawlers, that is a short check rather than a project, and we will tell you plainly if the answer is that nothing needs doing.