Should You Block AI Crawlers? A Small Business Answer

A deadline that was not written for you

On July 1, Cloudflare announced that it is changing the default treatment of AI crawlers. Starting September 15, 2026, the new defaults allow search but block training and agent use on pages that display ads. Crawlers that cannot tell you which of those three things they are doing get blocked on ad pages entirely. The defaults land on new customers, new sites belonging to existing customers, and existing free plan customers who have not changed their settings by that date.

Cloudflare sits in front of a large share of the web, including many small business sites whose owners have never opened the dashboard. The announcement traveled, got summarized, and landed in a lot of inboxes as a vague instruction to block the AI bots before the deadline.

For most of the businesses we work with, that instruction is backwards.

Publishers and service businesses are not in the same trade

The case for blocking is real, and it belongs to publishers. If your business is producing articles people read next to ads, a crawler that takes the article and answers the question elsewhere has taken your inventory and returned nothing. That is the arrangement Cloudflare is renegotiating, and the frustration behind it is legitimate.

Now describe a dog boarding facility, a cabinet shop, a therapy practice. The content on those sites is not the product. It is the pitch for the product. The service is delivered in a building, by people, in a place. Content that never gets read has zero value, so anything that reads it is doing you a favor.

That distinction decides the whole question. A publisher loses when a machine reads the page instead of a human. A local service business loses when nothing reads the page at all.

The 2026 Bad Bot Report put automated traffic at 53 percent of the web during 2025, up from 51 percent the year before. Read that as a business owner rather than a security engineer. Slightly more than half of what arrives at your site is not a person, and a growing share of that half is acting for somebody about to make a decision. Some of it is junk. Some of it is a customer with an intermediary.

Three categories, in plain language

Cloudflare split the old single switch into three, and the vocabulary is worth learning because it is becoming standard.

Search covers bots that index your page so a platform can point somebody at it later. This is the traditional bargain: you get crawled, you get referrals. Allow it. There is no version of your business that improves by leaving the index.

Agent covers software fetching your page in real time because a specific person asked it something specific right now. Somebody typed a question and an assistant went to look. This is the closest thing to a warm lead that exists in automated traffic, and it is the category people are most likely to switch off by accident. Allow it.

Training covers crawlers taking your content into a model's weights. This is the only one where reasonable owners land in different places. There is no referral, and the payoff is diffuse and unprovable. If you want to say no here, say no. It will not hurt your rankings and it will not stop an agent from reading your hours.

That is the whole decision for most small sites: yes, yes, and your call. It takes about two minutes once you find the panel.

The mistake that actually costs money

The expensive error is not choosing wrong among those three. It is reaching for a blunter tool.

Google's own documentation on AI features and your website is direct about this: robots.txt directives for Googlebot are the control for how your site is crawled for Search, and AI features are built into Search rather than bolted onto it. There is no line you can add that keeps you in the blue links and out of the AI answer. Blocking Googlebot to stay out of AI Overviews removes you from Google.

If your concern is specifically about training and grounding in Google's other systems, Google-Extended is the separate token for that. Google's documentation states plainly that it does not affect a site's inclusion in Search and is not a ranking signal. If your concern is how much of your text gets shown, the snippet controls are nosnippet, data-nosnippet, max-snippet, and noindex. Those are scalpels. Disallowing Googlebot is an amputation.

This is a different question from the one we covered in AI search optimization. That post is about whether your content is good enough to get cited once a machine reads it. This one is about whether the machine is allowed through the door at all. Getting the door wrong makes the content question irrelevant.

A middle setting most people have not heard of

If blanket allow feels too loose and blocking feels too severe, there is a third option that costs nothing.

Cloudflare's Content Signals Policy, published in September 2025, defines three machine readable signals you can express in robots.txt: search for building an index and returning links, ai-input for feeding your content into a model in real time, and ai-train for training or fine tuning. Each takes yes or no, comma delimited:

User-Agent: *
Content-Signal: search=yes, ai-train=no
Allow: /

Two honest caveats. This states a preference rather than enforcing one, and a crawler that ignores it is stopped by nothing but a real block. And omitting a signal means you have expressed no preference for that use, which is not the same as permitting it.

What ours says

Go read articulatevision.net/robots.txt right now. It is four lines. It allows everything except /api/, and points at the sitemap. Nothing about AI, nothing blocked, no exceptions carved out.

That is deliberate on a site whose entire job is to be found by somebody in Salt Lake City with a problem we can fix. Nobody arrives at our blog and stops, satisfied, having consumed the good. They read something that suggests we know the work, and some fraction fill out a form. Anything carrying our pages further into places people ask questions is working for us.

The check that matters more than the toggle

Here is where small business sites actually fail, and it has nothing to do with crawler settings.

An agent fetching your page on somebody's behalf usually takes the raw HTML. Many of them do not run JavaScript, and the ones that do are on a clock. If your site was built on a page builder that assembles content in the browser, the machine can receive a nearly empty shell while you look at a perfectly good page.

You can test this in one command:

curl -s https://yoursite.com/ | grep -i "555-1234"

Swap in your phone number. Then try your address, hours, prices, and main service. If those strings are not in the raw HTML, they do not reliably exist as far as an agent is concerned. Somebody's assistant just told them your competitor lists hours and you do not.

We ran that check on our own homepage while writing this. The phone number is in the source, along with four blocks of structured data. That is the baseline rather than a brag, and a surprising number of small business sites do not clear it.

This is the same discipline as building for accessibility: content that only exists after JavaScript renders it is content some visitors never get. Screen readers and AI agents fail on the same sites for related reasons, which is a good argument for fixing it once and collecting both benefits.

Where this lands for a local business

We rebuilt The Doggy Den's site and stayed on to maintain it. The results we publish are theirs and specific: 2.4 times the organic search traffic since the relaunch, and 203 calls a month straight from their Google Business Profile averaged over six months. None of those calls came from someone who read a blog post and felt intellectually satisfied. They came from somebody who needed boarding, found accurate information fast, and picked up the phone.

Every channel carrying accurate information about a business like that is worth being on, and those channels increasingly ask a machine first. Being legible to the machine is now part of being findable at all, which is the argument we made about what moves the map pack, one layer further out.

Set the crawler categories in an afternoon. Then spend the real effort on what outlives every platform change: complete, accurate, server rendered information about what you do, where you are, and what it costs. That has been the answer through ten years of algorithm updates.

Frequently asked questions

Do I need to do anything before September 15? Only if you are on Cloudflare's free plan and your site displays ads. Most small business sites do not, so the defaults will not change anything for you. Check your settings anyway. Knowing what they say beats discovering it later.

If I allow AI crawlers, am I giving away my content for free? Partly, and it depends what your content is. If your pages describe a service delivered in a specific place by specific people, the text was never the asset. If you publish original research people pay to access, the publisher logic applies to you instead.

Does blocking AI crawlers hurt my Google rankings? Blocking Googlebot does, severely, because that is the same crawler powering Search. Blocking a third party AI crawler does not affect Google at all. The distinction is which user agent you name, which is exactly why blunt toggles are dangerous.

How do I know which bots are actually hitting my site? Your hosting or CDN analytics will show user agents, and Cloudflare's dashboard now breaks them out by category. Look before you decide. The traffic mix on a five page local site looks nothing like a news site's, and most advice online was written about the second one.

If you are not sure what your site currently tells a crawler, send us the URL. We will read the robots.txt, fetch the raw HTML, and tell you in plain language what a machine can and cannot see. Our website team does this as the first step of any care plan.

Get your site checked