Your infrastructure may be quietly deciding your AI visibility for you (check before 15 September)
By Alex Golombeck — The Happy Cat, Manchester. Published August 2026.
Last week, while fixing a client's website, we found something that shouldn't have been possible: a robots.txt file nobody had written, blocking almost every crawler on the internet from the entire site.
It belonged to the hosting company. When no robots.txt file exists on a site, this host quietly serves its own default — a whitelist template that admits Google and Bing and bans everything else, including every AI crawler in existence. The site's owner had no idea. There was nothing to see in WordPress, nothing in any dashboard she used. Her website had likely been invisible to AI assistants for years, because of a template her host wrote before AI assistants existed.
We assumed we'd found an oddity. Then we looked at what the rest of the internet's infrastructure is doing, and it turns out the oddity is becoming the rule.
The default is flipping
In July 2025, Cloudflare — which sits in front of a vast slice of the world's websites — became the first major infrastructure provider to block AI crawlers by default, prompting new customers to decide whether AI may access their content.
This year it went further. On 1 July 2026 Cloudflare introduced controls that sort AI crawlers into three categories — Search (indexing to answer questions), Agent (acting in real time on a person's behalf), and Training (absorbing content into models) — and on 15 September 2026, the defaults change: for new domains, new sites added by existing customers, and all existing free-tier customers, Training and Agent crawlers will be blocked by default on pages that display ads, with mixed-purpose crawlers judged by their most restricted behaviour.
Read that list of who inherits the new defaults again. If you're a small business on Cloudflare's free tier — and an enormous number of small businesses are, often without knowing it, because their web designer set it up — your settings change on 15 September unless you act.
There are good reasons this shift exists: content creators deserve a say (and increasingly, payment) when their work feeds AI models, and Cloudflare is openly building the tools for that bargain. For a publisher whose product is their content, blocking training crawlers is a rational commercial choice. This isn't a villain story.
But it is a defaults story. And defaults have a property that policies don't: nobody chooses them, so nobody knows they're in effect.
Why this matters if you sell anything
Here's the part that matters for ordinary businesses — the therapist, the manufacturer, the agency, anyone whose website exists to be found rather than to be paid for:
A growing share of buyers now ask AI assistants for recommendations. Those assistants can only recommend businesses they can read. And the "Agent" category in the new taxonomy — blocked by default come September on affected pages — includes crawlers acting on behalf of a real person asking a real question, potentially the exact visit that ends in your business being recommended.
For a publisher, blocking that may be worth it. For a service business, it's closing the shop door during opening hours — and the September change means some businesses will do it without ever having decided to.
Our client's hosting company was simply ahead of the curve with a cruder version: a 2010-era template making a 2026 decision on her behalf. The question for you isn't whether your infrastructure might be making this decision for you. It's whether you've ever checked.
The five-minute audit
Three layers to check, cheapest first:
1. What your site says. Open yourwebsite.com/robots.txt in a browser. Under User-agent: *, look for Disallow: / — a site-wide ban on unnamed crawlers, which includes every AI assistant. If your robots.txt is missing entirely, be suspicious rather than relieved: as our client learned, someone else's default may be answering for you.
2. What your server does. On a Mac, open Terminal and run:
curl -A "GPTBot" -o /dev/null -s -w "%{http_code}\n" https://www.yourwebsite.com/200 means AI crawlers get in. 403 means something — a firewall, a bot-management layer, a host default — is turning them away regardless of what your robots.txt says. (The file is the policy; the response is the truth. They can disagree.)
3. What your providers have decided. If your site uses Cloudflare (ask your web designer; many sites do without the owner knowing), log in and look at Security settings → Configure AI bot policies — review the Search / Agent / Training settings and set them deliberately before 15 September, whichever way you choose. And whoever hosts your site: ask them one question — "do you serve a default robots.txt or block any crawlers when my site doesn't specify?" — and keep the answer with your other business records. It's now a vendor-selection question, the same as uptime.
Choose, don't inherit
The honest position isn't "let every AI crawler in" — for some businesses, blocking is right, and the emerging pay-per-crawl economy may reward it. The honest position is that AI access to your website is now a commercial decision with real consequences in both directions, and it's being made somewhere right now — by you, or by a template.
We found our client's answer buried in a host default she'd never seen, making a choice she'd never have made. Five minutes of checking would have found it years earlier. Spend the five minutes.
The Happy Cat is an AI visibility (GEO) agency in Manchester. Checking who — or what — is answering for your website is part of our audit. For the full saga of how we learned all this the hard way, see the four ways robots.txt lied to us.