The file that decides whether AI knows you exist (and the four ways it lied to me in three weeks)

Every website can serve a little file at yourwebsite.com/robots.txt. It's the sign on the door: which visiting robots — Google's crawler, Bing's, ChatGPT's, Claude's — are welcome, and which rooms they're allowed in. It was invented in 1994 and it is, to be clear, just a sign. Nothing enforces it. Which turns out to be the whole plot.

Because in 2026, the robots reading that sign decide something new: whether AI assistants can read your site at all — and a business that AI can't read is a business AI will never recommend. Your robots.txt quietly became one of the most commercially important files you own. Mine was telling everyone to go away. I checked yours might be too.

Here are the four ways this innocent file misbehaved for me in twenty-one days.

My own site launched blocking AI crawlers. I didn't choose that. I didn't choose anything — which was the problem. The platform default did the choosing, and the default said no.

The fix took an afternoon: an explicit robots.txt welcoming the AI crawlers by name, verified with a one-line terminal command. My site went from "refused connection" to fully readable in a day. Lesson one: if you've never looked at your robots.txt, you don't know whose policy you're enforcing.

A client's site had a robots.txt that looked perfectly normal in a browser. Underneath, it was corrupted — invisible characters (non-breaking spaces, the poltergeists of copy-paste) had mangled the directives so crawlers couldn't parse them. A file can pass every human eyeball test and still be gibberish to a robot.

Lesson two: never paste a robots.txt from a document. Type it. And verify with the terminal, not the browser, because the browser shows you what you expect and the terminal shows you what's served.

This one's my favourite. I paid a UK newswire to distribute a press release — a release about AI visibility, pleasingly enough. Their robots.txt: crawlers welcome. The actual server, when an AI crawler knocked: 403 Forbidden. Same page, same second, a browser got a cheery 200.

The sign said welcome. The bouncer had other instructions. The block lived in a firewall layer above the robots.txt entirely, in direct contradiction of the published policy — and nothing on the sales page mentions that the product's pages are unreadable to the systems doing modern citing. (Full test, receipts included, in the write-up.)

Lesson three: the robots.txt is the policy; the HTTP response is the truth. They can disagree. You have to test both.

Back to the client's site, where the deepest twist waited. We deleted the old corrupted file, expecting WordPress to serve its friendly built-in version. Instead, a new robots.txt appeared — one nobody had written. Google and Bing: welcome. Everyone else: banned from the entire site. Plus a crawl-delay so hostile it was basically a "go away" mat.

The hosting company, it turned out, serves its own default robots.txt whenever no real file exists — a whitelist template apparently unchanged since roughly 2010, from before "AI crawler" was a phrase anyone needed. The site had likely been invisible to anything newer than Bing for years, and no one could have known, because the file wasn't anywhere a site owner would ever look. It wasn't on the site. It was the host, wearing the site's robots.txt as a costume.

One deliberately created physical file later, both versions of the domain serve the same clean, welcoming policy, and an AI crawler got its first 200 from that site possibly ever.

Lesson four: an absent robots.txt isn't neutral. Someone will answer on your behalf, and you won't like their answer.

Two honourable mentions. First, caches: after every single fix above, some system somewhere kept showing the old blocked version — browsers, AI fetchers, page caches, all confidently serving yesterday. Fixes propagate in days, not seconds; if you test immediately and see the old behaviour, you may be looking at a ghost.

Second, and humbling: at one point my own verification command was written wrong — it searched for a formatting pattern the file didn't use, and cheerfully reported the schema missing when it was there all along. The measuring stick can lie too. Test your tests.

Open the terminal on any Mac (Cmd+Space, type Terminal) and run these two, with your own domain:

curl https://www.yourdomain.com/robots.txt

curl -A "GPTBot" -o /dev/null -s -w "%{http_code}\n" https://www.yourdomain.com/

The first shows the sign on your door — read it and check nothing under User-agent: * says Disallow: /. The second knocks as an AI crawler and prints one number: 200 means welcome; 403 means somewhere, a bouncer you may not know you employ is turning AI away. Run the second one again swapping "GPTBot" for "ClaudeBot" and "PerplexityBot" for the full picture.

Two commands. If either surprises you, you've just learned something about your business that your analytics will never show you: whether the fastest-growing referrer of the next decade can see you at all.

And if the surprise is unwelcome and you'd like company while fixing it — this is, at this point, extremely my specialist subject.