Cloudflare is blocking your AI crawlers (GPTBot 403s, a robots.txt you didn't write, and the fix)

If you put a site behind Cloudflare recently and AI crawlers are getting HTTP 403 everywhere, that is not a misconfiguration you caused. Since July 2025, Cloudflare blocks AI crawlers on new zones by default, and it may also prepend a managed section to your robots.txt that tells those same crawlers to go away, even if your own rules say the opposite. We know because it happened to specmora.com this month, and we sell AI visibility work for a living. This post is the incident, the honest case for why Cloudflare’s default exists, the testing mistake that makes the problem look fixed when it isn’t, and the exact steps to open the door back up.

What we found on our own site

We moved specmora.com onto Cloudflare in July 2026 — ordinary DNS migration, nothing exotic. Shortly after, we ran a routine crawler-access check, the same one we run for clients. Every AI crawler we tested — ClaudeBot, GPTBot, ChatGPT-User, PerplexityBot, CCBot — got an HTTP 403 with a “Your request was blocked” page. Not on some URLs. On every URL, including /llms.txt (a file that exists purely for AI systems) and the XML sitemap. Meanwhile Googlebot passed, Bingbot passed, and every browser passed.

That combination is the signature worth remembering: humans fine, classic search fine, AI crawlers walled out. Nothing looks broken. Analytics don’t dip, no one files a support ticket, Search Console stays green. The only place the damage shows up is in a channel most dashboards don’t measure: whether AI answer engines can read you at all. And because the 403 is served at Cloudflare’s edge, the requests never reach your origin, so your own server logs won’t even show the blocked hits. The evidence lives in Cloudflare’s security events and nowhere else.

The irony was not lost on us. Our white-hat GEO service page literally lists “the CDN may be blocking the bots you want” as a standard failure mode we audit for. Then our own CDN did it to us, on day one, by default.

Why Cloudflare does this — and why the default is defensible

This is not a bug, and it deserves a fair reading. On July 1, 2025, Cloudflare changed its default to block AI crawlers on newly onboarded domains unless the owner grants permission, the first infrastructure provider to do so, sitting in front of a large share of the web. The press release frames it as permission-based crawling: new customers are asked whether they want AI crawlers in, and the answer defaults to no.

For most site owners, that default is right. AI training crawlers consume bandwidth and give nothing back; scrapers rebrand your content inside someone else’s answer box; and most businesses get zero customers from being in a training corpus. If that describes you, close this tab. Cloudflare already did the sensible thing on your behalf.

The problem is the silent mismatch. If part of how you win business is being retrieved and cited by ChatGPT, Perplexity, Claude, or Google’s AI features, which is exactly the bet behind GEO and increasingly behind ordinary SEO, then a default designed for content protection is now enforcing the opposite of your strategy, and no one told you. Being blocked at the edge means you are not in the candidate set that AI engines choose their citations from. You can have perfect server-rendered HTML, the thing we wrote about when SPAs hide content from crawlers, and it counts for nothing if the crawler eats a 403 before the first byte of it.

The robots.txt you didn’t write

The 403 was only half of what we found. Cloudflare had also injected a managed section into our robots.txt, prepended above our own rules, on the same URL. Per Cloudflare’s docs, when the managed robots.txt setting is on and your site already serves its own file, Cloudflare prepends its managed block and combines both into one response.

The managed section carries Cloudflare’s Content Signals, a machine-readable line declaring ai-train=no, plus explicit Disallow: / groups for named AI crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended, and others. Our own file, right below it, said Allow. So the combined file contradicted itself: the top half told GPTBot to leave, the bottom half invited it in. Robots.txt parsers do not resolve conflicts uniformly — some merge groups, some take the first match — so a self-contradicting file means your crawl policy is whatever each bot’s parser happens to decide. That is not a policy.

Two layers, two different mechanisms, worth keeping straight. The managed robots.txt is advisory: it asks compliant crawlers to stay out. The 403 is enforcement: a WAF-level block that stops even the crawlers your robots.txt welcomes. Fixing one without the other leaves you either politely inviting bots that get bounced at the door, or bouncing the invitation while the door stands open.

The testing trap: bare-token user agents lie to you

Here is the part that nearly fooled us, and the reason this post has exact commands in it.

Our first quick check spoofed user agents with bare tokens — curl -A "GPTBot" — and got 200s across the board. Conclusion: crawlers fine. Then the full check, using each crawler’s complete user-agent string exactly as the real bot sends it, came back 403 on everything. Same URLs, same minute. The bare token GPTBot passed; the real string containing compatible; GPTBot/1.2; was blocked.

The explanation is mundane once you see it: Cloudflare’s block matches the signatures of the actual crawlers, and a hand-typed bare token doesn’t look like one. But the consequence is nasty — the lazy test produces a false negative that tells you the door is open while every real crawler is being turned away. If you take one operational lesson from this incident: never test crawler access with a bare token. Use the full string, verbatim, from the vendor’s own documentation.

The fix, step by step

Three steps, all in the Cloudflare dashboard, all reversible.

Step 1 — allow the named crawlers. Go to your zone, then AI Crawl Control, then the Crawlers tab. Each known AI crawler has an Actions column; set the ones you want to Allow. We allowed GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, and CCBot. Per the docs, blocks here are enforced as WAF custom rules on your zone, which is exactly why they fire before robots.txt or your origin ever sees the request. Decide bot by bot: you might allow retrieval-and-search bots (the ones that produce citations with links) while leaving training-only crawlers blocked. That split is legitimate — it is the difference between being quoted and being ingested.

Step 2 — turn off the managed robots.txt, or make peace with it. In the dashboard go to Security settings, filter by Bot traffic, and disable “Set your preference to block training in robots.txt” if you want your own robots.txt served untouched. If you genuinely want the ai-train=no signal, you can keep it — but then reconcile your own rules with the per-bot Disallows it prepends, because shipping a file that contradicts itself is worse than either policy alone. We turned it off and kept our own explicit rules.

Step 3 — verify with full user-agent strings. Current strings, from the vendors’ own docs (OpenAI, Perplexity, Common Crawl); Anthropic documents its bot names but not the full string, so the ClaudeBot line below is what the bot actually sends in our access logs. Version segments drift (our logs showed GPTBot/1.2; OpenAI’s docs now list 1.4), so copy the current string from the docs rather than from this post:

# GPTBot (OpenAI, training)
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://www.specmora.com/

# ChatGPT-User (OpenAI, live fetches for ChatGPT users)
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
  https://www.specmora.com/

# ClaudeBot (Anthropic)
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
  https://www.specmora.com/

# PerplexityBot (Perplexity, search index)
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://www.specmora.com/

# CCBot (Common Crawl)
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "CCBot/2.0 (https://commoncrawl.org/faq/)" \
  https://www.specmora.com/

Every line should print 200. Run the same set against your robots.txt and sitemap URLs, since those were blocked too in our case. Then check that robots.txt is yours again:

curl -s https://www.specmora.com/robots.txt

If a Content Signals block you didn’t write still sits above your rules, step 2 didn’t take. After our fix, all five crawlers returned 200 on every URL we tested, and robots.txt was back to the file in our repo.

Decide on purpose, not by default

The uncomfortable truth in this incident is that both sides are behaving reasonably. Cloudflare is protecting the median customer from uncompensated scraping; the median customer benefits. But defaults are policies applied to people who didn’t choose them, and if AI answers are a channel you sell through, this one is a policy against your own interests, applied silently, invisible in every dashboard you normally watch.

So make the decision deliberately. If AI visibility means nothing to your business, keep the block and take the free protection. If it does mean something, allow the named crawlers, own your robots.txt, and verify with real user-agent strings, then remember that an open door is only the plumbing. What earns the citation once the crawler gets in is the same extractable, authoritative, answer-first content it always was. Either answer is fine. Not knowing which one your CDN gave on your behalf is the only wrong state.

Sources

1. Content Independence Day: no AI crawl without compensation — Cloudflare’s July 1, 2025 announcement that new domains default to blocking AI crawlers (Cloudflare, 2025).
2. Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large — press release describing permission-based crawling as the new default (Cloudflare, 2025).
3. AI Crawl Control — Cloudflare docs: monitor AI crawler traffic and set allow or block rules per crawler (Cloudflare docs).
4. Manage AI crawlers — the Crawlers tab, Allow/Block actions, and the WAF custom rules that enforce blocks (Cloudflare docs).
5. Managed robots.txt — how Cloudflare prepends a managed section (Content Signals plus per-bot Disallow rules) above an existing robots.txt, and how to disable it (Cloudflare docs).
6. OpenAI crawler documentation — official full user-agent strings for GPTBot, ChatGPT-User, and OAI-SearchBot (OpenAI docs).
7. Perplexity crawler documentation — official full user-agent strings for PerplexityBot and Perplexity-User (Perplexity docs).
8. Does Anthropic crawl data from the web? — Anthropic’s documentation of ClaudeBot, Claude-User, and Claude-SearchBot and their robots.txt handling (Anthropic help center).
9. CCBot — Common Crawl’s documented user-agent string (Common Crawl docs).