HOSTING-SIDE SEO DIAGNOSTICS · 01
Bot Fight Mode is challenging Googlebot — and the free plan has no exceptions
You turned on Bot Fight Mode to stop bad bots. But Cloudflare's free version is all-or-nothing: it can challenge Googlebot too, and there is no way to let Googlebot through with a WAF rule.
👀 1. What clients actually see
- In Search Console, “Test live URL” shows a short challenge page instead of your content
- Indexed pages drop for no apparent reason, even though nothing changed on the site
- “Discovered – currently not indexed” spikes
- Cloudflare → Security → Analytics → Events shows many requests labelled
Bot Fight Mode - Origin access logs show Googlebot IPs receiving 403/503, or a 0-byte response
If you recognise two or more of these, skip the guesswork and run the checks below.
🧨 2. Why the free plan is the risky one (per Cloudflare's docs)
| Design choice | What it means for SEO |
|---|---|
| Issues CPU-intensive challenges | Crawlers that do not run JavaScript can never pass |
| Protects the entire domain, no path scoping | Your articles and product pages are challenged too |
| Cannot be bypassed with WAF custom rules or Page Rules | You cannot allow Googlebot separately |
| JavaScript Detections is on by default and cannot be disabled | An extra JS check that crawlers fail |
In other words, Bot Fight Mode is an all-or-nothing switch. If you want exceptions you need Super Bot Fight Mode (Pro or above), which runs on the Ruleset Engine and supports a Skip action.
🕒 3. Three-minute self-check
Step 1 — see what Google sees
Search Console → URL Inspection → paste your most important page → Test live URL → open the HTML tab. Full content means fine; an empty body or a short script means you are being challenged.
Step 2 — pretend to be Googlebot
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://yourdomain.com/This only tests the user-agent layer, not Google's real IPs — but it is enough to expose a blanket rule.
Step 3 — find the evidence in Cloudflare
Dashboard → Security → Analytics → Events, filter by Service and look for Bot Fight Mode. Then filter the user agent for Googlebot. If both appear together, that is your proof.
Step 4 — check against Google's published IPs
Google publishes its crawler ranges (googlebot.json). If the challenged IP is not on the list it may be an impersonator — blocking it is correct. If it is on the list, you are blocking the real Googlebot.
📉 4. The real cost to SEO and GEO
- Indexing stalls: every Googlebot visit returns a challenge page — a wasted trip; new content appears slowly or not at all
- Crawl budget is burned on challenge pages instead of content
- AI crawlers are wiped out: GPTBot, ClaudeBot and PerplexityBot do not run JavaScript either
- Rankings do not drop overnight — they erode as Google's copy of your site gets older
✅ 5. The right configuration (protection is not the problem)
| Setting | Free Bot Fight Mode | Super Bot Fight Mode (recommended) |
|---|---|---|
| Can you allow exceptions? | ❌ Not at all | ✅ WAF custom rule + Skip |
| Verified bots (Googlebot etc.) | Not configurable | Set to Allow |
| Definitely automated | Challenged globally | Set to Managed Challenge (not Block) |
| Static resource protection | None | Optional (useful against hotlinking) |
| Fixing a false positive | Turn the whole thing off | Add one Skip rule |
In practice: if all you want is to stop scanners and junk bots, WAF rate limiting (60 requests per IP per minute) is enough — and it will not take search engines down with it. That is exactly what we configure for clients.
🛠️ 6. What Timway does (three steps)
- Check — review your Cloudflare configuration, crawl logs and Search Console data to identify which crawlers are blocked
- Configure — disable or downgrade Bot Fight Mode per your plan, then apply WAF rules, rate limiting and crawler allow-lists
- Verify — test with real user agents and IP ranges, confirm Googlebot, GPTBot and PerplexityBot return 200, then send you the report
Think Bot Fight Mode is blocking your Googlebot?
One phone call: we check your Cloudflare settings against your crawl logs and confirm whether search engines and AI crawlers can actually see you.
Timway · just say “Bot Fight Mode”