Home › Blog › Cloudflare for Shopify: AI shopping agents

Cloudflare for Shopify: the three settings that silently block AI shopping agents

Your robots.txt says Allow: / for GPTBot. Your Shopify theme hasn’t been touched in months. And yet CatalogScan shows your robots-open signal failing at 0 points. The culprit is almost certainly Cloudflare — and it’s almost certainly one of three specific settings. This guide shows you which one, how to find it in the Cloudflare dashboard, and how to fix it without making your store an open target for real scrapers.

Published 2026-05-30 · ~14 min read · By the CatalogScan team

Step 0 — confirm Cloudflare is the problem

Before touching any Cloudflare setting, run this curl command from your terminal. It impersonates GPTBot, OpenAI’s combined training-and-shopping crawler, and fetches your robots.txt:

curl -si \
  -A "GPTBot/1.0 (+https://openai.com/gptbot)" \
  https://yourstore.com/robots.txt | head -30

Replace yourstore.com with your actual domain. Look at the first 30 lines of the response:

• If you see User-agent: and Disallow: rules — robots.txt is returning normally. Cloudflare is not blocking the crawler. The issue is in the robots.txt content itself. See the robots-open signal guide for content-level fixes.
• If you see HTTP/2 403 or a redirect to a challenge URL — Cloudflare is blocking GPTBot before it reaches your server. You’re in the right place; continue with the three settings below.
• If you see HTML with “Just a moment” or “Checking your browser” — Bot Fight Mode is serving a JavaScript challenge. The crawler can’t pass it; it sees your store as closed. This is Setting 1.
• If the command times out or returns no output — could be a rate limit, a firewall rule that drops connections silently, or your origin being down. Add -v and try a second time.

Run the same test with the other major AI shopping crawlers to understand the full scope of the problem:

# OAI-SearchBot: the ChatGPT shopping retrieval crawler (separate from GPTBot)
curl -si -A "OAI-SearchBot/1.0 (+https://openai.com/searchbot)" \
  https://yourstore.com/robots.txt | head -5

# PerplexityBot: Perplexity Shopping
curl -si -A "PerplexityBot/1.0 (+https://docs.perplexity.ai/guides/bots)" \
  https://yourstore.com/robots.txt | head -5

# Google-Extended: Google AI Mode and AI Overviews
curl -si -A "Googlebot/2.1 (+http://www.google.com/bot.html)" \
  -H "X-Forwarded-For: 66.249.66.1" \
  https://yourstore.com/robots.txt | head -5

# ClaudeBot: Anthropic AI training and Claude features
curl -si -A "ClaudeBot/0.5 (+https://www.anthropic.com/claude-web-crawler)" \
  https://yourstore.com/robots.txt | head -5

It’s common for one crawler to get through while another is blocked. GPTBot and OAI-SearchBot come from different IP ranges and are classified differently by Cloudflare’s bot intelligence, so a rule that lets GPTBot through can still block OAI-SearchBot. Test each one.

Why Cloudflare can pass your robots.txt text and still fail your store

Cloudflare operates at the edge, between the internet and your Shopify store. It sees every request before Shopify does. If Cloudflare decides a request looks bot-like, it can intercept it and return a challenge page (HTTP 200 with JavaScript) or a hard block (HTTP 403) — without ever asking your Shopify origin what to do.

This creates a gap that trips up most Shopify operators who check their robots.txt: they read the file in their browser, it looks fine, they assume all crawlers can see it. But AI crawlers are fetching via curl-like HTTP clients, not a browser, and Cloudflare’s bot detection looks at the user agent, TLS fingerprint, and connection behaviour to classify the request. A GPTBot fetch looks very different from a Chrome browser fetch, and Cloudflare classifies it accordingly.

The four AI crawlers that matter for shopping surfaces right now, and what they do with your catalog:

If any of these return anything other than your actual robots.txt content, they can’t proceed to index your catalog, read your JSON-LD, or include your products in AI-generated shopping results. The robots-open signal is the floor signal: at 0 points, every other signal becomes irrelevant because the crawler never reaches your product pages.

Setting 1 — Bot Fight Mode

How to diagnose it

The curl test is your fastest check: if you see <!DOCTYPE html> or “Just a moment” in the response to a GPTBot user agent curl, Bot Fight Mode is the culprit. Alternatively, in the Cloudflare dashboard, navigate to Security → Bots. You’ll see a toggle labelled “Bot Fight Mode.” If it’s on and you don’t have a Skip rule for AI crawlers, every AI shopping crawler is being challenged.

Fix option A: disable Bot Fight Mode entirely

If your store sells to end consumers (not a B2B API with sensitive data), disabling Bot Fight Mode is safe and simple. The toggle is a blunt instrument for many legitimate use cases — it blocks Googlebot on misconfigured zones, price-comparison crawlers that drive legitimate traffic, and AI shopping crawlers that represent growing retail revenue. Turn it off, run the curl test again, and verify you get robots.txt content back.

Fix option B: create a Skip rule for AI crawlers (recommended if you must keep Bot Fight Mode)

If your store has a real DDoS problem or you sell high-value inventory that attracts scrapers, you may want to keep Bot Fight Mode for genuine bad actors. Cloudflare lets you create “Skip” WAF rules that bypass Bot Fight Mode for specific conditions. Navigate to Security → WAF → Custom Rules and create a rule with:

Action: Skip
Skip: All Bot Fight Mode features
When: (http.user_agent contains "GPTBot") or
      (http.user_agent contains "OAI-SearchBot") or
      (http.user_agent contains "PerplexityBot") or
      (http.user_agent contains "ClaudeBot") or
      (http.user_agent contains "Google-Extended") or
      (http.user_agent contains "Applebot-Extended")

Place this rule above any block rules in the Custom Rules list. Order matters: Cloudflare evaluates rules top-to-bottom and stops at the first match. After saving, run the curl test — you should now get the robots.txt content back for each of these agents.

Super Bot Fight Mode (Pro plan and above)

Pro, Business, and Enterprise zones have a more granular version called Super Bot Fight Mode. It classifies bots into “Definitely automated,” “Likely automated,” and “Verified bots.” AI shopping crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) are generally classified as “Verified bots” by Cloudflare since they include their documentation URL in the user agent string. If your Super Bot Fight Mode action for “Verified bots” is “Allow,” you’re fine. If it’s “Challenge” or “Block,” that’s the issue — change it to “Allow.”

Setting 2 — AI Scrapers and Crawlers managed rule

How to diagnose it

Navigate to Security → WAF → Managed Rules in your Cloudflare dashboard. Look for a rule with “AI Scrapers” or “AI Crawlers” in the name. If its action is “Block” or “Challenge,” it’s the problem. You can also check Security → Events (formerly Firewall Events) and filter by the rule ID — if you see GPTBot requests being blocked with a managed rule match, that’s it.

Fix: disable the managed rule

Click the rule and set its action to “Disabled” or delete it. If you want to keep blocking AI training crawlers (CCBot, the Common Crawl crawler used by many LLM training pipelines) without blocking AI shopping crawlers, Cloudflare doesn’t currently offer a single toggle that makes that distinction. Your options are:

• Disable the managed rule entirely and accept that all AI crawlers can fetch your store. This is the correct setting for a Shopify store that wants AI shopping visibility.
• Disable the managed rule and add a custom block rule targeting only CCBot specifically: http.user_agent contains "CCBot". CCBot is the Common Crawl crawler; blocking it opts you out of Common Crawl-based LLM training without affecting shopping crawlers.

You cannot use a robots.txt Disallow rule to block CCBot while keeping the managed rule on. The managed rule acts at the network layer before robots.txt is ever served, and it doesn’t distinguish between training crawlers and shopping retrieval crawlers.

The “training vs. shopping” decision is now yours to make deliberately

The managed rule was created when the only use case for GPTBot was training ChatGPT. That’s no longer true. GPTBot now also drives ChatGPT Shopping product indexing. Blocking GPTBot opts you out of both. Most Shopify operators who enabled this rule in 2023 or 2024 made a deliberate decision about LLM training but an accidental decision about AI shopping revenue. Review it now with both use cases in mind.

Setting 3 — custom WAF rules using cf.client.bot

How to diagnose it

Navigate to Security → WAF → Custom Rules. Read through your active rules. Any rule with cf.client.bot in the expression and a “Block,” “Challenge,” or “Managed Challenge” action will block AI shopping crawlers. Also check for rules using cf.bot_management.score with a threshold you don’t control (bot management score is a 0–100 scale; a threshold of 30 blocks everything AI-crawler-shaped).

Also look for rules that are “allow except Googlebot” shaped — a common pattern from 2022–2023 security tutorials:

# This rule blocks all bots except Googlebot — including every AI shopping crawler
(cf.client.bot and not http.user_agent contains "Googlebot")

If you find a rule like this, you’re allowing Google’s traditional web crawler (which was the SEO gold standard in 2022) but blocking every AI shopping crawler that didn’t exist when the rule was written.

Fix: rewrite the rule to allow AI shopping crawlers explicitly

Two options depending on whether you want to keep a bot-blocking rule at all:

Option A — delete the cf.client.bot rule entirely and rely on Bot Fight Mode (with a Skip rule for verified bots) for bot mitigation. This is cleaner and lets Cloudflare’s own bot intelligence do the work instead of a blanket boolean.

Option B — add AI shopping crawlers to the allowlist in your existing rule:

# Before (blocks all bots Cloudflare identifies, including GPTBot):
(cf.client.bot)

# After (blocks unverified bots, allows known AI shopping crawlers):
(cf.client.bot and
  not (http.user_agent contains "GPTBot") and
  not (http.user_agent contains "OAI-SearchBot") and
  not (http.user_agent contains "PerplexityBot") and
  not (http.user_agent contains "ClaudeBot") and
  not (http.user_agent contains "Google-Extended") and
  not (http.user_agent contains "Applebot-Extended"))

Be aware that cf.client.bot checks a Cloudflare-maintained list of known bots. Newly launched AI shopping crawlers (and new bot types from existing providers) won’t be in the list immediately, so a rule built on cf.client.bot is inherently brittle as the AI shopping crawler landscape evolves. Adding explicit user-agent allowlists for the current set of crawlers is a better long-term approach than relying on Cloudflare’s internal classification alone.

The cf.bot_management.score variant

If you have Cloudflare Bot Management (an Enterprise add-on), you may have rules using the cf.bot_management.score field. This is a 0–100 score where lower scores indicate more bot-like traffic. AI shopping crawlers typically score between 1 and 20. A rule like (cf.bot_management.score lt 30 and not cf.bot_management.verified_bot) should pass known verified bots (GPTBot, Googlebot, ClaudeBot are all in Cloudflare’s verified bot list) but block unverified low-score traffic. That’s a reasonable setup. A rule like (cf.bot_management.score lt 50) with no verified-bot exception will block most AI crawlers along with the bad actors you actually want to stop.

The safe Cloudflare config for a Shopify store that wants AI shopping visibility

Here’s the minimum Cloudflare setup that keeps your store protected against real threats while remaining fully open to AI shopping crawlers:

There’s one more layer that’s easy to miss: IP reputation challenges. Cloudflare can issue Managed Challenges (previously “non-interactive CAPTCHAs”) for requests from IP addresses with poor reputation scores. AI crawlers that come from data-center IP space (all of them do) sometimes have elevated risk scores simply because data-center IPs are overrepresented in bot traffic. If you’re seeing Managed Challenge responses for AI crawler user agents rather than hard 403s, this is the likely cause. Navigate to Security → WAF → Tools → IP Access Rules to review active challenge rules for data-center IP ranges.

5 mistakes that make this harder than it needs to be

Verification playbook

Work through these steps in order after making changes. Each step takes <2 minutes. Don’t skip ahead — a failure at Step 1 makes all subsequent steps meaningless.

If you pass Steps 1–4 (all four curl commands return the robots.txt body) but CatalogScan’s robots-open signal is still failing, the issue moved from Cloudflare to the robots.txt content itself. See the robots-open signal guide for content-level diagnosis: blanket Disallow rules, targeted AI-bot blocks added during a “no AI training” stance, and headless storefronts that never emit a robots.txt at the canonical path.

One edge case worth knowing: CatalogScan’s fetch comes from a data-center IP range. If you have a Cloudflare IP Access Rule that challenges or blocks data-center IPs (a common anti-scraper measure), the scan fetch will be challenged even when real AI crawlers get through. In that case, the curl test will pass but the CatalogScan scan will show the signal failing. Both are real conditions, and both need to be fixed: the real AI crawlers come from data-center IPs too.