Shopify robots.txt for AI shopping agents

There is exactly one signal where a single text-file edit by a teammate at 4 PM on a Friday can make your entire store invisible to every AI shopping agent on the planet by Monday morning. That signal is /robots.txt. Get every other signal perfect — flawless Product JSON-LD, complete GTIN coverage, an industry-leading sitemap — and one stray Disallow: / renders all of it irrelevant. Crawlers honour robots.txt before they fetch anything else, so when the file says no, no fetch happens, no signals get scored, no products get surfaced. This is the most-fixable signal at CatalogScan and also the one most likely to fail catastrophically and silently — there is no error page, no Shopify alert, no Slack ping. Your store just stops appearing in answers.

Last updated 2026-04-30 · Floor signal · 15 pts

What it is

Robots Exclusion Protocol (the formal name for robots.txt) is the original, plain-text contract between a website and the crawlers that visit it. The file lives at the root path /robots.txt and consists of one or more rule blocks, each starting with a User-agent: declaration followed by Allow: and Disallow: lines. Crawlers fetch this file before they fetch anything else and obey the rules they match against their user-agent string.

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /orders
Disallow: /account
Disallow: /*?sort_by*

Sitemap: https://store.com/sitemap.xml

User-agent: *
Disallow: /

Shopify's default robots.txt blocks the few paths that should never be crawled — admin, checkout, cart, account — and explicitly allows everything else. The faceted-nav blockers (Disallow: /*?sort_by*) are also good defaults because every ?sort_by permutation is content-equivalent to the unsorted page, and serving 50 sorted variants of the same collection wastes crawl budget. Everything else is open, every other Shopify floor signal becomes scoreable, and every AI agent can read your catalog.

The bad shape on the right is rarer than it used to be (Shopify's password-gate workflow has improved) but we still see it on stores that copied a snippet from a generic SEO blog post that was written for static-site Jekyll deployments, on stores that flipped storefront_password on for "stealth launch" and never flipped it back, and on Hydrogen rebuilds where the developer copied a default Vercel robots.txt generator scaffold without thinking about the catalog impact.

The AI-agent crawler user-agents to know

If your robots.txt singles out any of these for Disallow: / — even with the rest of the file fully open — the corresponding AI shopping surface skips your catalog. Allow them all. A blanket User-agent: * with no targeted blocks is the cleanest posture and what we score for full credit.

Why AI shopping agents care

• Pre-flight check, hard-fail. Every well-behaved AI crawler fetches /robots.txt first. A failed fetch (403, timeout) or a body that disallows the path it wanted means the agent does not fetch — full stop. Your Product JSON-LD is impeccable, your sitemap is perfect, and none of it gets scored, because no crawl ever happens.
• Targeted blocks compound across surfaces. Some operators block GPTBot while leaving everything else open, hoping to opt out of LLM training. The same block also opts you out of ChatGPT Shopping product retrieval — separate use, same crawler. The unintended consequence: Perplexity, Google AI, and Claude still see you, but the largest AI shopping surface does not.
• Trust signal in tied rankings. A clean, well-formed robots.txt that explicitly welcomes the major AI agents reads as "this brand wants to be found by AI shoppers" — a positive prior on tied ranking decisions. A robots.txt with a long list of Disallows feels defensive, even when no rule actively blocks the bot.
• Sitemap discovery anchor. Robots.txt is also where you advertise your sitemap. The Sitemap: directive points crawlers at /sitemap.xml; missing it doesn't kill discovery (most agents check the conventional path anyway) but it's a one-line confidence add.

How to test it on your store

Two curls. From any terminal:

curl -sI https://yourstore.com/robots.txt | head -3
curl -s  https://yourstore.com/robots.txt

Three things to verify:

1. HTTP 200 (or 404). A 200 is the normal case; a 404 means no robots.txt exists, which is fully permissive (every URL is crawlable). A 403 is the bug — usually Cloudflare Bot Fight Mode or a WAF rule intercepting bot traffic. A 301/302 is half-credit on low-reliability bots; emit the file at the canonical hostname.
2. No blanket Disallow. Read the body. Inside the User-agent: * block, there must be no Disallow: /, no Disallow: /products, no Disallow: /products.json, no Disallow: /collections, no Disallow: /pages, no Disallow: /blogs. Disallow: /admin, /checkout, /cart, /account, and /*?sort_by* are all fine.
3. No targeted AI-bot blocks. Search the body for GPTBot, PerplexityBot, Google-Extended, ClaudeBot, Applebot-Extended. If any of them have their own User-agent: rule with Disallow: /, that's a deliberate (or copy-pasted) block. Either remove the rule or change to Allow: /.

The free CatalogScan scan checks all three plus a deeper "what would each major bot get if it tried to fetch /products.json right now" simulation — so a User-agent: * open posture with a hidden Cloudflare WAF challenge gets caught even when the static text reads fine.

How to fix it

{%- comment -%} Welcome AI shopping agents explicitly {%- endcomment -%}
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

5 mistakes we keep finding