Let me walk through how I audit robots.txt and AI crawler access for an ecommerce store, because the gaps I find are almost never where the owner expects. Most people look at their robots.txt file, see it looks reasonable, and assume AI crawlers can reach their products. That assumption breaks in practice more often than not. A CDN bot-management rule, an overly broad Disallow entry, or a WAF pattern can quietly block GPTBot or ClaudeBot while the site serves human visitors without a problem.

The process behind how I audit robots.txt and AI crawler access for an ecommerce store has four distinct layers. First, I clarify what the store owner actually intends to allow or block, because intent and configuration often diverge. Second, I fetch the live robots.txt from every relevant hostname. Third, I send test requests using actual AI crawler user-agent tokens and record real HTTP responses. Fourth, I pull server logs to see which crawlers have been arriving, how often, and what responses they received. All four layers together tell a complete story that no single layer can tell on its own.

What Most Stores Get Wrong Before the Audit Starts

The most common mistake is treating robots.txt as the whole picture of crawler access. It’s a guidance file, not network enforcement. A CDN rule, a WAF policy, or a Cloudflare Bot Management setting operates independently of robots.txt and can block a crawler entirely with no trace in the file itself. I’ve audited stores where GPTBot was cleanly allowed in robots.txt but blocked at the network edge for months, with no error alerts surfacing in any dashboard that anyone checked regularly.

The second widespread mistake is writing rules for Googlebot and assuming other crawlers fall into line. Many stores carry robots.txt files written before AI crawlers existed, with a User-agent: * wildcard and broad Disallow lines but no specific rules for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent, or Bytespider. According to Google Search Central’s robots.txt documentation, crawlers without a matching user-agent group default to the wildcard group. A broad Disallow: / under that wildcard blocks any crawler without its own dedicated entry.

Before touching any files, I write down what the store owner actually wants: which crawlers should have open access to product and category pages, and which paths should always be closed regardless of crawler, including cart, checkout, account pages, internal search results, session-parameterized URLs, and admin endpoints. Getting this documented first means every finding I report can be compared against a stated intent rather than judgment calls made mid-audit.

Fetching the Live File Across Every Hostname

I fetch robots.txt with curl on every hostname separately, because www and non-www can carry different policies when redirects aren’t configured cleanly. On stores with incomplete redirect setups, those two versions can serve entirely different rules, and a browser hides that by following the redirect silently. I check every in-scope hostname: www, non-www, regional or country-code domains, any subdomain serving public store pages, and staging environments that are publicly reachable.

Fetching a robots.txt file directly from each hostname:

curl -L https://www.example.com/robots.txt
curl -L https://example.com/robots.txt
curl -L https://staging.example.com/robots.txt

With the live file in hand, I read through every User-agent group and note all Disallow and Allow rules. I look for groups covering Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, DuckDuckBot, meta-externalagent, and Bytespider. A broad Disallow: / with a handful of specific Allow paths is a common pattern that can accidentally block crawlers when the Allow rules aren’t precise enough or rely on path-precedence logic the author didn’t intend. I validate the syntax with a robots.txt validator and confirm the Sitemap directive points to a reachable, current XML sitemap.

The URL Matrix I Test on Every Store

I test at least seven URL types every time: homepage, a category page, a product detail page, an editorial page, a paginated or faceted navigation result, the XML sitemap, and a path that should be blocked such as /cart or /account. Testing this range surfaces problems that affect specific page types while leaving others untouched, which a single-URL test would never catch.

For each URL, I compare two requests. The first uses a standard desktop browser user-agent to establish a baseline: status code, response size, canonical tag, and whether product information is visible in the HTML. The second uses the AI crawler user-agent token under test. I record status code, response size, any redirects, and whether the response returns the same content. Differences between the two often point to server-side user-agent detection or CDN rules targeting specific bot strings.

Sending crawler-specific user-agent requests to a product URL:

curl -A "GPTBot/1.0" -L -I https://www.example.com/products/example-product
curl -A "ClaudeBot/1.0" -L -I https://www.example.com/products/example-product
curl -A "PerplexityBot/1.0" -L -I https://www.example.com/products/example-product

I also check whether product pages return meaningful content without JavaScript execution. Many AI crawlers don’t run JavaScript. If a product page loads name, price, availability, and description via client-side scripts and returns a near-empty HTML shell to a non-rendering request, a crawler sees essentially nothing useful. That matters directly for AI shopping features that depend on crawled content rather than submitted feeds.

What Server Logs and Network Controls Reveal

Server logs show which crawlers actually reached your store and what responses they got. Network controls are where silent blocks most often hide, independent of anything in robots.txt.

I pull logs for at minimum the past 30 days and filter for known AI crawler user-agent strings: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, and Amazonbot are the main ones. Before treating any entry as genuine, I check the source IP against each operator’s published IP ranges and run forward-confirmed reverse DNS. OpenAI publishes its GPTBot IP ranges on its GPTBot documentation page, and Anthropic provides ClaudeBot’s IP ranges in its crawler documentation. Impersonating a known crawler user-agent is common among scrapers, so raw log volume means nothing until identity is verified.

With verified entries, I look at response code distributions by crawler. A healthy access pattern shows a majority of 200 responses across product and category URLs. A high share of 403 responses indicates a block at the server or network layer. A significant 429 rate points to rate-limiting throttling legitimate crawlers too aggressively. A 404-heavy pattern usually means the crawler is navigating from a stale sitemap or following dead internal links that need cleaning up.

I also look at which URLs each crawler is requesting and how deeply it’s navigating into the store. A crawler that has visited product detail pages recently and returned 200 responses is in a different position from one that only ever hits the homepage before stopping. The URL distribution across verified entries tells me whether a crawler is reading the store or bouncing off a barrier at the surface level.

Cloudflare Bot Management, WAF custom rules, and firewall settings all operate independently of robots.txt. A store using Cloudflare with an aggressive bot-score threshold may be blocking GPTBot, ClaudeBot, and PerplexityBot with no robots.txt rule in sight. I check the Cloudflare dashboard under Security > Bots for managed challenge or block actions applied to AI crawler user-agents, and I review WAF custom rules for string patterns that might match crawler names. This layer most often explains the situation where a store owner says “but I allowed it in robots.txt” while server logs show a steady stream of 403s.

Schema, Rendering, and How I Document Every Run

For AI crawlers that don’t run JavaScript, server-side rendered HTML and complete Product JSON-LD are the technical floor. Without them, a crawler sees an empty shell, regardless of what robots.txt permits.

I check Product schema for the fields AI shopping features rely on most: name, offers (with price, priceCurrency, and availability), description, and image. A schema validator confirms the markup is present and well-formed, but I also curl the page directly and search the raw HTML response for the JSON-LD block to confirm it is rendered server-side rather than injected only after JavaScript executes. A product page that passes a browser-based schema test but returns no JSON-LD to a non-rendering request is invisible to any AI crawler that cannot run scripts, regardless of what robots.txt permits.

Documentation is what separates a one-time check from a repeatable audit process. For every run I record the date, the store’s hostname list, which layers I tested, and findings per layer: robots.txt status for each crawler group, HTTP response outcomes for each URL type and user-agent pair, server log highlights with counts of verified and unverified entries, and any network-layer rules that affect AI crawler access. I note recommended changes and assign a next-review date. That record becomes the comparison baseline for the following quarter. Without it there is no way to determine whether a block that was found and cleared stayed cleared, or whether a CDN configuration change quietly reintroduced it in the months between audits.

Running through all four layers (intent, live file, HTTP responses, and server log evidence) gives a complete picture that none of them provides alone. The most common finding is a divergence between what robots.txt declares and what the network actually delivers, a gap that remains invisible until you test it directly. A store that completes this audit and aligns all four layers has removed the technical barriers standing between AI crawlers and its product catalog. What follows from that depends on content quality, schema completeness, and authority signals, but none of that matters if the crawlers cannot get through to begin with.

Frequently Asked Questions

What is the most important thing to check when auditing AI crawler access?

Start with live HTTP response testing rather than reading the robots.txt file. CDN rules and WAF policies can block every AI crawler request at the network edge while robots.txt shows open access, so a 200 response on a real product URL using the actual crawler user-agent token is the only reliable confirmation that access genuinely works end to end.

How do I know if GPTBot is actually blocked on my store?

Filter your server access logs for the GPTBot user-agent string, then validate the originating IP addresses against OpenAI’s published IP ranges using forward-confirmed reverse DNS. Verified log entries showing 200 status codes on product pages confirm access is working; a consistent pattern of 403 responses or a complete absence of verified entries points to a block somewhere in your infrastructure stack.

Does allowing AI crawlers guarantee my products appear in AI answers?

Allowing access is the technical entry point, not a visibility guarantee. Structured data completeness, content quality, authority signals, and product information freshness all shape whether an AI assistant treats your pages as worth citing. Open access removes a technical barrier; it does not by itself produce recommendations or citations in AI-generated results.

Which paths should always be blocked in robots.txt for ecommerce?

Cart, checkout, user account pages, internal search results, session and tracking parameter URLs, and admin paths should be closed to all crawlers. These URLs add nothing of value for AI assistants and risk generating duplicate content signals or surfacing transactional data that has no legitimate reason to be crawled or indexed by any crawler.

How often should I run this robots.txt and AI crawler audit?

A quarterly review is a sensible minimum for most stores, with an additional check after any change to CDN configuration, security plugin settings, server infrastructure, or robots.txt itself. Network-layer changes often happen without the content team’s knowledge, so a scheduled recheck catches quietly introduced blocks before they affect crawler access for an extended period.