Audit what your site lets AI clients fetch

01

What robots.txt specifies

Applies to marketplace onlyown storefronthybrid

The Robots Exclusion Protocol is published as RFC 9309, a Standards Track document specifying robots.txt syntax, crawler caching expectations, parsing limits, and how an unreachable file is handled.¹ The semantics are written down rather than conventional.

Two properties of the document carry further than its syntax. It specifies conformant crawler behavior, so a client that ignores it is non-conformant rather than in breach of anything the document enforces. And it states that the protocol is not a substitute for valid content security measures.¹ A path whose exposure matters is behind authentication rather than behind a disallow line.

02

The fetches robots.txt does not govern

Applies to marketplace onlyown storefronthybrid

The operators document this themselves. OpenAI’s crawler documentation states that robots.txt rules may not apply to user-initiated ChatGPT-User fetches.² Google publishes a separate category for user-triggered fetchers, invoked on a person’s request, and states that these generally ignore robots.txt rules.³

Both describe the same distinction. A client crawling on its own schedule and a fetch made because a person asked a question about a specific page are treated differently, with the second handled closer to a browser request. The consequence for an audit is that reading robots.txt covers scheduled crawling and leaves this category unaccounted for.

So a robots.txt covering every named AI crawler is a rule set covering those named crawlers, with a documented class of fetches outside its scope.

03

Rules that are not in the file

Applies to marketplace onlyown storefronthybrid

Access decisions are made at several layers, and one of them is a text file readable in a browser. The rest are settings screens.

• Host or CDN bot rules, which can act on categories rather than on named clients.
• Per-crawler allow and block rules, which some infrastructure exposes alongside crawler-activity visibility.⁴
• Rate limiting, which names no client and produces the same result under load.
• Geographic or ASN restrictions added for fraud reasons and not revisited since.
• Platform-level toggles on hosted storefronts, sometimes labeled as an AI or scraping setting.

Verified-bot machinery sits alongside these. Infrastructure maintaining verification criteria for bots and agents supplies a category that rules can be written against, and the same documentation states that verification places no obligation on a site owner to admit the client.⁵ The list is an input to a policy rather than the policy.

04

Running the inventory

Applies to marketplace onlyown storefronthybrid

The exercise takes an afternoon and requires no tooling. The output is an inventory of what is currently live, produced before any decision about whether to keep it.

1. Fetch the site’s own robots.txt and read every line, including lines nobody in the business wrote.
2. For each disallow, name the client it targets and the reason it is there. A rule with no available reason is recorded as such.
3. Open the CDN or host bot settings and capture the current state before changing anything.
4. Check whether a category-level rule catches clients the seller would admit.
5. Confirm that paths whose exposure matters are behind authentication rather than a disallow line.
6. Record the result with a date, so the next run has something to compare against.

The second row is a rule with no recorded author covering the pages that carry the one-of-one detail. The third stays Unknown until someone with older account access can answer, and recording it as Unknown assigns it to a person rather than leaving it as an unexamined toggle.

05

Classifying what the inventory returns

Applies to marketplace onlyown storefronthybrid

Findings sort into three groups. A rule with a current reason stays. A rule whose reason has ended — an incident two years ago, a theme default — is removed one at a time with a date recorded. A rule that cannot be classified is assigned an owner and a date rather than a guess.

Opening everything is a separate decision from removing a rule nobody set. The cited sources do not state that admitting more clients produces more sales, and unrestricted fetching of a small storefront has costs that are observable directly: bandwidth, rate-limit exhaustion during a sale, and bulk copying of photography and copy.

For a reseller, the product pages are usually the only public record of one-of-one stock and are the pages a shopping surface would read. Where restriction is applied, bulk paths — search result pages, exports, partner feeds — are a different category from the item pages.

06

Re-running it

Applies to marketplace onlyown storefronthybrid

Access configuration changes without producing a visible failure. A plugin update adds a rule, a host changes a default, a limit tightened during an incident stays tightened. A dated inventory is what makes the change visible on the next pass.

The audit is re-run on a fixed schedule and after three events: a platform migration, a security incident, or an operator publishing new crawler documentation. The third is the one specific to this area, since the operator documentation cited here is described as updated over time, and a rule verified last quarter was verified against the text as it then stood.

07

The access rules and where they are documented

Applies to marketplace onlyown storefronthybrid

RFC 9309 specifies robots.txt and states that it is not a substitute for content security measures.¹ OpenAI documents user-initiated fetches as possibly outside those rules,² and Google documents user-triggered fetchers as generally ignoring them.³ Cloudflare publishes verification criteria for bots and agents⁵ and separate per-crawler controls with crawler-activity visibility.⁴

Between them these cover what a conformant client does, which fetches fall outside that, and where a site owner’s controls sit. None of them states what a given storefront currently allows, which is what the inventory records.