Security and privacyAvailable

AI Crawler Access Checker

Check robots.txt policies for training and search crawlers, product-control tokens and user-triggered AI fetchers with their roles kept distinct.

Protected server-side checkNo permanent storage by default

Request scope

One bounded public HTTP response is fetched. Local, private and reserved networks are blocked.

About this check

What the tool checks

The report applies the published robots rules to current named identifiers and explains when a token is not a conventional crawler.

How this check works

The public robots.txt file is parsed and the most specific matching Allow or Disallow rule is applied to the submitted path for each identifier.
Automated crawlers, product tokens and user-triggered fetchers are presented in separate categories.

What this check cannot establish on its own

Robots.txt is a voluntary signal, not request authentication, access control or a legal guarantee. Some user-triggered fetchers may not use it.
The check does not validate crawler IP ranges, reverse DNS or experimental Web Bot Auth signatures.

Data sources