Free AI Crawler Access Checker
See which AI assistants your website lets in, and get the robots.txt rules for the policy you want
Get the robots.txt rules for the policy you want
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Amazonbot Allow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: /
Add them to the robots.txt file at the top level of your site. Well-behaved crawlers follow these rules; they are a request, not a lock.
What this checker looks at
+
AI assistants read websites with their own crawlers, and a site's robots.txt file can allow or block each one. Many sites block them without knowing, through a site builder's default or a security plugin. This tool reads your robots.txt and shows where each crawler stands.
Training crawlers and answer crawlers are different
Some crawlers collect text to train models. Others fetch a page at the moment someone asks a question, and cite it. You can block the first kind and still be recommended by assistants, which is the policy many publishers choose.
Blocked by name or by a general rule
A crawler can be blocked by a rule written for it, or caught by a rule for all crawlers. The checker says which, because the fix is different.
What counts as blocked
A crawler counts as blocked when it is disallowed from the whole site. Rules that only cover one section, such as a login area, are not counted.
Google-Extended is not Google Search
Blocking Google-Extended keeps your pages out of Gemini's training and grounding. It has no effect on how you appear in ordinary Google results.
robots.txt is a request
Reputable crawlers follow it. It does not stop a crawler that ignores the rules; that takes a firewall.
Is blocking common?
Yes. Our live study of the world's most popular websites counts how many block at least one AI crawler; the current figure is on the study page.
The list covers the crawlers that OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Common Crawl and ByteDance have published. New ones appear regularly, and nobody outside those companies can guarantee whether a site is cited in AI answers.
See the live study: which top websites block AI assistants
This page in other languages: Español · Português · Français · Deutsch