| Crawler | Purpose | Status | What that means |
|---|
Generate a policy you chose on purpose
Pick a stance, copy the lines into your robots.txt. Each mode is explained below the fold.
Check your llms.txt
llms.txt is a proposed convention: a markdown file at your site root that tells AI systems what your site is and where the important pages are. Paste yours for a structural check, or read the primer below.
Not all AI bots do the same job
Lumping every AI crawler together leads to bad policies. There are four kinds, and blocking each has a different consequence:
- Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) collect content that may end up inside future models. Blocking them is about long-term content rights, and costs you nothing today.
- AI search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) power answer engines that cite and link sources. Blocking them removes your site from an increasingly common way people find things.
- User fetchers (ChatGPT-User, Claude-User, Perplexity-User) load one page because a person asked about it right now. Blocking them mostly frustrates your own potential visitors.
- Control tokens (Google-Extended, Applebot-Extended) are not crawlers at all. They are switches inside robots.txt that tell an existing crawler what the collected pages may be used for. Blocking Google-Extended does not touch your Google Search ranking.
That is why the generator's default mode blocks training while leaving AI search and user fetches alone. It is the stance most small businesses actually want once the difference is explained: keep the traffic, opt out of the training data.
Honest limits of robots.txt
robots.txt is a convention, not an access control. Reputable operators honor it, and there is commercial pressure to keep doing so, but nothing enforces it. Bytespider in particular has been reported by many site operators to ignore it. If a bot will not listen, the next steps are server-side: user-agent and IP filtering, or a WAF rule. This tool tells you what you have declared; your server logs tell you who is complying.
Also worth knowing: a robots.txt group only applies to the exact token it names. A rule for GPTBot says nothing about ChatGPT-User, which is how sites end up blocking training but silently allowing everything else, or the reverse. That mismatch is exactly what the audit table above surfaces.
A two-minute llms.txt primer
llms.txt is the inverse of a blocklist: a small markdown file at /llms.txt that gives AI systems a map of your most useful content. It starts with an H1 naming the site, a one-line blockquote summary, then H2 sections containing annotated links. Adoption is still early and no one is obligated to read it, but it costs ten minutes, and for documentation-heavy sites it is a reasonable bet on where AI-assisted browsing is heading.
Questions people ask
Will blocking AI crawlers hurt my Google ranking?
No, if you block the right tokens. Googlebot handles Search; Google-Extended only controls AI training use. The same separation holds for Applebot and Applebot-Extended. The audit table marks these as control tokens so they are hard to confuse.
Why does the table show bots I have never heard of?
The dataset covers the crawlers that show up in real server logs, with each one's vendor documentation linked where it exists. Two entries have no official docs and are labeled accordingly, so you can decide how much weight to give them.
Can this tool fetch my robots.txt for me?
Browsers cannot read other sites' files (CORS), which is why the page works on pasted content. The repo ships a small CLI that fetches live: npx github:JaydenYoonZK/ai-crawler-audit example.com.
How current is the crawler list?
The dataset carries an update date, shown next to the audit button, and lives in one reviewable JSON file. New bots appear regularly; pull requests with a log sample and documentation link are the fastest way to get one added.