| Crawler | Purpose | Status | What that means |
|---|
π Generate a policy you chose on purpose
Pick a stance, copy the lines into your robots.txt. Each mode is explained below the fold.
πΊοΈ Check your llms.txt
llms.txt is a proposed convention: a markdown file at your site root that tells AI systems what your site is and where the important pages are. Paste yours for a structural check, or read the primer below.
π€ Not all AI bots do the same job
Lumping every AI crawler together leads to bad policies. There are four kinds, and blocking each has a different consequence:
- Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) collect content that may end up inside future models. Blocking them expresses a training preference, but may also affect downstream services that use shared crawl datasets.
- AI search indexers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) power answer engines that cite and link sources. Blocking them removes your site from an increasingly common way people find things.
- User fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) load pages for live requests. Vendor behavior differs: some honor robots.txt, while others say these user-triggered requests may bypass it.
- Control tokens (Google-Extended, Applebot-Extended) are not crawlers at all. They are switches inside robots.txt that tell an existing crawler what the collected pages may be used for. Blocking Google-Extended does not touch your Google Search ranking.
The generator defaults to blocking training and control tokens while leaving AI search and user fetches alone. It is a starting point, not a universal recommendation. Review the crawler notes and choose the policy that fits your publishing goals.
π§ Honest limits of robots.txt
robots.txt is a convention, not an access control. Reputable operators honor it, and there is commercial pressure to keep doing so, but nothing enforces it. Bytespider in particular has been reported by many site operators to ignore it. If a bot will not listen, the next steps are server-side: user-agent and IP filtering, or a WAF rule. This tool tells you what you have declared; your server logs tell you who is complying.
Also worth knowing: a robots.txt group only applies to the exact token it names. A rule for GPTBot says nothing about ChatGPT-User, which is how sites end up blocking training but silently allowing everything else, or the reverse. That mismatch is exactly what the audit table above surfaces.
β±οΈ A two-minute llms.txt primer
llms.txt is a proposed markdown file at /llms.txt that maps useful site content. The only required section is an H1 naming the site. A summary blockquote and H2 link sections are optional. Adoption is still voluntary, and publishing the file does not guarantee that any crawler or product will use it. This site practices what it preaches and publishes its own llms.txt.
β¨οΈ Use it from your terminal or CI
The same audit runs outside the browser as a zero-dependency command line tool, so you can check any site from a shell or gate a deploy on the crawler policy it declares.
Audit a site
npx github:JaydenYoonZK/ai-crawler-audit example.comFetches the site's live robots.txt and checks all 19 documented AI crawler tokens against it, printing each one's verdict. The site's llms.txt is checked along the way.
Gate a deploy on policy
npx github:JaydenYoonZK/ai-crawler-audit --policy block-training example.comExits nonzero when the live file does not satisfy the chosen policy: block-training, block-all-ai, or allow-all. An unreachable robots.txt also fails, so a broken deploy cannot pass silently.
JSON for pipelines
npx github:JaydenYoonZK/ai-crawler-audit --json example.comThe full audit as machine-readable JSON: a verdict per crawler, the policy result, and the llms.txt status, ready for dashboards and scheduled checks.
Full flags and exit codes are in the README.
π¬ Frequently asked questions
No, if you block the right tokens. Googlebot handles Search; Google-Extended only controls AI training use. The same separation holds for Applebot and Applebot-Extended. The audit table marks these as control tokens so they are hard to confuse.
The dataset covers the crawlers that show up in real server logs, with each one's vendor documentation linked where it exists. Two entries have no official docs and are labeled accordingly, so you can decide how much weight to give them.
Browsers cannot read other sites' files (CORS), which is why the page works on pasted content. The repo ships a small CLI that fetches live: npx github:JaydenYoonZK/ai-crawler-audit example.com.
The dataset carries an update date, shown next to the audit button, and lives in one reviewable JSON file. New bots appear regularly; pull requests with a log sample and documentation link are the fastest way to get one added.
π± Why I built this
When I published my first pages, I had to write a robots.txt policy for AI crawlers and realized I did not know half the current tokens. Every list I found disagreed with the next one. This audit is the check I wanted for my own sites: paste the file, see what each bot may actually do.