Free AI crawler tool
Can AI crawlers reach your site?
Paste a domain and see which of 34 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Googlebot and more — your robots.txt allows or blocks, with the exact line to fix. Server-side, so it works where browser checkers fail. Free, no sign-up.
Straight answer
What this checks — and what it can't.
It checks permission, not visibility.robots.txt tells crawlers what they may fetch. Allowing GPTBot or ClaudeBot doesn't mean ChatGPT or Claude will cite you — that depends on your content and each engine's own ranking, which no robots file controls. Anyone selling “unblock the bots and get cited” is overstating it.
But the opposite failure is real and common. Plenty of sites block the exact AI crawlers they want to reach them — often by an old Disallow: / rule or a CMS default. This catches that in one paste, and shows the precise line to change.
And robots.txt isn't the whole story. A bot that robots.txt allows can still be blocked by a WAF or Cloudflare, and some bots ignore robots.txt entirely. We flag those caveats honestly — and our Agent Readiness scan verifies what crawlers actually receive.
From a domain to a crawler-access report in three steps
- 01 · Give
A domain or URL
Enter any site. We fetch its /robots.txt server-side — so it works even on sites that block browser requests.
- 02 · Evaluate
34 AI crawler rules
We parse robots.txt against 34 known AI crawler user-agents — OpenAI, Anthropic, Google, Perplexity, Meta and more — for the homepage path.
- 03 · Fix
The exact blocking line
You see every allowed and blocked crawler, grouped by purpose, and the precise Disallow line and number to change for each block.
The crawlers, briefly
Not all AI crawlers do the same job
We group the 34 crawlers by purpose so a block is a deliberate choice, not an accident. AI-search bots (like OAI-SearchBot and PerplexityBot) feed live AI answers and citations. Training bots (like GPTBot and Google-Extended) collect data to train models — some sites block these on purpose. User-prompted fetchers (like ChatGPT-User and Claude-User) retrieve a page only when a person asks about it. Blocking the wrong group can quietly remove you from AI answers while doing nothing for the privacy goal you actually had.
For the bigger picture on getting found by AI, read our guide: what generative engine optimization actually is.
FAQ
AI crawlers, answered honestly
- 01Does allowing AI crawlers help my site get cited by ChatGPT or Perplexity?Not on its own. Allowing a crawler in robots.txt is permission to fetch your pages — nothing more. Whether an AI engine actually cites you depends on your content, your authority, and each engine's own ranking, none of which robots.txt controls. This tool checks reachability, not visibility, and never claims that unblocking a bot earns you citations. What it does prevent is the opposite failure: accidentally blocking the crawlers you want to reach you.
- 02What's the difference between this and a robots.txt tester?A generic robots.txt tester checks one user-agent and one path at a time. This checks the homepage against all 34 AI-specific crawlers at once — GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, Bytespider and the rest — and groups them by purpose (AI search, model training, user-prompted fetches). It's built for the question "can AI systems reach me?", not "is this one URL crawlable?".
- 03Which AI crawlers does it check?34 documented AI crawler user-agents from OpenAI, Anthropic, Google, Perplexity, Meta, Apple, Microsoft, Amazon, ByteDance, Common Crawl, Cohere, Mistral, xAI and others. The list is maintained against official provider docs and the ai.robots.txt project. A few bots (for example Bytespider) are known to ignore robots.txt — we flag those, because for them an "allowed" verdict isn't a guarantee they'll respect a block.
- 04Why does my robots.txt allow a bot that still can't reach my site?robots.txt only states a policy. A WAF, Cloudflare bot-fight mode, rate limiting, or a 403 can still block a crawler that robots.txt permits — and some bots ignore robots.txt altogether. That gap between "permitted" and "actually served" is exactly what our Agent Readiness scan measures by live-fetching your site as each crawler. This free checker covers the robots.txt layer only.
- 05Should I block AI crawlers?It depends on your goals, and this tool doesn't push you either way. If you want visibility in AI answers, blocking the crawlers that feed them is usually counterproductive. If you're protecting proprietary content from model training, you may deliberately block training bots (like Google-Extended or GPTBot) while allowing AI-search bots (like OAI-SearchBot). Grouping crawlers by purpose helps you make that call deliberately rather than by accident.
- 06Is it free?Yes — single-domain checks are completely free and need no sign-up. There's no LLM or paid API behind a check, so there's nothing to meter.
robots.txt is permission. Agent Readiness checks what crawlers actually receive.
Live-fetch your site as each of 34 AI crawlers, render it like a headless agent, and see what's really blocking you — WAF rules, 403s, JavaScript walls. Free.