Free AI crawler tool
Can AI crawlers reach your site?
Paste a domain and see which of 34 AI crawlers your robots.txt allows or blocks. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest, each with the exact Disallow line doing the blocking. Server-side, so it works where browser checkers fail. Free, no sign-up.
Straight answer
What this checks, and what it can't.
It checks permission, not visibility.robots.txt tells crawlers what they may fetch. Allowing GPTBot or ClaudeBot does not mean ChatGPT or Claude will cite you; that depends on your content and each engine's own ranking, which no robots file controls. Anyone selling "unblock the bots and get cited" is overstating it.
But the opposite failure is real and common. Plenty of sites block the exact AI crawlers they want to reach them, often by an old Disallow: / rule or a CMS default. This catches that in one paste, and shows the precise line to change.
And robots.txt isn't the whole story. A bot that robots.txt allows can still be blocked by a WAF or Cloudflare, and some bots ignore robots.txt entirely. We flag those caveats honestly, and our Agent Readiness scan verifies what crawlers actually receive.
From a domain to a crawler-access report in three steps
- 01Give
A domain or URL
Enter any site. We fetch its /robots.txt server-side, so the check works even on sites that reject browser requests.
- 02Evaluate
34 AI crawler rules
We parse the file against 34 documented AI user-agents from OpenAI, Anthropic, Google, Perplexity, Meta, Apple, Microsoft, Amazon, ByteDance and others, for the homepage path.
- 03Fix
The exact blocking line
Every crawler comes back allowed or blocked, grouped by purpose, with the precise Disallow line and line number to change. No detective work.
The crawlers, briefly
Not all AI crawlers do the same job
We group the 34 crawlers by purpose so a block is a deliberate choice, not an accident. AI-search bots (like OAI-SearchBot and PerplexityBot) feed live AI answers and citations. Training bots (like GPTBot and Google-Extended) collect data to train models, and some sites block these on purpose. User-prompted fetchers (like ChatGPT-User and Claude-User) retrieve a page only when a person asks about it. Blocking the wrong group can quietly remove you from AI answers while doing nothing for the privacy goal you actually had.
For the bigger picture on getting found by AI, read our guide: what generative engine optimization actually is.
FAQ
AI crawlers, answered honestly
- 01Does allowing AI crawlers get me cited by ChatGPT or Perplexity?Not on its own. robots.txt access is permission to fetch, nothing more. Citation depends on content, authority, and each engine's own ranking. What allowing does prevent is the opposite failure: being invisible to AI because a stale rule blocks the crawlers you wanted. That failure is common, silent, and fixable in one line.
- 02Which AI crawlers does it check?34 documented user-agents, maintained against official provider docs and the ai.robots.txt project: OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-User), Google (Google-Extended), Perplexity, Meta, Apple, Microsoft, Amazon, ByteDance, Common Crawl, Cohere, Mistral, xAI and more. Bots known to ignore robots.txt are flagged as such.
- 03Should I block AI crawlers?Depends on your goals, and this tool won't push you either way. Want AI visibility? Blocking the bots that feed AI answers is counterproductive. Protecting proprietary content? Block the training bots and keep the AI-search bots. The grouping exists so you can split that decision instead of making it wholesale.
- 04What's the difference between this and a robots.txt tester?A generic tester answers "is this one URL crawlable by this one bot?" This answers the question you actually have: can AI systems reach me at all, for every bot that matters, in one paste.
- 05Why does my robots.txt allow a bot that still can't reach my site?robots.txt only states a policy. A WAF, Cloudflare bot-fight mode, rate limiting, or a 403 can still block a crawler that robots.txt permits, and some bots ignore robots.txt altogether. That gap between "permitted" and "actually served" is exactly what our Agent Readiness scan measures by live-fetching your site as each crawler. This free checker covers the robots.txt layer only.
- 06Is it free?Yes. Single-domain checks are free with no sign-up. There's no LLM or paid API behind a check, so there's nothing for us to meter.
robots.txt is permission. Agent Readiness checks what crawlers actually receive.
Live-fetch your site as each of 34 AI crawlers, render it like a headless agent, and see what's really blocking you: WAF rules, 403s, JavaScript walls. Free.