Google removed the robots.txt tester. Is this the same thing?

It does the same job. Google retired the robots.txt tester in Search Console, but the need didn't go anywhere — you still have to know whether a rule blocks a URL before you publish it. This implements the same matching behaviour Googlebot uses: user-agent tokens matched by equality with the documented fallback chain, all groups for the winning token merged, longest matching rule wins, Allow beats Disallow on an exact-length tie, plus wildcard and $ anchor support and percent-encoding normalization. Search Console still has a robots.txt report showing the fetched file and its status; what it no longer has is the interactive URL tester.

How do I find and read a site's robots.txt?

It's always at the root of the host: example.com/robots.txt, straight in a browser. A file in a subfolder is ignored, and each subdomain has its own — blog.example.com is not covered by the robots.txt on example.com, which is a common blind spot. Reading it is where it gets slippery, because the file doesn't execute top to bottom the way it looks. Find the group whose User-agent names your crawler (or the * group if none does), remembering that a named group replaces the * group rather than adding to it, then merge every group declaring that same token. Within that set the longest matching path wins, not the first — so order genuinely doesn't matter, contrary to a lot of advice. Rather than doing that by hand, paste the domain above with the URLs you care about and the tool shows you the resolved group and the deciding rule for each one.

Why does my named crawler ignore the rules in the * group?

Because that's the spec, and it surprises nearly everyone. A crawler obeys exactly one user-agent token — the most specific one that names it — and every group declaring that token is merged into the set it follows. If your file has a `User-agent: *` group and a `User-agent: Googlebot` group, Googlebot follows the Googlebot rules and ignores the * group entirely — the two are not combined. So adding `User-agent: Googlebot / Disallow: /private/` while your * group blocks /admin/ means Googlebot can now crawl /admin/. This tool shows which group each crawler actually follows so you can see it happening.

What does 'longest match wins' mean in practice?

When several rules match a URL, the one with the longest path wins — not the first, and not the last. So `Disallow: /private/` plus `Allow: /private/public-page` means that one page is crawlable, because the Allow path is longer. If two matching rules are exactly the same length, Allow wins. This is why rule order in robots.txt genuinely doesn't matter, which contradicts a lot of advice you'll read.

Does robots.txt keep a page out of Google?

No, and this is the most expensive misunderstanding in the format. robots.txt controls crawling, not indexing. A blocked URL can still appear in results — usually as a bare link with no description — if Google finds it linked elsewhere. To keep a page out of the index you need a noindex meta tag or X-Robots-Tag header, and the page must be crawlable for Google to see that directive. Blocking a page you also want deindexed is self-defeating: the crawler can't fetch the noindex you added.

Should I block GPTBot, ClaudeBot and the other AI crawlers?

It depends which one, and the distinction matters more than the decision. Training crawlers (GPTBot, ClaudeBot, CCBot) feed model training — blocking them keeps your content out of training data and won't remove you from any AI answer given today, since those are retrieved live. What you give up is slower-burning: future model generations know less about you natively, which shows when an assistant answers from memory instead of browsing. Google-Extended is the awkward middle case: it governs Gemini training and Gemini grounding, so blocking it can cost you grounded Gemini answers, though it never affects Google Search or AI Overviews ranking. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the indexes those assistants answer from — blocking those removes you from the answers. People routinely block the second kind while meaning to block the first. Our generator labels each crawler by purpose for exactly this reason, and the AI Crawler Checker shows what your current file does across 34 of them.

Is a missing robots.txt a problem?

No. A 404 on /robots.txt means everything is allowed, which is the correct outcome for most sites. You need the file when you have something specific to keep crawlers out of, or when you want to declare your sitemap. An empty file and no file behave identically.

Do you handle Crawl-delay and wildcards?

Yes to both. Wildcards (*) and end-anchors ($) are matched with Google's own linear algorithm rather than a regex, so pathological patterns can't hang the tool. Crawl-delay is parsed and reported, with an important caveat: Bing still honours it, Googlebot never has, and Yandex dropped support in 2018 — so on most sites the line does nothing. Google auto-tunes its crawl rate and retired the Search Console crawl-rate setting in January 2024, so if Googlebot is genuinely too aggressive the levers are 429/503 responses or Google's report-a-crawl-problem form.

Free robots.txt tool

Free robots.txt Tester & Checker

Google retired the robots.txt tester in Search Console. This does the same job: check any URL against a live or draft robots.txt, per crawler, with Google's real precedence rules — and see the exact line that decided each verdict. Free, no sign-up.

The rule everyone gets wrong

A named group replaces the `*` group

A crawler follows exactly one group— the most specific one that names it. The rules aren't combined. So this file does not do what it looks like it does:

User-agent: *
Disallow: /admin/

User-agent: Googlebot
Disallow: /private/

Googlebot follows the second group only, so /admin/ is now crawlable by Google — the exact opposite of the intent. Every other crawler still respects it. To block both, the Googlebot group has to repeat the /admin/ rule.

There's a second half to this that trips up plugin-heavy sites: if the same user-agent appears in several groups, those groups are merged. WordPress sites often end up with two separate User-agent: * blocks because a plugin appended its own, and both sets of rules apply. Tools that read only the first block silently under-report what's blocked.

This tester shows you which group each crawler resolves to, and the exact rule behind every verdict, so neither case has to be taken on faith.

From a domain to a per-crawler verdict

01Fetch
A domain, or paste the file
Give a domain and we fetch its live robots.txt server-side, or paste a draft you haven't published yet — the second is how you check a change before it costs you traffic.
02Test
Real URLs against real crawlers
List the paths you care about and pick the crawlers: Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended. Every combination gets a verdict.
03Understand
Which rule decided, and why
Each verdict shows the exact line that produced it, and we tell you which group a crawler actually follows — because a named group replaces the * group rather than adding to it, which is the mistake we see most.

Worth knowing

robots.txt controls crawling, not indexing

Blocking a URL asks crawlers not to fetchit. It does not remove it from search results. A blocked page that's linked from elsewhere can still be listed — typically as a bare URL with no description, because the crawler was never allowed to read the page it's describing.

The trap: to deindex a page you need noindex, and Google has to crawl the pageto see it. Blocking the page in robots.txt prevents that. If you want something gone from the index, allow crawling and add noindex — then block it later if you like, once it's dropped.

And robots.txt is never a security control. It is a public file listing the paths you'd rather people didn't visit, which is closer to a map than a lock.

FAQ

robots.txt, answered properly

01Google removed the robots.txt tester. Is this the same thing?
It does the same job. Google retired the robots.txt tester in Search Console, but the need didn't go anywhere — you still have to know whether a rule blocks a URL before you publish it. This implements the same matching behaviour Googlebot uses: user-agent tokens matched by equality with the documented fallback chain, all groups for the winning token merged, longest matching rule wins, Allow beats Disallow on an exact-length tie, plus wildcard and $ anchor support and percent-encoding normalization. Search Console still has a robots.txt report showing the fetched file and its status; what it no longer has is the interactive URL tester.
02How do I find and read a site's robots.txt?
It's always at the root of the host: example.com/robots.txt, straight in a browser. A file in a subfolder is ignored, and each subdomain has its own — blog.example.com is not covered by the robots.txt on example.com, which is a common blind spot. Reading it is where it gets slippery, because the file doesn't execute top to bottom the way it looks. Find the group whose User-agent names your crawler (or the * group if none does), remembering that a named group replaces the * group rather than adding to it, then merge every group declaring that same token. Within that set the longest matching path wins, not the first — so order genuinely doesn't matter, contrary to a lot of advice. Rather than doing that by hand, paste the domain above with the URLs you care about and the tool shows you the resolved group and the deciding rule for each one.
03Why does my named crawler ignore the rules in the * group?
Because that's the spec, and it surprises nearly everyone. A crawler obeys exactly one user-agent token — the most specific one that names it — and every group declaring that token is merged into the set it follows. If your file has a `User-agent: *` group and a `User-agent: Googlebot` group, Googlebot follows the Googlebot rules and ignores the * group entirely — the two are not combined. So adding `User-agent: Googlebot / Disallow: /private/` while your * group blocks /admin/ means Googlebot can now crawl /admin/. This tool shows which group each crawler actually follows so you can see it happening.
04What does 'longest match wins' mean in practice?
When several rules match a URL, the one with the longest path wins — not the first, and not the last. So `Disallow: /private/` plus `Allow: /private/public-page` means that one page is crawlable, because the Allow path is longer. If two matching rules are exactly the same length, Allow wins. This is why rule order in robots.txt genuinely doesn't matter, which contradicts a lot of advice you'll read.
05Does robots.txt keep a page out of Google?
No, and this is the most expensive misunderstanding in the format. robots.txt controls crawling, not indexing. A blocked URL can still appear in results — usually as a bare link with no description — if Google finds it linked elsewhere. To keep a page out of the index you need a noindex meta tag or X-Robots-Tag header, and the page must be crawlable for Google to see that directive. Blocking a page you also want deindexed is self-defeating: the crawler can't fetch the noindex you added.
06Should I block GPTBot, ClaudeBot and the other AI crawlers?
It depends which one, and the distinction matters more than the decision. Training crawlers (GPTBot, ClaudeBot, CCBot) feed model training — blocking them keeps your content out of training data and won't remove you from any AI answer given today, since those are retrieved live. What you give up is slower-burning: future model generations know less about you natively, which shows when an assistant answers from memory instead of browsing. Google-Extended is the awkward middle case: it governs Gemini training and Gemini grounding, so blocking it can cost you grounded Gemini answers, though it never affects Google Search or AI Overviews ranking. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the indexes those assistants answer from — blocking those removes you from the answers. People routinely block the second kind while meaning to block the first. Our generator labels each crawler by purpose for exactly this reason, and the AI Crawler Checker shows what your current file does across 34 of them.
07Is a missing robots.txt a problem?
No. A 404 on /robots.txt means everything is allowed, which is the correct outcome for most sites. You need the file when you have something specific to keep crawlers out of, or when you want to declare your sitemap. An empty file and no file behave identically.
08Do you handle Crawl-delay and wildcards?
Yes to both. Wildcards (*) and end-anchors ($) are matched with Google's own linear algorithm rather than a regex, so pathological patterns can't hang the tool. Crawl-delay is parsed and reported, with an important caveat: Bing still honours it, Googlebot never has, and Yandex dropped support in 2018 — so on most sites the line does nothing. Google auto-tunes its crawl rate and retired the Search Console crawl-rate setting in January 2024, so if Googlebot is genuinely too aggressive the levers are 429/503 responses or Google's report-a-crawl-problem form.

Need to write the file, not just test it?

The robots.txt generator builds one from presets — WordPress, Shopify, block-admin — with per-crawler AI directives labelled by what blocking each one actually costs you. Free.

Generate a robots.txt

Free robots.txt Tester & Checker

A named group replaces the * group

From a domain to a per-crawler verdict

A domain, or paste the file

Real URLs against real crawlers

Which rule decided, and why

robots.txt controls crawling, not indexing

robots.txt, answered properly

Need to write the file, not just test it?

A named group replaces the `*` group