Free robots.txt tool
Free robots.txt Generator
Start from a preset, add your paths, declare your sitemap — and decide about the AI crawlers with each one labelled by what blocking it actually costs you. Validated as you type against the same checks our tester runs. Free, no sign-up, nothing metered.
WordPress core's own defaults: keep /wp-admin/ out of the index but allow admin-ajax.php, which themes and plugins need.
One line, no downside — it's how any crawler finds your sitemap without you submitting it anywhere.
Bing honours this.Googlebot never has, and Yandex dropped it in 2018 — so today it's really a Bing setting.
Blocking a training crawler keeps your content out of model training. Blocking a searchone removes you from that engine's answers. They're different decisions and people conflate them constantly.
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/
Save at your site root: /robots.txt. It must be at the root — a robots.txt in a subfolder is ignored.
Checked against the same rules as our tester
- infoNo sitemap declared. Adding a `Sitemap: https://…/sitemap.xml` line is the standard way for any crawler to discover your sitemap without you submitting it anywhere. One line, no downside.
The decision this page exists for
“Should I block the AI bots?” is two different questions
Most robots.txt generators were written before this mattered and still offer a list of search engines from 2015. The live question in 2026 is about AI crawlers — and the reason it goes wrong so often is that people treat them as one category when they're two.
Training crawlers— GPTBot, ClaudeBot, CCBot — collect content to train models. Blocking them keeps you out of training data and costs you nothing in any answer given today, because they aren't what answers a user's question — the trade is that future models know less about you natively, which surfaces when an assistant replies without browsing. Google-Extended is the awkward middle case: it governs Gemini training and grounding in Gemini Apps, so blocking it can cost you grounded Gemini answers even though it never touches Google Search ranking.
Search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — build the indexes those assistants actually search when someone asks a question. Block these and you disappear from the answers.
The costly version of this mistake is copying a “block AI bots” snippet that lumps both together, and quietly removing yourself from ChatGPT and Perplexity results while believing you only opted out of training. Two specifics worth remembering: blocking GPTBot does not remove you from ChatGPT search, and blocking Google-Extended does not affect Google Search or AI Overviews.
That's why every crawler in the picker is labelled by purpose. You may well want to block training and keep search — that's a perfectly coherent position, and it's hard to express if your tool won't tell you which is which.
From a preset to a publishable file
- 01Pick
A preset that matches your stack
WordPress keeps /wp-admin/ out while allowing admin-ajax.php, which themes and plugins need. Shopify blocks cart, checkout and account flows. Or start from allow-everything and add your own paths.
- 02Decide
About the AI crawlers, deliberately
Each one is labelled by what it does: training crawlers feed model training, search crawlers build the indexes AI assistants answer from. Blocking the second kind removes you from the answers, which is almost never what people mean to do.
- 03Ship
A file that passes our own tester
The output is validated as you type against the same checks our robots.txt tester runs on other people's files — including the own-goals like blocking your own CSS or the sitemap you just declared. Copy or download, save at your site root.
The format
What the directives actually mean
A robots.txt is groups of rules. Each group starts with one or more User-agent: lines naming the crawlers it applies to, followed by Allow: and Disallow: paths.
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml
Three things that surprise people. Order doesn't matter — the longest matching path wins, with Allow breaking an exact-length tie, which is why the admin-ajax Allow above beats the broader Disallow. An empty Disallow means allow everything, not block everything. And a named group replaces the * group for that crawler rather than adding to it — so GPTBot above follows only its own rule.
Once you've published, test it against real URLs — and if you need the sitemap URL for that last line, the sitemap extractor finds it from a bare domain.
Templates
robots.txt examples you can copy
The generator above builds these for you and validates the result, but if you just want the file, here it is. The caveat under each one is the part that usually gets left out.
Allow everything
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
An empty Disallow: means allow everything — the opposite of Disallow: /, which blocks everything. One character between the two. If you have nothing to block, no file at all does the same job; the only reason to publish this one is the Sitemap line.
Block everything
User-agent: * Disallow: /
Correct for a staging site, and dangerous anywhere else — this is the single most expensive line in SEO, usually because it survived a launch. It also does not deindex anything: blocked URLs already in the index tend to stay there as bare links, because Google can no longer fetch the page to see a noindex. If the goal is removal, keep the pages crawlable and serve noindex instead. For staging, HTTP auth is the real answer; robots.txt is a request, not a lock.
WordPress
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Sitemap: https://example.com/wp-sitemap.xml
That Allow line is not optional decoration — plenty of themes and plugins route front-end functionality through admin-ajax.php, and blocking it can stop Google rendering parts of your pages. Note the sitemap path: WordPress core has published /wp-sitemap.xml since 5.5, but Yoast and Rank Math replace it with /sitemap_index.xml — check which one your site actually serves before pasting.
Block a single folder
User-agent: * Disallow: /private/ Allow: /private/public-page
Paths are prefixes, so Disallow: /private without the trailing slash would also catch /private-beta/. The exception works because the longer path wins regardless of ordering.
Block AI training, keep AI search
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: /
This is the distinction almost every “block AI bots” snippet in circulation gets wrong. These three feed model training. The search crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — are deliberately absent, because blocking those removes you from the answers those assistants give. Copy-paste lists that lump all of them together quietly cost you AI visibility. Check what your current file does across 34 of them.
Removing a file is a different question from writing one. On a hosted platform like Shopify or Wix you can't delete /robots.txt— every store gets a generated default. Shopify's official mechanism for changing it is the robots.txt.liquidtheme template, though worth knowing before you start: Shopify classes edits to that file as an unsupported customization, so their support team won't help you with them. Serving a 404 there wouldn't be an improvement anyway — a missing robots.txt and one that allows everything mean the same thing to a crawler.
FAQ
robots.txt, answered properly
- 01Where does robots.txt go?At the root of the host: https://example.com/robots.txt. It must be at the root — a file in a subfolder is ignored entirely. Each subdomain needs its own, so blog.example.com is not covered by the robots.txt on example.com, and http and https are technically separate origins too.
- 02Should I block GPTBot, ClaudeBot and the rest?Decide per crawler, because they do different jobs and the labels here reflect that. Training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, meta-externalagent, Applebot-Extended — feed model training; blocking them keeps your content out of training data and won't remove you from any AI answer given today, since those are retrieved live. The cost is slower-burning — future models know less about you natively, which shows when an assistant answers without browsing. Search and user-request crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User — fetch pages to answer questions; blocking those removes you from those answers. The common mistake is blocking a search crawler while meaning to block training. One specific point worth internalizing: blocking GPTBot does NOT remove you from ChatGPT's search results, and blocking Google-Extended does NOT affect Google Search or AI Overviews ranking, though it does also govern Gemini grounding, so it isn't entirely free.
- 03Why does the WordPress preset allow admin-ajax.php?Because that's what WordPress core does, and for good reason. Plenty of themes and plugins route front-end functionality through /wp-admin/admin-ajax.php, so blocking the whole of /wp-admin/ without that exception can stop Google from rendering parts of your pages properly. Since the longest matching rule wins, the more specific Allow beats the broader Disallow — no ordering tricks needed.
- 04Does robots.txt stop a page being indexed?No. It controls crawling, not indexing, and conflating the two is the most expensive robots.txt mistake there is. A blocked URL can still show up in results as a bare link if it's referenced elsewhere. To keep a page out of the index, use a noindex meta tag or X-Robots-Tag header — and note the page must stay crawlable for Google to see that directive. Blocking a page in robots.txt actively prevents the noindex from working.
- 05What about Crawl-delay?Bing still honours it. Googlebot never has, and Yandex dropped support in 2018 — so its real audience today is Bing. Google auto-tunes its crawl rate and retired the Search Console crawl-rate setting in January 2024; if Googlebot is genuinely too aggressive, serve 429/503 responses or use Google's report-a-crawl-problem form. We keep the field because it still works for Bing, and flag the caveat rather than letting you believe it does more.
- 06Should I add the Sitemap line?Yes, essentially always. It's one line and it's how any crawler — not just the ones you've submitted to — discovers your sitemap. There's no downside and it's the single easiest thing on this page to get right. If you don't know your sitemap URL, our sitemap extractor will find it from a bare domain.
- 07Is this really free?Yes, and there's nothing to meter — the generator runs entirely in your browser. No crawl, no API call, no sign-up, no email wall. The validation happens client-side too, using the same rule engine as our tester.
Which AI crawlers does your current file already block?
The AI Crawler Checker reads your live robots.txt and reports across 34 AI crawlers — including the ones you didn't mean to block. Free.