AI Crawlers
Also: robots file
robots.txt is a plain-text file at the root of a site that tells crawlers which paths they may or may not fetch, by user agent. For AI search it is the primary control for allowing or blocking crawlers like GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Most well-behaved AI crawlers respect it.
Updated
A single overly broad rule, such as a catch-all User-agent: * disallow meant to stop scrapers, can silently block the AI crawlers you actually want. Check each AI user agent explicitly.
robots.txt governs crawling, not user-initiated fetches: agents like ChatGPT-User and Perplexity-User act on a specific user request and generally ignore it.
Go deeper
How to Optimize for AI SearchRelated terms