G
GEO Toolbox

AI Crawlers

meta-externalagent

Also: Meta AI crawler, Meta-ExternalAgent

meta-externalagent is Meta's web crawler, used to gather public content to train its AI models and index the web. It identifies with the user agent meta-externalagent and respects robots.txt. It is distinct from Meta-ExternalFetcher, which fetches links on a user's behalf.

Updated

meta-externalagent identifies with a user agent beginning meta-externalagent and follows robots.txt, so you control it the standard way. Its job is collecting public data for Meta's AI models and indexing content.

It is worth separating from Meta-ExternalFetcher, the agent that retrieves a specific link when a user shares or acts on it. User-initiated fetches like that may not honor robots.txt, so a robots rule governs the training crawler more reliably than the fetcher.