What Is Meta-ExternalAgent?
Meta-ExternalAgent is the web crawler Meta uses to collect content for its AI systems. Meta's developer documentation says it "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." It identifies itself as meta-externalagent/1.1, and the robots.txt token is meta-externalagent.
Meta's AI Crawlers Compared
Meta documents three AI-related agents plus the older link-preview crawler. They do different jobs and follow robots.txt differently.
| Agent | Purpose, in Meta's words | robots.txt |
|---|---|---|
| meta-externalagent | Training foundation AI models or improving products by indexing content directly | Standard rules apply |
| meta-externalfetcher | Fetches individual links at a user's request and supports evaluating and improving agentic AI capabilities | Meta says this crawler may bypass robots.txt rules |
| meta-webindexer | Navigates the web to improve Meta AI search result quality | Controlled by its own token |
| facebookexternalhit | Crawls content shared on Meta's apps to build link previews | May bypass rules for security or integrity checks |
Meta says robots.txt changes can take up to 24 hours to register.
robots.txt Examples
To opt out of training while staying eligible for Meta AI search results:
User-agent: meta-externalagent
Disallow: /
User-agent: meta-webindexer
Allow: /Blocking meta-externalagent does not stop meta-externalfetcher, which acts on a user's request and which Meta says may bypass robots.txt. If you need to refuse it, that has to happen at the firewall or CDN.
Why It Matters for AI Visibility
Meta AI is built into WhatsApp, Instagram, Facebook, and Meta's glasses, and Meta now runs a separate agent, Muse, that browses and buys on a user's behalf. What these products know about a brand comes partly from what Meta's crawlers could read. A site that blocks all Meta agents is opting out of that entire surface. A site that blocks only the training crawler keeps search and user-initiated fetches working. See Meta AI citation patterns and Meta Muse: what it means for brands.
Commonly Confused With
Meta-ExternalAgent is not facebookexternalhit, the link-preview crawler that has existed for years and has nothing to do with AI training. It is also not the browser a Muse agent uses to shop on a site, which Amazon has said does not identify itself. That dispute is covered in Amazon blocks Meta Muse.