Meta-ExternalAgent in robots.txt
meta-externalagent is Meta's crawler. Here's exactly what it does, whether it respects robots.txt, and the rules to control it. To apply a policy in one click, use the AI Crawler Manager.
What Meta-ExternalAgent is
Meta-ExternalAgent is Meta's crawler for collecting data to train its AI models, including the Llama family and Meta AI. It is distinct from facebookexternalhit, which only fetches pages to build link previews.
| Property | Value |
|---|---|
| User-agent | meta-externalagent |
| Operator | Meta |
| Category | AI training crawlers |
| Honors robots.txt | Yes |
| Affects search ranking | No |
Official documentation: Meta crawler docs.
What Meta-ExternalAgent does
- Crawls publicly available pages to gather AI training data for Meta.
- Is separate from facebookexternalhit (link previews) and from any Facebook/Instagram indexing.
Why site owners care
- Blocking it keeps your content out of Meta's AI training.
- Because link previews use a different agent, blocking Meta-ExternalAgent does not break Facebook/Instagram sharing.
How to allow or block Meta-ExternalAgent
Add a group targeting the meta-externalagent user-agent. Disallow: / blocks it from your whole site; an empty Disallow: allows it.
User-agent: meta-externalagent
Disallow: /User-agent: meta-externalagent
Disallow:No effect on search ranking
How to verify Meta-ExternalAgent
Match the Meta-ExternalAgent user-agent token and Meta's published crawler IP ranges in your logs.
Does it honor robots.txt?
Recommendation
Recommended: Block
Will blocking Meta-ExternalAgent break Facebook link previews?
No. Link previews on Facebook and Instagram are generated by facebookexternalhit, a separate agent. Blocking Meta-ExternalAgent only opts you out of Meta's AI training and leaves sharing previews intact.
What does Meta-ExternalAgent do?
It crawls public web pages to collect training data for Meta's AI models, such as Llama and Meta AI. It is not a search crawler and does not affect your visibility on Facebook or in search engines.
AI Crawler Manager
Allow or block GPTBot, ClaudeBot, PerplexityBot and more in one place.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.