facebookexternalhit in robots.txt
facebookexternalhit is Meta's crawler. Here's exactly what it does, whether it respects robots.txt, and the rules to control it. To apply a policy in one click, use the AI Crawler Manager.
What facebookexternalhit is
facebookexternalhit is Meta's link-preview crawler. When someone shares your URL on Facebook, Instagram, Messenger, or WhatsApp, this bot fetches the page to read its Open Graph tags and build the preview card. Blocking it removes your rich previews on Meta platforms.
| Property | Value |
|---|---|
| User-agent | facebookexternalhit |
| Operator | Meta |
| Category | Social crawlers |
| Honors robots.txt | Partial |
| Affects search ranking | No |
Official documentation: Meta crawler docs.
What facebookexternalhit does
- Fetches a shared URL to read Open Graph / meta tags and generate the preview thumbnail, title, and description.
- Runs when links are posted or scraped on Facebook, Instagram, Messenger, and WhatsApp.
Why site owners care
- Blocking it means links to your site show no image or description when shared on Meta platforms — hurting click-through.
- It is not used for AI training (that's Meta-ExternalAgent) and is not a search crawler.
How to allow or block facebookexternalhit
Add a group targeting the facebookexternalhit user-agent. Disallow: / blocks it from your whole site; an empty Disallow: allows it.
User-agent: facebookexternalhit
Disallow: /User-agent: facebookexternalhit
Disallow:No effect on search ranking
How to verify facebookexternalhit
Match the facebookexternalhit user-agent and Meta's published IP ranges.
Does it honor robots.txt?
Recommendation
Recommended: Allow
Why are my Facebook share previews blank?
Often because facebookexternalhit is blocked in robots.txt (or your Open Graph tags are missing). Allow the facebookexternalhit user-agent and make sure your pages expose og:title, og:description, and og:image.
Is facebookexternalhit the same as Meta's AI crawler?
No. facebookexternalhit builds link previews; Meta-ExternalAgent collects AI training data. They are separate agents, so you can allow previews while blocking AI training.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.