Next.js robots.txt

A static public/robots.txt is simplest and best when your rules never change. The App Router's app/robots.ts (a MetadataRoute.Robots export) is better when rules depend on the environment, a CMS, or your base URL — it's typed, testable, and keeps the sitemap URL in sync with the rest of your metadata.

Export a default function returning a MetadataRoute.Robots object. Next.js serves it at /robots.txt automatically. This is the approach this very site uses.

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: ['/dev', '/api/'] },
      { userAgent: ['GPTBot', 'CCBot', 'Google-Extended'], disallow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
    host: 'https://example.com',
  }
}

Drop a plain file at public/robots.txt and Next.js serves it verbatim at /robots.txt:

User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

A major advantage of app/robots.ts: you can return Disallow: / on preview deployments and the real rules in production, so staging never gets indexed.

export default function robots(): MetadataRoute.Robots {
  const isProd = process.env.VERCEL_ENV === 'production'
  return isProd
    ? { rules: [{ userAgent: '*', allow: '/' }], sitemap: 'https://example.com/sitemap.xml' }
    : { rules: [{ userAgent: '*', disallow: '/' }] }
}

• Shipping a dev Disallow: / to production

  If you env-gate robots.txt, double-check the production branch returns the allow rules.

• Having both a static file and robots.ts

  The static public file silently overrides the route. Keep only one.

• Hardcoding the wrong host

  Use an absolute https sitemap URL that matches your canonical domain.

• Blocking /_next/static

  That holds your JS and CSS chunks. Blocking it breaks rendering for crawlers.