Fix Missing robots.txt: Add a Crawl Directives File

What's happening

robots.txt is the Robots Exclusion Protocol file that lives at the root of every host. It is the first URL Googlebot, Bingbot, and most legitimate crawlers fetch when they visit a domain. The file specifies which paths each user-agent may or may not crawl, where the sitemap lives, and any crawl-delay hints.

When robots.txt returns a 404, well-behaved crawlers assume an implicit 'allow all' and continue crawling. The site is not penalized, but you give up several capabilities: you cannot point crawlers to your sitemap, you cannot block AI crawlers like GPTBot or ClaudeBot, you cannot prevent crawl-budget waste on faceted URLs, and you cannot restrict staging or admin paths.

More dangerously, when robots.txt returns a 5xx error, Google interprets that as a temporary 'do not crawl anything' directive. If the file is intermittently unreachable due to a server config bug, indexation can stall site-wide for hours. A 404 is safer than a 5xx, but a properly served 200 is best.

Why it matters

Crawl budget waste is the most direct cost. Without disallow rules, crawlers chase parameter URLs (?sort=,?page=,?utm_source=), session-ID URLs, and internal search result pages. On large sites this consumes a meaningful fraction of crawl budget that should go to canonical content.

Sitemap discovery is delayed. Without a Sitemap: directive in robots.txt, you rely on Search Console submission alone, which means non-Google crawlers may never find the sitemap. Bing, Yandex, and AI training crawlers all read robots.txt for sitemap hints.

Bot management is impossible. Many sites block aggressive AI training crawlers (GPTBot, ClaudeBot, PerplexityBot, Bytespider) via robots.txt User-agent directives. Without a robots.txt file you have no documented opt-out, and many AI vendors treat the absence as consent.

Common causes

• The site was deployed without a robots.txt route or static file.
• The framework's robots.txt generator was disabled or never configured.
• A reverse proxy (Cloudflare, Nginx) is rewriting /robots.txt to a 404.
• The file exists but is at a non-root path like /static/robots.txt and not served at /robots.txt.
• robots.txt returns a 5xx error due to a misconfigured Function or middleware.
• A migration removed the file and nobody noticed because Search Console does not warn loudly.

Detect this on your site

Run a quick scan with the SEO Auditor. The tool surfaces this exact issue with the records and context needed to apply the fix below.

How to fix it

1. 1

   Verify the current state

   Run curl -I https://yourdomain.com/robots.txt. A 200 response with text/plain content type is what you want. A 404 means the file is missing. A 5xx means the server is broken — fix that immediately, it can stall crawling site-wide.

2. 2

   Create a permissive baseline

   Create /robots.txt at the site root with at minimum: a User-agent: * line, an Allow: / line, and a Sitemap: directive pointing to your sitemap.xml. This unblocks the basics without restricting crawling.

3. 3

   Add disallow rules for waste

   Disallow paths that crawlers should not waste budget on: /search, /cart, /checkout, /api, /admin, query-string variants. Use Disallow: /*?sort= to block sort-parameter URLs while allowing the canonical path.

4. 4

   Add user-agent specific rules if needed

   If you want to block AI training crawlers, add explicit User-agent: GPTBot, User-agent: ClaudeBot, User-agent: PerplexityBot blocks each followed by Disallow: /. Each user-agent block is independent.

5. 5

   Test in Google Search Console

   Use the robots.txt Tester (under Settings > Crawling) to confirm Googlebot is not blocked from any URL you care about. Test specific paths to make sure the Disallow rules behave as expected — order does not matter, but the most specific match wins.

6. 6

   Monitor for 5xx responses

   Set up uptime monitoring on /robots.txt specifically. A 5xx response on this file pauses crawling much faster than other types of outages, so it deserves dedicated alerting.

Example

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /*?sort=
Disallow: /*?utm_

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Baseline robots.txt with sitemap and AI-bot opt-out

Related fixes