Fix robots.txt Disallow All: Unblock Googlebot Site-Wide

What's happening

A robots.txt file containing User-agent: * followed by Disallow: / blocks every well-behaved crawler from every URL on the host. Googlebot honors this immediately and stops fetching new pages. Pages already in the index begin to drop out as Google revisits them and finds them blocked from crawl. The Search Console Page indexing report flags these URLs as 'Blocked by robots.txt'.

This rule is the canonical staging-site robots.txt — most teams use it on dev.example.com and preview environments to keep them out of search. The disaster scenario is when this file is deployed to production unchanged, either through an environment-variable mistake, a misconfigured CDN edge rule, or a copy-paste during a migration.

Google's behavior here is asymmetric. Once unblocked, recovery is not instant — the indexer must recrawl every URL to re-verify it is allowed, and only then re-eligibility for ranking is restored. Recovery typically takes 1-4 weeks depending on site size and crawl budget.

Why it matters

Organic traffic disappears within days. Pages drop out of Google's index as Google's recrawl cycle hits them. For high-traffic pages this can mean a 60-90% drop within a week, with full collapse after two weeks. Bing and Yandex behave the same way.

Recovery is slow. Even after fixing robots.txt, you depend on Google's recrawl schedule. URL inspection in Search Console with 'Test live URL' followed by 'Request indexing' speeds up a handful of priority pages, but bulk recovery requires patience plus a freshly submitted sitemap.

Reputation in third-party indexes (Ahrefs, Semrush, AI training datasets) also degrades, because they crawl according to robots.txt and stop revisiting blocked URLs. Some never come back without manual intervention.

Common causes

• A staging robots.txt was deployed to production unchanged.
• A reverse proxy or CDN serves a global Disallow: / for all environments.
• An environment variable like ROBOTS_DISALLOW_ALL=true was flipped on by mistake.
• A framework-level config (e.g. Next.js robots.ts) was committed with the staging defaults.
• A WordPress 'Discourage search engines from indexing this site' setting was checked.
• A Cloudflare worker is rewriting /robots.txt with a hardcoded blanket disallow.

Detect this on your site

Run a quick scan with the SEO Auditor. The tool surfaces this exact issue with the records and context needed to apply the fix below.

How to fix it

1. 1

   Confirm the disallow

   Open https://yourdomain.com/robots.txt directly in a browser or run curl. If you see User-agent: * followed by Disallow: /, every crawler is blocked. Also check Search Console > Settings > robots.txt — Google's last-fetched copy is shown there with the timestamp.

2. 2

   Identify the source

   Trace where the file is generated. For Next.js it is app/robots.ts; for WordPress it is the Yoast/RankMath settings or the wp-admin > Settings > Reading checkbox. For Cloudflare check Page Rules and Workers. Fixing in the right layer matters — overriding higher up will get reverted on next deploy.

3. 3

   Replace with a production-appropriate ruleset

   Use User-agent: * followed by Allow: /, plus targeted Disallow rules for /admin, /cart, /api, and parameter URLs. Add a Sitemap: directive. Re-deploy and verify the new file is live.

4. 4

   Validate with the robots.txt Tester

   In Search Console > Settings > Crawling > robots.txt, click Open report. The Tester lets you submit specific URLs and confirm Googlebot is no longer blocked. Test high-value pages first.

5. 5

   Submit the sitemap and request indexing

   Resubmit your sitemap in Search Console to trigger a fresh crawl pass. For your top 10-20 priority URLs, use URL Inspection > Request Indexing to push them to the front of the recrawl queue.

6. 6

   Add a guardrail to prevent recurrence

   Gate the disallow-all version of robots.txt behind the production environment variable explicitly. A unit test or deployment check that fails if the production /robots.txt contains Disallow: / on a bare User-agent: * is cheap insurance.

Example

# WRONG - the disaster file
User-agent: *
Disallow: /

# RIGHT - production baseline
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

The bad file vs. a safe production replacement

Related fixes