SEO
robots.txt Best Practices
robots.txt sits at the root of your domain and controls which paths crawlers may access. Misconfiguring it is one of the most common ways to accidentally tank your SEO.
The basics
User-agent: * applies to all crawlers. Disallow: /admin blocks crawl. Allow: /admin/public allows a sub-path. Sitemap: https://yoursite.com/sitemap.xml points crawlers to your sitemap.
Don't block CSS or JS
Modern Google needs to render your pages to evaluate them. Blocking CSS/JS causes Google to see broken layouts and rank you lower. Allow /static/ /assets/ /_next/ etc.
robots.txt is not security
robots.txt is a polite request. Bad bots ignore it. Use auth + IP allowlists to actually block sensitive paths. Disallowing /admin in robots.txt doesn't prevent access — and signals to attackers where to look.
Test before publishing
Google Search Console has a robots.txt tester. Always test changes against your most important paths before pushing to production.
Want to verify your setup?
Run the check now