Most robots.txt files fall into one of two traps: completely unrestricted (no rules at all) or accidentally blocking something important. The file is small enough to look trivial - a few lines at the domain root - but mistakes are silent until you notice pages missing from search results or traffic declining without an obvious explanation.
This guide covers what robots.txt actually controls, which directives matter in practice, the patterns that cause real crawling problems in production, and how to test your configuration before it affects your site. Whether you're setting up a new site or auditing an existing file that may have accumulated stale rules over years of changes, the principles are the same.
What robots.txt Controls (and What It Doesn't)
robots.txt sits at the root of your domain (https://yourdomain.com/robots.txt) and tells compliant web crawlers which URLs they are allowed to fetch. The key word is "fetch." Disallowing a URL prevents the crawler from reading the page's content - it does not prevent that URL from being indexed.
If a page is blocked in robots.txt but receives inbound links from other indexed pages, Google may still list it as an empty entry with a note that the content was unavailable. It shows in your Search Console coverage report as a "Blocked by robots.txt" URL, ranking for nothing and consuming crawl budget for no benefit.
The correct uses for robots.txt are narrow: blocking admin panels, staging environments, internal search result pages, parameterized URL variants that create duplicate content, and back-end utilities that have no business appearing in search results. Anything that needs to stay genuinely private requires authentication. robots.txt is a public file that anyone can read - scrapers that ignore it are common.
robots.txt is also not a substitute for noindex. If a page is already indexed and you add a Disallow rule, Google may keep the page indexed indefinitely because it can no longer access the noindex meta tag. To deindex a page cleanly, keep it crawlable and use a noindex tag, or submit it for removal through Google Search Console.
Photo by The Four Directives You Will Use in Practice
The robots.txt specification is simpler than most documentation suggests. Four directives handle the vast majority of production requirements.
User-agent identifies which crawler the rules below it apply to. User-agent: * applies to all compliant bots. You can also target specific crawlers: User-agent: Googlebot, User-agent: Bingbot. A single file can contain multiple user-agent blocks with different rules for different crawlers.
Disallow specifies paths the crawler should not fetch. Disallow: /admin/ blocks all URLs under that directory. Disallow: / (no path suffix) blocks the entire site. An empty Disallow: value - no path at all - means allow everything under that agent's rules.
Allow creates exceptions inside a Disallow block. If you block /wp-content/ but need /wp-content/uploads/ accessible, placing Allow: /wp-content/uploads/ before the Disallow statement handles it. Googlebot and Bingbot both implement Allow reliably.
Sitemap tells crawlers where your XML sitemap lives. Multiple Sitemap lines are supported. This is the most underused directive given the crawl efficiency it provides, especially on large or frequently updated sites. One line pointing to your sitemap index takes 30 seconds to add.
A fifth directive you may see in older guides is Crawl-delay. Googlebot ignores it. If you need to limit Googlebot's crawl frequency, adjust it directly in Google Search Console. Bingbot does honor Crawl-delay, so it is useful in a Bingbot-specific block if your server has capacity limitations.
Patterns That Break Crawling in Production
Wildcard behavior is inconsistent. In the path portion of Disallow and Allow rules, * matches any sequence of characters and $ anchors a pattern to the URL's end. Googlebot and Bingbot implement both correctly. Many other crawlers ignore them entirely or treat * as a literal character. For rules that must hold reliably, use explicit path prefixes rather than wildcard patterns.
Blocking WordPress theme and plugin assets. The standard robots.txt for WordPress blocks /wp-admin/, which is correct. Some templates also block /wp-includes/. That directory contains JavaScript and CSS files that Googlebot needs to render pages. Blocking it means Google sees unstyled, potentially broken page renders, which depresses Core Web Vitals scores and may cause JavaScript-dependent content to be missed during indexing.
Trailing slash differences. Disallow: /admin blocks only the exact string /admin. Disallow: /admin/ blocks everything under that directory including /admin/settings and /admin/users. These behave differently, and writing one when you mean the other is an easy mistake.
Stale rules after URL restructure. A path that was correctly blocked on an old URL structure may now apply to a different path - or nothing at all - after a migration. Running a quick audit of your robots.txt after any significant URL change takes five minutes and catches issues that are otherwise difficult to trace.
Query parameter sprawl. E-commerce and content sites often generate thousands of parameterized URLs from filters, sorting options, and session tokens. Blocking these paths in robots.txt reduces duplicate content in the crawl queue and can meaningfully improve how quickly Googlebot reaches new pages. A rule like Disallow: /*?sort= keeps sorting variants out of the crawl without touching the canonical product pages themselves.
Blocking pages you also want to deindex. If you disallow a URL and it has existing inbound links, Google may keep it indexed indefinitely because it cannot read the noindex tag. This is a common situation that traps URLs in the index permanently. The correct fix is to allow crawling and use noindex rather than blocking the path.
Photo by Testing Before You Deploy
Never push a robots.txt change without validating it against real paths first.
Before deployment, use a dedicated validator. The EvvyTools Robots.txt Generator builds rules visually with templates for WordPress, Next.js, Shopify, and Laravel. It validates rules as you write them, flags common mistakes, and includes a URL tester where you enter a specific path and see exactly what your current rules allow or block. You can also paste in an existing file to audit what you already have.
After deployment, Google Search Console includes a URL Inspection tool that shows Googlebot's current access status for any path. Run it on a few representative URLs the day after a change goes live to confirm nothing was accidentally restricted.
"The most damaging robots.txt mistakes I see in technical SEO audits aren't the obvious ones - they're accidental wildcard patterns that quietly block a large share of a site's product pages without anyone noticing until the traffic drops." - Dennis Traina, founder of 137Foundry
Platform-Specific Configurations
WordPress. The minimal correct setup for most WordPress sites:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap_index.xml
The Allow rule for admin-ajax.php is required by many plugins and themes. Without it, AJAX-dependent frontend functionality can break silently.
Next.js. For App Router or static exports, the /_next/static/ path must remain crawlable. Blocking it prevents Googlebot from loading the JavaScript needed to render client-side content:
User-agent: *
Disallow: /api/
Allow: /_next/static/
Sitemap: https://yourdomain.com/sitemap.xml
If your API routes don't expose sensitive data, removing Disallow: /api/ is harmless. Crawlers won't index JSON responses, but blocking them doesn't provide security either.
Shopify. Standard Shopify plans auto-generate a robots.txt file that cannot be edited through the admin. Shopify Plus supports Liquid-based customization via a robots.txt.liquid template. For standard plans, the practical approach is to manage crawling preferences through canonical tags and noindex meta tags rather than the auto-generated file.
Photo by What Google Does With Your File
Google caches robots.txt for up to 24 hours and re-fetches it on its own schedule. Changes are not instantaneous - a newly blocked path may still be crawled for a day or two while cached rules remain in effect, and a newly unblocked path may not be crawled until the cache refreshes.
Google also imposes a 500 KB file size limit. Content past that limit is ignored. Any robots.txt file approaching 500 KB is almost certainly using rules incorrectly. A correct production configuration for most sites fits in under 30 lines.
The crawling protocol itself is now formalized. The Robots Exclusion Protocol was published as RFC 9309 in September 2022, defining exactly how Googlebot and Bingbot parse rules, resolve conflicts, and handle extended pattern syntax. For debugging crawler-specific parsing issues, that document is the authoritative reference.
For additional technical SEO and developer utilities, the EvvyTools tools directory includes free browser-based tools for sitemap generation, schema markup, meta tag analysis, and other common tasks without requiring any installation.
Writing a Minimal, Correct robots.txt
A good robots.txt is short, specific, and tested before deployment. Start with what you know needs restricting - admin areas, staging environments, internal utilities. Add a Sitemap line. Test every rule against real paths using the EvvyTools Robots.txt Generator before pushing the file live.
If parameterized URLs or faceted navigation are generating large volumes of thin or duplicate pages, handle that through canonical tags and noindex meta tags. They give finer control without the risk of accidentally blocking paths that should be crawled.
Keep the file short, keep it tested, and revisit it any time the URL structure changes. The Google developer documentation covers crawl budget, sitemaps, and indexing signals in depth for anyone who wants to go further with crawl management.
That is the complete playbook for a robots.txt configuration that does exactly what you intend and nothing it doesn't.
Photo by pixelcreatures on Pixabay
