Clever Marketing The most damaging robots.txt mistakes are not exotic syntax errors — they are five ordinary lines that either block nothing, block everything, or block the wrong layer entirely. Robots.txt controls crawling, not indexing, it is case-sensitive, and a single character (/ after Disallow, a misplaced *, a BOM at byte zero) changes what an entire site exposes to search engines and AI crawlers. Here are the mistakes that do real damage, ordered by how quietly they do it.
Mistake 1: Using Disallow to deindex a page
Disallow stops compliant bots from fetching a URL — it does not remove it from the index. A blocked URL that has external links can be indexed anyway and shows up in Google as a bare link with the note “No information is available for this page.” Worse, blocking a page you want deindexed is self-defeating: Google cannot see the noindex tag on a page it is forbidden to crawl, so the block actively preserves the indexed entry. The correct sequence to deindex is the opposite: leave the URL crawlable, add noindex (meta robots or X-Robots-Tag header), wait for a recrawl, and only optionally block it afterwards. Since 2019, Noindex: as a robots.txt directive is not supported at all — Google formally dropped it — so any robots.txt file still carrying Noindex lines is enforcing nothing.
Mistake 2: Blocking CSS, JS, and API endpoints that rendering depends on
Google renders pages with a headless Chromium, and rendering needs the same resources a browser does. A robots.txt that blocks /assets/, /static/, /wp-includes/, or internal API routes used for client-side data fetching leaves Googlebot with broken layouts or empty content — and pages get evaluated in that state. The mobile-friendliness and layout signals suffer first; on JS-heavy sites, the indexed content itself can be incomplete. The same failure mode now extends further: most AI crawlers (GPTBot, ClaudeBot, PerplexityBot) do not execute JavaScript at all, so anything that depends on blocked or client-side resources is invisible to them from the start. Audit rule of thumb: no Disallow should ever cover a path that appears in your rendered HTML as a script, link, or fetch dependency.
Mistake 3: Wildcard and matching errors — *, $, case, and the trailing slash
Robots.txt matching is literal, and four details catch even experienced teams. Case: paths are case-sensitive, so Disallow: /Admin/ leaves /admin/ open. Trailing slash: Disallow: /private blocks /private, /private/, and /private-offer.html (prefix match), while Disallow: /private/ blocks only the directory — the shorter rule often over-blocks by accident. $ anchor: Disallow: /*.pdf$ blocks URLs ending in .pdf, but not /file.pdf?download=1 — the query string defeats the anchor. Precedence: for Google, the most specific (longest) matching rule wins, and on a tie Allow beats Disallow — but not every crawler resolves conflicts the same way, which is exactly the kind of divergence worth testing per user agent rather than assuming.
Mistake 4: The staging block that ships to production
User-agent: * / Disallow: / is correct on a staging environment and catastrophic in production — and it migrates in deploys more often than anyone admits, because robots.txt sits in the codebase like any other static file. The symptom is a slow bleed, not an outage: crawling stops, rankings decay over days to weeks, and nothing errors anywhere. Two cheap safeguards: serve robots.txt dynamically per environment (or generate it at build time from an environment variable), and put an automated check on the production file — even a simple uptime monitor asserting the response does not contain Disallow: / as a site-wide rule. The reverse migration is quieter still: a permissive dev robots.txt shipped to staging gets test environments indexed, which is how internal URLs end up in search results.
Mistake 5: Misunderstanding what HTTP status codes do to robots.txt
The counterintuitive part: a missing robots.txt is completely safe — 4xx responses are treated as “no restrictions apply” and crawling proceeds normally. A failing one is not: when robots.txt returns a 5xx, Google initially interprets it as the site asking not to be crawled and pauses crawling; if the error persists, it falls back to the last cached copy for a period before eventually treating the file as unavailable. A misconfigured WAF or bot-protection rule that serves errors or challenges on /robots.txt can therefore suspend crawling of a perfectly healthy site — the same mechanism we have seen in the wild where firewall rules silently blocked specific crawlers. While auditing, also check the file’s size and encoding: Google reads at most 500 KiB, and a UTF-8 BOM before User-agent can invalidate the first group of rules.
How do you verify what your robots.txt actually does?
Reading the file is not verification — matching is. Test the exact URL against the exact user agent, because behavior differs per bot: Google ignores crawl-delay, other crawlers respect it; precedence rules vary; and AI crawlers bring their own user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) that your rules may or may not cover intentionally. Our Robots.txt Tester does exactly this: paste your robots.txt, enter a URL, and check the verdict per user agent — including presets for the AI crawlers most robots.txt files were never written with in mind.