The SEO Emergency Was a Regex Crossing a Boundary

The SEO Emergency Was a Regex Crossing a Boundary

My nightly crawler reported that one of my sites was blocking the entire site in its robots.txt. Every crawler, every page, denied. An indexing emergency, sitting at the top of the morning report above every other finding.

I opened the live file. It allowed general crawling and denied a short list of named bots. Googlebot was welcome. The site was fine.

The crawler was wrong, and the way it was wrong is worth a post.

How a regex crosses a boundary

A robots.txt is a list of groups. Each group starts with one or more User-agent lines and is followed by the rules that apply to those agents. The rules belong to the group above them and no other.

The crawler's check started at User-agent: * and scanned forward looking for Disallow: /. It found one. It was in a later group, belonging to a named AI scraper, several groups down. The regex had walked straight through two group boundaries and attributed a rule to the wrong agent.

So the report said "everyone is blocked" when the file said "this one bot is blocked". A pattern that ignores scope cannot check a scoped configuration. It will always find the most alarming line in the file and pin it to whichever group it started in.

Verify the detector before changing production to satisfy it

The dangerous move here would have been to open the robots file and start editing. The report said blocked, the file had a Disallow: / in it, and a quick fix would have been to remove that line. That would have un-blocked the scraper I had deliberately blocked, to satisfy a tool that had misread the file.

Instead we ran three checks against the same live file:

  • The old detector: reports a blanket block.
  • A corrected, group-aware detector: reports no blanket block.
  • A separate, independent robots parser: allows Googlebot, denies the scraper.

Two against one, and the one was the thing that had raised the alarm. The site's robots policy was left exactly as it was. The regex got fixed, and a regression test was added with a genuine blanket ban in it, so that the corrected detector still catches the real thing and not just the false alarm.

That last part matters. A fix that only makes the false positive go away has not been tested. You need both a known-bad file that should fail and a known-good file that should pass, or you have just moved the bug.

The next eight findings were also in the detector

Below the robots alarm, the same report listed eight groups of pages with duplicate meta descriptions. That one took longer to doubt, because duplicate descriptions are a real and common problem.

The detector respected canonical URLs when it compared titles. It did not respect them when it compared descriptions. So eight sets of pages that canonicalize to one page each were being reported as eight duplicates. Correct by the site's own declared structure, flagged by a checker that applied the structure to half its checks.

We replayed twenty fetched pages through the detector. All eight groups appeared before the fix and none after it. Distinct canonical pages with the same description still trigger, which is the actual bug the check exists to find.

Two habits

When a tool reports an emergency, verify the tool before touching production. Especially when the fix would undo a decision you made on purpose. The report is a claim, the tool is a program, and programs have bugs with exactly the same confidence as their correct output.

A scoped format needs a scoped parser. Any file with sections, groups, blocks or nesting cannot be checked by a flat pattern. The pattern will cross the boundary the first time the alarming line is in the wrong section. If you find yourself writing a regex against robots.txt, nginx config, YAML, or anything with indentation that means something, stop and parse it.

Nine findings at the top of the report. Nine bugs in the reporter. The site was fine all along.