If your robots.txt has a section for user-agent: Googlebot and a separate section for user-agent: *, Google reads only the Googlebot section. Any disallow that lives under the wildcard is invisible to it. John Mueller explained this in a discussion reported by Search Engine Journal on 25 July 2026, after a store owner asked why search-generated spam URLs kept getting indexed despite a disallow rule that was, on paper, right there in the file.
Mueller said that with robots.txt the more specific rules win, so if a Googlebot section exists, Google will only use that section. His advice was to copy all rules into the specific user-agent section, or to consolidate multiple user-agents under shared rules rather than maintaining separate blocks.
This is not a change
Nothing about how Google handles robots.txt changed on 25 July. Group selection by the most specific matching user-agent is how the file has always worked, and Mueller was restating it in response to a case where it had bitten someone. That matters for how you talk about it internally: this is a configuration bug to look for, not a new rule to comply with.
It is a quiet failure mode. The file parses. Testing tools report the file as valid. The disallow is visible when a human opens the file and reads it top to bottom, which is exactly what a human does. Googlebot does not read the file top to bottom looking for anything that applies. It picks one group and follows it.
The pattern that breaks
Written generically, a broken file looks like this, with each line shown separately:
- User-agent: *
- Disallow: /search
- Disallow: /cart
- User-agent: Googlebot
- Disallow: /admin
Googlebot uses the second group only. It sees one instruction, and it crawls /search and /cart freely, because those disallows were written for everyone else.
Two ways to fix it
The first fix is to delete the specific block and keep a single set of rules for everyone:
- User-agent: *
- Disallow: /search
- Disallow: /cart
- Disallow: /admin
The second is to keep the Googlebot block and repeat every shared rule inside it, so the group Google actually reads is complete:
- User-agent: Googlebot
- Disallow: /search
- Disallow: /cart
- Disallow: /admin
| Setup | What Googlebot uses |
|---|---|
| Wildcard group only | The wildcard group, in full |
| Wildcard group plus a Googlebot group | The Googlebot group only. Wildcard rules are ignored |
| Googlebot group containing every shared rule | The Googlebot group, which now covers everything |
How sites end up here by accident
Almost nobody writes this pattern on purpose. It accumulates. Someone adds a Googlebot block years ago to set a crawl instruction for Google specifically. Later, a different person adds a disallow for a problem URL pattern and puts it in the wildcard group at the top of the file, because that is where the other disallows are. Both edits look correct in isolation. Together they produce a file where Google ignores the newest and most important rule in it.
The same accident happens with generated files. Platforms, plugins and CDN layers all append to robots.txt, and they do not agree about which group to append to. On a store built from a template plus three apps, the file can end up with several user-agent groups nobody planned.
What this means for Thai marketers
This is a five-minute check with a real chance of finding something, which is a rare combination. Open the robots.txt of every site you look after and count the user-agent lines. One group, no problem. More than one, read the Googlebot group and ask whether it contains every disallow you believe is in force.
The case reported by Search Engine Journal involved a Shopify store, and that is worth noting rather than dismissing. Hosted platforms generate search and filter URLs in large numbers, and those URLs are what most disallow rules are written to contain. If the disallow is sitting in the wrong group, the crawler is working through parameter URLs while the crawl budget that should be spent on product and category pages goes elsewhere. Anyone maintaining Shopify SEO on a large catalogue is a candidate for this bug.
The wider lesson is about verification. A rule that exists in a file is not a rule that is in force. Robots.txt is one of a handful of small files where a silent misconfiguration can run for a year before anyone reads the crawl stats closely enough to notice.
Questions people are asking
Does Google ignore robots.txt?
No. It follows the group that matches it most specifically. When a Googlebot group is present, that group is the one it follows, and the wildcard group is not applied on top of it.
Should I just delete my Googlebot section?
If it does not contain anything Google-specific that you still need, consolidating into one group is the simpler file to maintain. Mueller mentioned both consolidation and copying the rules across as options.
How do I tell whether this affects my site?
Count the user-agent lines in robots.txt. If Googlebot has its own group, compare that group line by line against the wildcard group and look for disallows that appear in one and not the other.
Does this apply to other crawlers?
The same most-specific-group logic is how the file is designed to work generally, which is why a Bingbot or AdsBot group has the same property.
The short version
A Googlebot section in robots.txt is not an addition to your wildcard rules. It replaces them for Google. If your file has both, every disallow you care about needs to exist inside the Googlebot group, or the group needs to go. It is worth adding to whatever list you work through during an SEO review, since the check costs a few minutes and the failure is invisible until something you meant to block turns up in the index.
