TL;DR
- Gary Illyes of Google told the Odys Podcast that AI is behind rising Googlebot impersonation; Search Engine Roundtable reported the interview on 30 October 2026.
- Illyes said anyone can fake a Googlebot user-agent in about 30 seconds, and his honeypots logged Googlebot-claiming requests from IP addresses unrelated to Google.
- Google's verification page, updated 20 March 2026, pairs a reverse DNS lookup to googlebot.com, google.com or googleusercontent.com with a matching forward lookup, or IP checks against JSON range files.
- Illyes called robots.txt a suggestion, pointed to WAFs for enforcement, discussed Web Bot Auth as a way to make verification more reliable, and is exploring crawler reputation scores as an early-stage idea.
Google's Gary Illyes says AI has made Googlebot impersonation a bigger problem, because companies collecting content to train models pretend to be Googlebot to get past sites that would otherwise block them. He made the point in an interview with Aneta Samkoff on the Odys Podcast, which Search Engine Roundtable summarised on 30 October 2026. His advice is not to trust a Googlebot user-agent string and to check the request's IP address and reverse DNS instead. That is the same method Google's own documentation sets out for verifying its crawlers.
What Gary Illyes said about bot impersonation
Bot impersonation is when a crawler claims to be someone it is not, most often by sending Googlebot's user-agent. Explaining why it has grown, Illyes said: "I think it is AI." According to Search Engine Roundtable's report by Barry Schwartz, he went on to say that "more and more companies are looking for tokens for training their own models", and that this takes content. He described how those companies see the web as "a big farm where you can just go out and harvest whatever you need."
Illyes said firewalls and CDNs have got better at identifying these crawlers but "are not perfect", and they still let through some traffic they meant to stop. His explanation of the motive was direct: a smaller AI engine, or anyone who wants to scrape a site, can say it is Googlebot and may be let in where it would otherwise be blocked. Knowing who is actually crawling, he said, lets a site owner decide for themselves who should have access.
The other points from the interview
Search Engine Roundtable listed further points from the video. The table below summarises them as that outlet reported them. They come from its write-up, not from a full transcript.
| Topic | What Illyes said, per Search Engine Roundtable |
|---|---|
| User-agent spoofing | Anyone can impersonate Googlebot in about 30 seconds by changing a user-agent header. His online honeypots saw requests claiming to be Googlebot from IP addresses unrelated to Google. |
| Server load | A badly configured bot can request URLs over and over, use up bandwidth and server resources and lock out real visitors. Much of the damage may come from ignorance or poor implementation rather than deliberate attacks. |
| robots.txt | A suggestion, not an enforceable restriction. Legitimate crawlers generally respect it, but bots can ignore it, so blocking needs network-level tools such as a web application firewall (WAF). |
| AI controls | The Google-Extended token lets a site allow Googlebot for Search while restricting certain AI uses. Apple has Applebot-Extended. He expects controls to become more granular. |
| Web Bot Auth | Cryptographically signed HTTP requests that a site checks against a public key, which could simplify verification as IP addresses get harder and more expensive to manage. |
| Reputation scores | An early-stage idea he is exploring for rating how crawlers behave. He cautioned that a system controlled by one company could introduce bias. |
| Small sites | Check whether bot traffic is actually causing a problem before investing in protection. A CDN with bot management is a practical start, and hosted platforms such as Shopify already provide protection. |
How Google says to verify Googlebot
Google's documentation page "Verify requests from Google crawlers and fetchers", last updated on 20 March 2026, describes two methods. The manual method uses command-line DNS lookups and, in Google's words, "is sufficient for most use cases." The automatic method matches the visiting IP address against IP range lists that Google publishes as JSON files, and is meant for large-scale checks.
The manual check has two steps:
- Run a reverse DNS lookup on the IP address from your server logs, for example with the host command, and confirm that the hostname returned is on googlebot.com, google.com or googleusercontent.com.
- Run a forward DNS lookup on that hostname and confirm that it returns the same IP address that appeared in the logs.
Google's first example shows what a pass looks like: the IP address 66.249.66.1 resolves to crawl-66-249-66-1.googlebot.com, and that hostname resolves back to 66.249.66.1. Both steps are needed. Google's page does not spell out why, but the reason is how DNS works: whoever controls an IP block controls its reverse DNS and can make it return a name that looks like Google's. Only the forward lookup on Google's own domain confirms that the name and the address belong together.
For the automatic route, Google publishes its ranges in CIDR format. For other Google addresses that may reach a site, such as Apps Script, the page points to Google's general list of IP addresses. A log tool or WAF rule that refreshes these files and checks every request claiming to be Googlebot against them can do at scale what the host command does for one address.
Three types of Google crawler and their hostnames
Google's page splits its crawlers into three groups. Each resolves to a different hostname pattern and is listed in a different JSON file, and they treat robots.txt differently. That matters when reading logs, because not every verified Google address is Googlebot crawling for Search.
| Type | Reverse DNS pattern and IP list | robots.txt |
|---|---|---|
| Common crawlers, such as Googlebot | crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com; common-crawlers.json | Always respected for automatic crawls |
| Special-case crawlers, such as AdsBot | rate-limited-proxy-***-***-***-***.google.com; special-crawlers.json | May or may not be respected |
| User-triggered fetchers, such as Google Site Verifier | ***-***-***-***.gae.googleusercontent.com or google-proxy-***-***-***-***.google.com; user-triggered-fetchers.json, user-triggered-fetchers-google.json and user-triggered-agents.json | Ignored, because a user requested the fetch |
One detail is easy to miss. Google says the IPs in user-triggered-fetchers.json resolve to gae.googleusercontent.com hostnames and are used, for example, when a site running on Google Cloud fetches external RSS feeds at the request of its own user. Such a request passes the googleusercontent.com check, yet it is not Googlebot. Relevant Audience's reading of Google's table is that a log entry sending a Googlebot user-agent from a gae.googleusercontent.com host deserves a closer look rather than an automatic allow. Google's page does not discuss that exact case.
Why the user-agent string proves nothing
Google's Googlebot overview page already warns that "the HTTP user-agent request header used by Googlebot is often spoofed by other crawlers." It says the best way to confirm a request is a reverse DNS lookup on the source IP, or a match against the Googlebot IP ranges. The same page notes that Googlebot Smartphone and Googlebot Desktop share one robots.txt product token, and that for most sites Googlebot "shouldn't access your site more than once every few seconds on average."
That last line gives site owners a rough yardstick. A self-declared Googlebot that hits a small site many times a second does not match Google's own description of its crawler. It is a reason to run the DNS check, not proof on its own, because Google also says the rate can appear slightly higher over short periods.
robots.txt, WAFs and blocking the right bot
Illyes's description of robots.txt as a suggestion matches how Google's documentation frames the file. Google's Googlebot page lists robots.txt for preventing crawling, noindex for preventing indexing, and "another method, such as password protection" for keeping a page away from both crawlers and users. A scraper that ignores robots.txt and fakes a Googlebot user-agent is stopped by neither of the first two. Enforcement has to happen at the server, the CDN or the firewall.
The opposite risk is blocking the real Googlebot. Google's page says blocking Googlebot affects Google Search, including Discover and all Search features, as well as Google Images, Google Video and Google News. A firewall rule that blocks on user-agent alone, or relies on an outdated copy of Google's IP lists, can cut a site out of Search. A safer rule allows verified Google ranges and challenges or blocks only the requests that fail verification.
What the sources did not say
- Search Engine Roundtable's summary gives no figures for how much fake Googlebot traffic exists or how fast it is growing.
- Illyes did not name any company that impersonates Googlebot, according to the summary.
- No date or product was reported for Google support of Web Bot Auth or crawler reputation scores. Search Engine Roundtable described the reputation idea as early-stage.
- Google's verification page does not mention AI scraping. It was last updated on 20 March 2026, months before Search Engine Roundtable reported this interview.
What this means for Thai marketers
This section is Relevant Audience analysis, not Google guidance. For a Thai business site on shared hosting, a managed WordPress host or a platform such as Shopify, Illyes's advice points to one first question: is bot traffic causing a measurable problem? Signs would be slow pages, bandwidth overages or server errors that line up with crawler spikes in the logs. If there is no such problem, a hosted platform's built-in protection may be enough for now.
For sites that do see load problems, a short check list:
- Pull a week of server logs and list the IP addresses that sent a Googlebot user-agent.
- Verify a sample with the reverse and forward DNS steps, or match all of them against common-crawlers.json.
- Block or rate-limit the addresses that fail, at the CDN or WAF rather than in robots.txt.
- Keep verified Google ranges allowed, and refresh the JSON files on a schedule so the rule follows Google's current lists.
- If AI training is the concern, use the Google-Extended token for Google's own AI uses, and remember that it does nothing against crawlers that ignore robots.txt.
Crawl access also bears on visibility in AI search. A site that blocks real Google crawlers by mistake loses Search and the features built on it, while fake crawlers use up server capacity meant for customers. Relevant Audience's technical SEO audit includes log and crawl checks, and the GEO service covers how AI crawlers and AI answers use a site's content. For wider organic work, see Relevant Audience's SEO services in Thailand.
Frequently asked questions
Why has fake Googlebot traffic increased?
Gary Illyes attributes it to AI. Speaking on the Odys Podcast, as reported by Search Engine Roundtable on 30 October 2026, he said companies gathering content to train models claim to be Googlebot so that sites which would otherwise block them let them in.
How do I check if a request is really from Googlebot?
Run a reverse DNS lookup on the IP address and confirm the hostname is on googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the same IP. For large volumes, Google says to match IP addresses against its published JSON range files, such as common-crawlers.json for Googlebot.
Can robots.txt block fake Googlebots?
No. Illyes called robots.txt a suggestion that legitimate crawlers generally follow and other bots can ignore. To block unwanted bots, he pointed to network-level protection such as a web application firewall.
Does a small website need to act on this?
Not unless bot traffic is causing a measurable problem, according to Illyes's advice as summarised by Search Engine Roundtable. He suggested a CDN with bot management as a starting point and said hosted platforms such as Shopify already provide protection.
Is Web Bot Auth available from Google now?
The sources do not say so. Illyes discussed Web Bot Auth, which uses cryptographically signed HTTP requests checked against a public key, as a way to make verification more reliable, but no Google launch date was reported.
The practical takeaway is narrow: a user-agent string is a claim, and DNS or Google's published IP lists are how to check it.
