TL;DR
- Google's Search Central blog said on 11 November 2026 that crawler traffic to an origin server can spike right after a site moves behind a CDN.
- The cause is hostload: faster CDN edge responses raise Google's crawl capacity limit, while a new CDN's cold cache sends every first request for a URL back to the origin.
- Blanket bot protection such as CAPTCHA blocks Google's crawlers; Google says pages and shopping listings left behind it will begin to drop out of Search.
- Google's fixes: allow verified bots, pre-warm caches before cutover, and use 429 or 503 only for a few hours or 1-2 days at most, never 403 walls.
Moving a website behind a content delivery network can make Google crawl it harder for a while, not less. In a Google Search Central blog post published on Wednesday 11 November 2026, Google explained that fast CDN edge responses tell its crawlers the site can handle more requests, while a brand-new CDN cache is still empty, so many of those extra requests land on the origin server anyway. The same post warns that switching on blanket bot protection to stop the surge can block Googlebot outright, and that pages and shopping listings left behind a challenge page will begin to drop out of Search.
What Google published on 11 November 2026
The post is titled "Moving to a CDN without overwhelming your origin (or blocking Googlebot)" and was posted by Cherry and Gary on the Google Search Central Blog. It accompanies Episode 5 of The Non-Canonical Cabinet, the Google video series, and Google calls that episode the Season 1 finale. The tone is light (a rubber chicken stands in for an edge server in the video), but the substance is a plain description of how Google's crawling systems react to an infrastructure change.
Google opens with a complaint it says it hears from site owners: right after putting a site behind a CDN, crawler traffic to their own server spikes almost immediately, which is exactly what they were trying to fix. The post then explains why it happens, why the obvious reaction makes things worse, and what to do instead. It points readers to two existing Google documents for background, the crawl budget guide and the guide to reducing Google's crawl rate in an emergency.
Why crawling speeds up after a CDN switch
The explanation rests on what Google calls the crawl capacity limit, also known as hostload. Google's crawling infrastructure keeps monitoring how healthy and responsive a host is. According to the post, slower and slower responses decrease the crawl rate, because Google does not want to overwhelm a server or degrade the experience for its human visitors. Fast and stable responses work the other way. Google reads them as a sign of healthy, capable infrastructure, and, provided there is crawl demand, the crawl capacity limit gradually goes up.
A CDN changes those signals in one step. When DNS starts pointing at a fast edge network, connection times and response times drop sharply. In Google's words, "it looks like your server just got a massive hardware upgrade", so its crawlers start ramping up their request rate to match. Nothing about the site's content has changed. The measurement the crawler relies on has.
Two qualifiers in the post are worth holding on to. First, the increase depends on crawl demand: faster responses raise the ceiling, but they do not create interest in URLs Google has no reason to fetch. Second, Google says its crawlers do not ramp up their crawl rate immediately, and many CDNs include built-in connection pacing that smooths out traffic spikes. The surge is real, but it is gradual.
The cold cache problem: the origin still does the work
A faster crawl would be harmless if the CDN answered every request from its edge. At first it cannot. Google stresses that a brand-new CDN starts with an empty, or cold, cache. The first request for any URL at an edge location is a cache miss: the edge server has to go back to the origin, fetch the response, store a cacheable copy and only then serve the client. That applies to HTML documents, CSS, JavaScript and images alike. Only later requests for the same cached URL become cache hits that never touch the origin.
Here people and crawlers behave very differently. Google notes that human visitors usually cluster around a relatively small set of popular pages, such as the homepage, top articles or active promotions, and those warm up in the cache quickly. Crawlers do the opposite. Their job is to explore the breadth of a site's URL inventory, so a large share of what they request has never been cached at that edge.
Put the two behaviours together and the spike makes sense. Fast edge responses tell Google it is safe to crawl faster, and every uncached URL that the faster crawl reaches passes straight through to the origin. Google's own summary is that until the edge caches are populated, the CDN is not acting as a shield yet, and the origin server is still doing the heavy lifting. For a large product catalogue, filtered category pages or a deep article archive, that can mean a lot of origin traffic in the first days after cutover.
How blanket bot protection pushes pages out of Search
The second half of the post deals with the reaction Google expects from a nervous admin. Seeing automated requests climb in the origin logs right after a migration, it is tempting to log into the CDN dashboard and turn on blanket bot protection. Google's objection sits in the word "blanket": without exemptions, these rules treat all automated traffic the same, including the crawlers a site depends on.
When a CDN or web application firewall (WAF) flags automated requests and answers them with a challenge such as a CAPTCHA, Google's crawlers fail the test. The post says this leaves crawlers stuck on an error page that is not useful for Search, or for other products that rely on crawled data. From Google's side, the site appears deliberately restricted or broken. If the setting stays in place, Google says its crawlers will back off, and the site's pages and shopping listings will begin to drop out of Search.
Retailers should note that last detail. The warning is not limited to organic web results. Shopping listings are named explicitly, so a firewall rule meant to calm server load can also cut into product visibility that a store depends on for sales.
Google's recommendations, point by point
Google closes with a short game plan. The table below sets out each recommendation as the post states it, next to the detail Google attaches to it.
| Google's recommendation | Detail given in the 11 November 2026 post |
|---|---|
| If you need to block bots, allow verified bots | Use the CDN or WAF toggle or managed rule for known or verified bots; Google's crawlers can be identified via its published IP ranges. Or target specific unwanted bots with robots.txt or rules. |
| Pre-warm caches before switching traffic | Fetch the highest-traffic pages, upcoming peak-traffic pages such as promotional or seasonal launches, and the most frequently crawled URLs through the CDN at a controlled pace. |
| Use 429 or 503 as a temporary brake | Return 429 (Too Many Requests) or 503 (Service Unavailable) for a short window, "a few hours, or 1-2 days at most". |
| Do not put 403 walls in front of crawlers | Do not let blanket WAF or CAPTCHA rules serve 403 walls or challenge pages to Google's crawlers. |
The order matters. Pre-warming deals with the cause, because a warm cache absorbs crawler requests that would otherwise reach the origin. Verified-bot exemptions stop whatever protection you do need from catching Googlebot. The 429 and 503 codes are the emergency option, and Google frames them as a short brake documented in its guide to urgently reducing crawl rate, not as a permanent setting. A 403 or a challenge page sends a different message: to a crawler, access looks refused.
Google also suggests a lighter alternative to blanket blocking. Rather than stopping every automated client, a site can configure robots.txt or write rules that target the specific unwanted bots it actually has a problem with. That keeps the crawlers a business wants while filtering the ones it does not.
What the post did not say
Several things are missing from the post, and it helps to be clear about them before acting on it. Google gives no figure for how large a crawl spike typically is or how long it lasts. It does not name any CDN or firewall vendor, and it does not describe dashboard settings beyond the general known or verified bots toggle that it says most major CDNs and WAFs offer. It does not say how quickly pages that dropped out because of a challenge page return once the block is lifted. And it does not claim that a CDN move changes rankings in itself; the risks it describes are origin overload and blocked crawlers.
The post also does not tell anyone to avoid CDNs. It opens by calling a CDN a major upgrade for any growing site, because it lowers visitor latency and shields the origin from heavy traffic. The advice is about the transition, not the destination.
A practical checklist before and after cutover
The steps below turn Google's recommendations into checks a site team can run. They come from the post and from standard Google Search Console features, not from any new data.
- Before changing DNS, list the URLs to pre-warm: top landing pages, the pages for the next sale or seasonal launch, and the URLs Googlebot requests most often, which server logs or the Crawl Stats report in Search Console can show.
- Fetch that list through the CDN at a controlled pace, so the first cache misses happen on your schedule rather than the crawler's.
- Review every bot-management, rate-limit and challenge rule. Confirm verified bots are allowed, and check against Google's published IP ranges rather than trusting user-agent strings, which anyone can fake.
- For the first days after cutover, watch the origin logs and the host status and average response time figures in the Crawl Stats report.
- If the origin genuinely struggles, return 429 or 503 to crawlers for hours rather than weeks, and remove the rule once caches are warm.
- Spot-check important URLs, including product pages, with the URL Inspection tool to confirm Google receives the real page and not a challenge screen.
Teams that want this reviewed alongside the rest of the site's crawl and indexing health can fold it into a technical SEO audit ahead of the migration date. The same checks are useful after any change to a firewall vendor or security plan, since a new default rule can block crawlers just as easily as a new CDN.
What this means for Thai marketers
For Thai online stores and brand sites that add a CDN or firewall shortly before a big sales period such as 11.11, 12.12 or year-end promotions, Google's post describes the risk of doing that in a hurry. The cache is cold exactly when the promotion pages need it, and a panic switch to strict bot protection can hide product pages and shopping listings from Google during the weeks that matter most. Pre-warm the promotion URLs, keep verified crawlers exempt and treat 429 or 503 as a short emergency measure. For stores whose revenue depends on organic and Shopping visibility, this belongs in the launch plan next to e-commerce marketing work, and in ongoing SEO in Thailand monitoring once the CDN is live.
FAQ
Why does Googlebot crawl more after I move to a CDN?
Googlebot crawls more because faster, stable responses from the CDN edge raise Google's crawl capacity limit, also called hostload. Google says those faster responses look like a massive hardware upgrade to its systems, and if there is crawl demand, its crawlers gradually ramp up their request rate to match.
Can turning on bot protection in my CDN or firewall hurt my SEO?
Yes, blanket bot protection can hurt search visibility if it challenges Google's crawlers. Google says its crawlers fail CAPTCHA-style challenges, back off if the block stays in place, and that pages and shopping listings then begin to drop out of Search. It recommends allowing verified bots instead.
Should I return 429 or 503 to slow Google down during a migration?
Only for a short window, and only if the origin is genuinely under critical load. Google's post says a few hours, or 1-2 days at most, following its guide to urgently reducing crawl rate, and it warns against using 403 walls or challenge pages for the same purpose.
Why are my origin logs busy when the CDN is supposed to protect the server?
Because a new CDN starts with a cold cache, so the first request for every URL is a cache miss that goes back to the origin. People mostly visit a few popular pages that warm up fast, while crawlers request the whole URL inventory, so the origin carries the load until the caches fill.
Does this guidance apply to sites in Thailand?
Yes, the post describes how Google's crawling systems behave in general and is not limited to any country. It does not mention specific markets, vendors or rollout dates, because it explains existing crawler behaviour rather than announcing a new feature.
A CDN is still a sensible upgrade for most growing sites. The practical message from Google's 11 November post is to plan the switch: warm the cache first, exempt verified crawlers and keep any emergency brake short. If a migration is scheduled before a sales peak, Relevant Audience can review the crawl and indexing risks with your team before the DNS change goes live.
