Google has rewritten its 'Optimize your crawl budget' documentation. The update tidies up wording and terminology, but it also adds several real clarifications about how Googlebot decides how much to crawl. The headline change: every site now starts on the same default, conservative crawl-capacity limit, and Google's systems raise that limit over time only if there is crawl demand and the site stays healthy. Search Engine Roundtable flagged the revision after Google refreshed the page.
If you run a large site and worry about whether Google is fetching all of your pages, this is worth a read. If you run a small site, the short answer is that crawl budget still is not something you need to manage.
Every site now starts on the same conservative limit
The clearest new detail is about how crawl capacity begins. Google says every site starts with the same default, conservative crawl-capacity limit. From there, Google's systems automatically raise the limit when two conditions hold: there is demand to crawl more of the site, and the site keeps responding well without signs of strain.
The practical takeaway is that a new or newly expanded site will not be crawled at full speed on day one. Capacity is earned. A server that answers quickly and reliably lets Google lift the ceiling; a server that slows down or throws errors keeps the ceiling low.
Crawl capacity is shared across all Google crawlers
Another clarification that carries weight: crawl capacity is shared across all of Google's crawlers, not split neatly per bot. So if one crawler is pulling heavily from your site, that demand can reduce the capacity available to the others.
This matters more than it first sounds. Google runs many crawlers beyond the standard Googlebot for search, and they draw from the same pool. On top of that, the wider point applies to your server as a whole: anything hammering it, including aggressive AI and third-party crawlers, competes for the same response headroom. If your origin is busy serving other bots, Googlebot sees a slower, busier server and holds its rate down.
Faster responses and 304 caching help crawl efficiency
The revised doc leans on two efficiency levers you already control.
Server response speed
Faster server responses let Google fetch more in the same window. Slow responses are read as a signal to back off, which caps how much of a large site gets crawled in a given period.
HTTP caching with 304
Google recommends proper HTTP caching so that unchanged pages return a 304 Not Modified response instead of the full page. When Googlebot sends a conditional request and your server answers 304, it skips re-downloading content that has not changed. That frees capacity to crawl pages that actually did change. For a site with a large, mostly-stable page count, correct 304 handling is one of the more direct ways to stretch the same crawl budget further.
Who this actually affects
Crawl budget is a large-site concern. Google has said for years that most sites do not need to think about it. This update does not change that.
- Large sites with hundreds of thousands of URLs, or sites that generate many pages, are where crawl budget can genuinely limit how much gets discovered and refreshed.
- Frequently changing sites such as big news publishers and marketplaces, where fresh crawling matters for getting new pages indexed quickly.
- Small and mid-size sites generally will not hit the limit. Publishing good pages, keeping the server healthy, and maintaining a clean sitemap is enough.
| Signal | Effect on crawl capacity |
|---|---|
| Fast, stable server responses | Google raises the limit over time |
| Slow responses or errors | Limit stays low or drops |
| Heavy load from other crawlers | Less capacity left for Googlebot |
| Correct 304 on unchanged pages | Crawl budget spent on changed pages |
What this means for Thai marketers
For most Thai business sites, nothing here is urgent. The pages that matter get crawled, and effort is better spent on content and links than on crawl mechanics.
The picture changes for large Thai e-commerce catalogues, classifieds, and high-volume publishers. Sites with tens or hundreds of thousands of product or article URLs are exactly where a conservative starting limit and shared capacity can slow full indexation. The shared-capacity note is the one to watch: if AI and other third-party crawlers are hitting your servers hard, they can eat into the headroom Googlebot needs. For those sites, server speed and correct 304 caching stop being nice-to-haves and become part of getting the full catalogue indexed. This is worth checking during a technical SEO audit, and worth building into decisions about hosting and site architecture before a catalogue scales.
FAQ
Is this a ranking update?
No. It is a documentation revision plus clarifications about crawling. Crawl budget affects how much of your site gets fetched, not how individual pages rank.
Do small sites need to do anything?
No. Crawl budget is a large-site concern. A healthy server and a clean sitemap cover it.
What is the fastest win for a large site?
Faster server responses and correct HTTP caching so unchanged pages return 304. Both let the same crawl budget cover more of your changed pages.
The revision does not hand large sites a new lever so much as it explains the ones that already existed. If your site is big enough for crawl budget to bite, the fixes are server performance and caching discipline. If you want a clear view of where crawl efficiency is leaking on a large catalogue, our team can help through a focused SEO service in Thailand.
