Crawl Budget: How Google Decides What to Crawl

TL;DR

Google defines a site as one hostname, so each subdomain such as th.example.com or shop.example.com has its own separate crawl budget.
Google's rough guide: crawl budget matters for sites over 1 million pages changing weekly, sites over 10,000 pages changing daily, or many 'Discovered - currently not indexed' URLs.
noindex does not save crawling because Google must fetch the page to read the tag; robots.txt blocks the fetch, and 404 or 410 is the signal for pages gone for good.
Returning 500, 503 or 429 slows Googlebot for the whole host, but Google advises keeping it to a few hours or 1-2 days or URLs may drop from the index.
The Crawl Stats report sits under Settings in Search Console and only works on Domain or root-level URL-prefix properties.

Crawl budget is the set of URLs on one hostname that Googlebot can crawl and wants to crawl in a given period, and it is decided by two things: the crawl capacity limit (how much fetching your server can take) and crawl demand (how much Google wants to fetch). Most sites never run short of it; Google's own guidance aims the topic at very large sites, sites whose content changes daily, and sites with a big pile of URLs stuck as "Discovered - currently not indexed" in Search Console.

This guide walks through how Google describes the two halves of crawl budget, how to tell whether your site is one of the few that should care, how to read the Crawl Stats report, what quietly wastes crawling, and which controls (robots.txt, noindex, status codes, sitemaps) actually change Googlebot's behaviour. Everything here follows Google's crawling documentation as it reads in October 2026.

What crawl budget actually is

Google treats a "site" as a single hostname. In practice, www.example.com, shop.example.com and blog.example.com each get their own crawl budget. Moving a section to a subdomain does not borrow budget from the main host.

Within one hostname, Google combines two separate ideas.

Crawl capacity limit

The crawl capacity limit (Google also calls it hostload) is about your server. It limits the total time your server spends holding connections open for Google, so that crawling does not overload the site. Two things move it:

Crawl health. If response times stay stable or improve, including latency, the limit goes up. If the site slows down, returns 5xx server errors, or sends rate-limiting signals such as HTTP 429, the limit comes down.
Google's own limits. Google has finite crawling resources and spreads them across the whole web, so there is a ceiling regardless of how fast your server is.

Google's documentation now states that every site starts on the same default, conservative capacity limit. The limit rises automatically over time only when there is demand to crawl more and the site keeps responding well. The documentation also says the capacity is shared by all of Google's crawlers on that host, so heavy fetching by one of them (the doc gives AdsBot as an example) leaves less room for the others.

Crawl demand

Crawl demand is about how much Google wants to fetch. For Googlebot it depends on site size, how often content changes, page quality and relevance. The parts you can influence most are:

Perceived inventory. Every URL Google knows about, including duplicates and URLs you never meant to expose. If Google thinks your site has ten million URLs when only two hundred thousand matter, it spends time on the wrong ones.
Popularity. URLs that are more popular tend to be crawled more often.
Staleness. Google recrawls to pick up changes, so pages that change get revisited.

Site-wide events such as a domain move can raise demand for a while as Google re-fetches everything at the new address.

Put together, crawl budget is the overlap: URLs Google is able to crawl given the capacity limit, and wants to crawl given demand. A fast server does not create crawling on its own. If demand is low, Google crawls less even when the capacity limit is nowhere near reached.

Which sites actually need to care

Google's crawl budget guide describes itself as written for advanced cases and gives rough sizes, stressing that they are estimates and not exact thresholds:

Large sites with more than 1 million unique pages whose content changes moderately often, around once a week.
Medium or larger sites with more than 10,000 unique pages whose content changes very rapidly, daily.
Sites where a large share of URLs sit in the Page Indexing report as "Discovered - currently not indexed".

If none of those describes you, or if your new pages tend to be crawled the same day you publish them, Google's advice is simply to keep your sitemap current and check the Page Indexing report from time to time. A 300-page company site or a 2,000-product shop with a sensible URL structure is not going to run out of crawl budget, and time spent "optimising" it would be better spent on content and links.

The sites that do hit limits tend to share a pattern: a modest number of real pages multiplied into a huge number of crawlable URLs by filters, sort orders, session parameters, internal search results or calendar pages. An online store with 40,000 products can expose millions of URLs if every combination of colour, size, brand, price band and sort order gets its own crawlable link.

How to read the Crawl Stats report

The Crawl Stats report lives under Settings (the gear icon) in Search Console, not in the main menu. It is only available for root-level properties: a Domain property, or a URL-prefix property at the root such as https://example.com. A property for a subfolder will not show it.

The top chart has three lines:

Total crawl requests: every request Google made to URLs on your site in the period, successful or not.
Total download size: bytes downloaded while crawling.
Average response time: the mean response time across everything fetched, with each page resource counted separately.

Below that are four breakdowns. Read them in this order:

By response. Most responses should be 200. A rising share of 301/302, 404 or 5xx tells you where requests are going that do not produce indexable content.
By purpose. "Discovery" is a URL Google has never crawled; "Refresh" is a recrawl of a known URL. On a site that publishes new pages every day, a very low discovery share can mean Google is busy recrawling old URLs instead of finding new ones.
By file type. HTML, image, JavaScript, CSS, PDF and others, measured as a share of responses. A site where JavaScript and JSON requests dwarf HTML may be spending capacity on resources rather than pages.
By Googlebot type. Smartphone, desktop, image, video, page resource load, AdsBot, StoreBot and other agents. Because capacity is shared, a big AdsBot share on a store running many ads is worth noticing.

The host status panel covers robots.txt fetching, DNS resolution and server connectivity over the past 90 days, with three states: no significant availability issues, some issues but not recently, and a recent issue (within the last week). Each category chart has a red threshold line; Google's example is DNS failures above 5% of requests on a day. You want a green status. If robots.txt cannot be fetched (a 429 or 5xx response), Google slows or stops crawling until it gets an acceptable robots.txt response, so an unreliable robots.txt is one of the fastest ways to lose crawling across the whole host.

Click into any row of a breakdown to see example URLs. Those samples are where you find the parameter patterns, redirect hops and soft 404s described below.

What wastes crawl budget

Waste is anything that makes Google fetch URLs that will never become useful search results. The usual offenders:

Filter and sort links create a near-endless supply of URLs that show the same products in a different order. Each one is a fetch. Google's guidance is to consolidate duplicate content and block URLs you never want crawled. For facets, that usually means deciding which filter combinations deserve to be landing pages (say, "running shoes / women / Nike") and keeping the rest out of crawlable links or blocked in robots.txt.

Redirect chains

Google lists long redirect chains as something that hurts crawling. A link that goes http to https, then to www, then to a trailing-slash version, then to a renamed URL costs four fetches to reach one page. Update internal links to point straight at the final URL and collapse chains to a single hop.

Soft 404s

A soft 404 is a page that says "not found" or "out of stock, nothing here" but returns HTTP 200. Google says these keep getting crawled and waste budget. The Page Indexing report lists them. Return a real 404 or 410 for pages that are gone, and make sure empty category or search pages do not answer 200 with a blank template.

Duplicates and near-duplicates

Printer versions, tracking parameters, mixed-case URLs, and the same product reachable under several category paths all inflate perceived inventory. Canonical tags help Google choose which version to index, but Google may still crawl the duplicates it can find, so the cleaner fix is to stop linking to them.

Slow pages and heavy resources

Because the capacity limit is measured in connection time, slow responses mean fewer fetches in the same window. Google recommends improving server response time and supporting 304 (Not Modified) responses, so that unchanged content is not downloaded again.

Robots.txt vs noindex for crawl control

The two tools do different jobs:

robots.txt stops Google from crawling a URL. Google notes that blocking also significantly lowers the chance the URLs are processed by other Google systems. It is the right tool for URLs you never want crawled, such as internal search results or infinite filter combinations.
noindex does not save crawling. Google has to request the page to see the tag, then drops it from the index. Using noindex to manage crawl budget wastes the very fetches you were trying to save.

Two further points from Google's guide are easy to miss. First, do not use robots.txt to temporarily "free up" budget for other pages: Google only shifts budget that way if the site is already hitting its capacity limit. Second, do not use robots.txt for pages that are permanently gone. A blocked URL stays in the crawl queue longer and is crawled again once the block is lifted; a 404 or 410 is the clean signal.

And if a server is struggling right now, Google's separate guidance on reducing crawl rate says to return 500, 503 or 429 for a short period (a couple of hours, or one to two days at most); with 503 or 429 you can add a Retry-After header. Keep it longer and URLs that keep returning those codes may be dropped from the index. The slowdown applies to the whole hostname and recovers automatically once errors stop.

Sitemaps and lastmod

An XML sitemap does not raise your capacity limit, but it shapes demand. Google asks site owners to keep sitemaps current, list all the content they want crawled, and include the lastmod tag for updated content. Two rules make lastmod worth trusting:

Set it only when the main content changes, not on every page build or when a sidebar widget refreshes. A lastmod that changes daily on every URL teaches Google to ignore it.
Keep the sitemap free of redirecting, noindexed, blocked or 404 URLs. A sitemap is a list of canonical, indexable pages; anything else is noise in the queue.

For large sites, split sitemaps by section or content type (products, categories, articles). Search Console then reports indexing per file, which shows you which section has the "Discovered - currently not indexed" problem.

Crawl budget problems and fixes at a glance

Crawl budget problems and fixes at a glance
Symptom	Likely cause	Fix per Google's guidance
Millions of parameter URLs in Crawl Stats samples	Faceted navigation and sort links	Consolidate duplicates; block never-wanted URLs in robots.txt
Many 301/302 responses	Redirect chains in internal links	Link to the final URL; collapse chains to one hop
Soft 404 entries in Page Indexing	Empty pages answering 200	Return 404 or 410 for removed content
Crawl requests falling, response time rising	Server strain lowering the capacity limit	Faster responses, more server resources, 304 support
Pages noindexed "to save budget"	noindex still requires a fetch	Use robots.txt for URLs that should never be crawled

A worked example (illustrative numbers)

Picture a fashion store with 25,000 live products and 600 category pages. Each category has filters for size, colour, brand and price, plus four sort orders, all as crawlable links. On paper the store has about 26,000 pages worth indexing. In practice Googlebot can discover hundreds of thousands of filter URLs. Crawl Stats shows most requests marked "Refresh", sample URLs full of ?sort= and &color=, and the Page Indexing report shows new products sitting in "Discovered - currently not indexed" for weeks.

The fix order would be: decide which filter pages are real landing pages and give them clean static URLs; stop generating crawlable links for the rest and block their parameter patterns in robots.txt; make out-of-stock products that are gone for good return 404 or 410 rather than an empty 200 page; fix internal links that pass through redirects; and keep a product sitemap whose lastmod changes only when the product itself changes. Then watch the discovery share and the time from publish to first crawl in the following weeks. None of these steps is a guarantee; they remove waste so that demand and capacity go to the pages you care about.

What this means for sites in Thailand

Crawl budget mainly matters for the Thai sites that can reach Google's rough thresholds: large e-commerce catalogues, marketplaces, classifieds and property portals, and news publishers that update daily.

Two local patterns are worth checking. If a bilingual site serves Thai and English versions on separate subdomains (th.example.com and en.example.com), then because each hostname has its own budget, check Crawl Stats for each property separately rather than assuming one report covers both. And Thai-language URLs are percent-encoded on the wire, so the same page can be linked with the Thai characters in one place and the encoded form in another. If those resolve through redirects or as separate URLs, you have created duplicates; pick one form and link to it consistently.

Hosting matters too. A shop on an overloaded shared server will see slower responses and a lower capacity limit, whatever its content quality. If a crawl budget concern turns out to be a server concern, the fix sits with hosting rather than with SEO settings. A technical SEO audit is usually the quickest way to see which of these applies.

FAQ

Does crawl budget affect rankings?

No, crawl budget is not a ranking factor; it only decides which URLs Google fetches and how often. A page that is never crawled cannot be indexed or ranked, so on a very large site, wasted crawling can delay new or updated pages from appearing, but crawling more does not make a page rank higher.

How do I know if my site has a crawl budget problem?

Check the Page Indexing report for a large and growing number of "Discovered - currently not indexed" URLs, then open Crawl Stats and look at sample URLs. If new pages get crawled the same day you publish them, you do not have a crawl budget problem.

Should I use noindex or robots.txt to save crawl budget?

Use robots.txt, because noindex does not save crawling. Google must fetch a page to see a noindex tag, so it still spends a request; robots.txt prevents the fetch. Use robots.txt only for URLs you never want crawled, and use 404 or 410 for pages that are permanently gone.

Can I ask Google to crawl my site more?

There is no setting to raise your crawl budget. Google raises the capacity limit itself when the server responds quickly and reliably and there is demand, so the levers are faster responses, more server resources, and content that people link to and visit.

No, each hostname has its own crawl budget, because Google defines a site as a unique hostname. A Thai subdomain and an English subdomain are budgeted separately and each needs its own check in Search Console.

If your catalogue or publishing volume puts you in the group Google is talking about, start by listing every crawlable URL pattern on the site. An e-commerce SEO service covers faceted navigation and product lifecycle handling, and broader SEO in Thailand covers the content and authority side that drives crawl demand.

Crawl Budget Explained: Capacity, Demand and Who Needs to Care