Technical SEO

Crawl Budget

Crawl budget is the number of pages a search engine will crawl on a site within a given window, set by the site's size, speed, and authority.

It only matters at scale: a 200-page site never hits the limit, but a 500,000-URL site that wastes budget on filter parameters and dead pages can leave important content undiscovered. Tight internal linking and pruning low-value URLs direct that budget where it counts. Most small sites do not need to worry about it.

The clearest symptom of a budget problem is a lag between publishing and indexing that keeps getting longer. New pages that once appeared in days start taking weeks, while the crawl stats report shows request volume flat or falling. The cause is almost always URL sprawl, not server capacity. Faceted navigation is the usual culprit: a handful of filters can generate more crawlable combinations than the site has real products. Fix it at the source, by controlling which combinations are linked and crawlable at all. Blocking the results after they exist is the harder job.