Technical SEO
Crawling
Crawling is the process by which search engine bots discover pages on the web by following links and reading content.
A crawler like Googlebot fetches a page, reads its content and links, then queues the new links it finds. If a page cannot be crawled, it generally cannot be indexed or ranked. Clear internal linking and a clean sitemap help crawlers find everything that matters.
Crawl behaviour is observable, and server logs are the only place you see it directly. Analytics shows humans, Search Console shows a sampled summary, but logs show every request Googlebot actually made and what status it received. That is where you find the expensive surprises: crawlers spending most of their requests on parameter URLs, or hitting a redirect chain four hops deep on every visit, or receiving 500 errors that no user ever saw. For sites over a few thousand URLs, a log review once or twice a year finds problems no crawl simulation surfaces.
Example
A store's server log for one day shows Googlebot made 9,400 requests: 6,100 hit filter combinations like /shoes?colour=red&sort=price, 200 hit pages that returned errors, and only 3,100 reached real product pages. New products were taking three weeks to appear in results. Blocking the filter parameters redirected that crawl effort to pages that sell.
Frequently asked questions
- Why isn't Google crawling some of my pages?
- Common reasons are that nothing links to them, robots.txt blocks the path, the pages sit too deep in the site, or the crawler exhausted its budget on low-value URLs first. Search Console's Pages report tells you which reason applies, so you are not left guessing.
- Do crawlers execute my JavaScript?
- Googlebot renders JavaScript, but in a second pass that can be delayed, and many other crawlers including several AI bots do not render it at all. If your content only exists after JavaScript runs, assume a meaningful share of crawlers never see it.
Related terms
Working on Crawling? See how our SEO audit can help.
Explore SEO audit