Crawl budget: when it matters and when you can ignore it

Crawl budget gets talked about a lot, but most small and mid-sized sites never need to worry about it. Here is how to tell if yours does.
Wandex Team
September 29, 2026
Magnifying glass

Crawl budget is the number of pages Google is willing and able to crawl on your site in a given period. It comes up in almost every technical SEO conversation, and for most websites it is not the problem.

The short answer

Google’s own crawl budget guide is written for very large sites (around a million pages or more) and for medium-sized sites (around ten thousand pages or more) whose content changes every day. It also says that if your new pages are usually crawled the same day you publish them, you do not need to worry about it. A 40-page website for a manufacturing company in Chakan or a 200-page blog is nowhere near that.

When it does matter

  • Large sites. E-commerce stores, property portals and job boards with tens of thousands of URLs or more.
  • Sites that generate URLs automatically. Filters, sort orders, search results pages and calendar pages can create thousands of near-duplicate URLs.
  • Sites with slow servers. If your server struggles, Google crawls more gently to avoid overloading it.

How to check

Open Google Search Console and go to Settings, then Crawl stats. You will see how many requests Googlebot made, how fast your server responded and which kinds of files it fetched. Then look at the Pages report: a large pile of “Discovered, currently not indexed” URLs on a big site can be a sign that Google is not getting round to them.

What actually helps

  1. Stop wasting crawls. Block endless filter and sort combinations with robots.txt or handle them with sensible URL design.
  2. Fix redirect chains and broken links. Every hop is a wasted request.
  3. Keep your XML sitemap clean. List only the pages you want indexed, with correct, final URLs.
  4. Speed up the server. Faster responses let Google crawl more without straining your hosting.
  5. Link to important pages. Pages with strong internal links get found and recrawled sooner.

What does not help

Adding a noindex tag does not save crawl budget, because Google still has to crawl the page to see the tag. And asking for indexing in Search Console page by page is not a strategy for a large site.

If you run a large catalogue or listings site and new pages are slow to appear, our technical SEO service starts with exactly this kind of crawl analysis.

Picture of Wandex Team

Wandex Team

The SEO team at Wandex in Pune. We have been helping businesses get found on Google since 2012, and we write here about what we see in our day-to-day work.

Free SEO audit

Want this applied to your site?

Send us your website and we will tell you what to fix first.

Keep reading

Find out what is holding your site back.

Get a free audit: the problems that matter most, in the order we would fix them.