That Marketing Buddy

What Is Crawl Budget? (And When You Can Ignore It)

Crawl budget is how many URLs Google can and wants to crawl on your site. Google says it only matters for very large or fast-changing sites. Here's how to tell which side you're on, and what to fix if it does matter.

Joonas Rotko
Joonas Rotko

SEO & GEO professional · Updated October 19, 2026 · 10 min read

Affiliate links on this page may earn me a commission, at no extra cost to you. They don't change what I recommend.

In short

Crawl budget is the number of URLs Google can and wants to crawl on your site, set by how much your server can handle and how much Google wants your pages. Google says it matters for sites with 1 million+ pages that change weekly, or 10,000+ pages that change daily. Most small business sites are far below that. If new pages get indexed within a few days, you don't have a crawl budget problem. If you do, the usual causes are filter URLs, duplicate URLs, long redirect chains and a slow server.

Contents (7)
  1. 1What is crawl budget?
  2. 2Does crawl budget matter for your site?
  3. 3What wastes crawl budget
  4. 4How to fix it
  5. 5Tools for checking crawl budget
  6. 6What the Crawl Stats report tells you
  7. 7The bottom line

Google crawls a huge number of pages every day, but it doesn't crawl every page on every site. Crawl budget is the word for how much of your site it gets to. For most small business sites, it's not something to lose sleep over. Google says so itself. But if your site is large, changes fast, or new pages take weeks to show up in search, it's worth understanding.

What is crawl budget?

Crawl budget is the number of URLs Google can and wants to crawl on your site. Google defines it with two parts:

  • Crawl capacity limit: how much crawling your server can take. If your site slows down or returns server errors, Googlebot crawls less. If it responds quickly, Google can crawl more.
  • Crawl demand: how much Google wants to crawl your pages. Popular pages and pages that change often get crawled more. Pages that rarely change, or that Google sees as low value, get crawled less.

You need both. A fast server doesn't help if Google has little interest in your pages, and strong demand doesn't help if your server can't keep up.

Does crawl budget matter for your site?

Google's own guide to crawl budget is written for two kinds of sites:

  • Large sites with 1 million+ unique pages whose content changes about once a week.
  • Medium or larger sites with 10,000+ unique pages whose content changes daily.

A typical small business site with a few dozen or a few hundred pages is nowhere near that. If that's you, Google will usually crawl everything that's worth crawling, and your time is better spent on content and links.

Crawl budget does become worth a look when:

  • You run a large online shop. Filters for size, colour, brand and sort order can turn a few thousand products into a huge number of crawlable URLs.
  • You publish a lot, fast. A news site or a big marketplace needs new pages found quickly.
  • You've just moved domains or changed your URL structure. Google has to work through all the redirects and the new URLs.
  • Your pages depend on JavaScript to show their content. Google renders JavaScript pages in a queue. Google says a page "may stay on this queue for a few seconds, but it can take longer than that."

Good to know

The simple test

Publish a new page, add it to your sitemap, and check it with the URL Inspection tool in Google Search Console. If it's indexed within a few days, crawl budget isn't your problem. If it takes weeks or never appears, you may have a crawl budget problem, or a quality problem: Google may have crawled the page and decided not to index it.

What wastes crawl budget

Every time Googlebot fetches a URL that doesn't matter, that's a request that could have gone to one that does. These are the usual causes.

1. Filter and sort URLs

The biggest one for online shops. Each combination of colour, size, brand, price and sort order can create its own URL, and most of those pages show almost the same products. Googlebot can't know that until it fetches them.

2. Duplicate URLs

HTTP and HTTPS. With and without www. With and without a trailing slash. Tracking parameters like ?utm_source= or session IDs. To Googlebot, each version is a separate URL to crawl.

3. Redirect chains

URL A redirects to B, which redirects to C. Google's guide says to "avoid long redirect chains, which have a negative effect on crawling." Each extra hop is another request.

4. Soft 404s

Pages that return a normal 200 status but show "no results" or nothing at all. Google has to fetch them to find out they're empty. Removed pages should return a real 404 or 410.

5. Thin pages

Tag pages, author archives with one post, empty categories. They use crawl requests and add little.

6. Crawl traps

Calendars with endless future dates, relative links that keep adding folders to the URL, filters that combine forever. These can keep Googlebot busy on pages that don't really exist.

How to fix it

1. Block URLs Google doesn't need in robots.txt

Use robots.txt to stop Googlebot crawling URL patterns that only waste requests, such as internal search results and filter or sort parameters. For example:

  • Disallow: /search
  • Disallow: /*?sort=

Don't block the CSS or JavaScript files your pages need. Google needs them to render the page.

Don't use noindex to save crawl budget. Google's guide is clear on this: Google still requests a noindexed page and only drops it after it sees the tag, so the crawl is still spent. noindex keeps a page out of search results, not out of the crawl.

2. Keep your XML sitemap clean

Your sitemap should list the pages you want in search, not every URL on the site:

  • Only include indexable, canonical URLs that return a 200 status.
  • Leave out redirected, noindexed and duplicate URLs.
  • Keep each sitemap file under 50,000 URLs.
  • Use the <lastmod> date for pages that have really changed, and keep it accurate.

3. Handle filters on purpose

For an online shop, decide which filter pages deserve to be found (say, "red running shoes") and which don't (sort orders, price sliders, combinations of five filters). Block the ones that don't in robots.txt, or build those filters so they don't create a new URL at all. Canonical tags help Google pick the main version of a page, but Google may still crawl the variations.

4. Pick one version of every URL

  • Redirect HTTP to HTTPS.
  • Redirect www to non-www, or the other way round.
  • Choose trailing slash or no trailing slash and stick to it.
  • Set canonical tags on parameter variations.

5. Shorten redirect chains

Make every redirect go straight to the final URL in one hop, and update your internal links to point at the final URL so Googlebot doesn't have to follow redirects at all.

6. Speed up your server

Google's guide says: "If Google can load and render your pages faster, we might be able to read more content from your site." Fix slow responses and server errors first. When a server keeps returning 5xx errors, Googlebot slows down.

7. Use status codes correctly

  • Return 404 or 410 for pages you've removed for good. Google calls a 404 "a strong signal not to crawl that URL again."
  • Return 503 during short maintenance windows, so Google knows to come back.

8. Link to new pages

Link to new content from pages Google already visits often, like your homepage or category pages, and add new URLs to your sitemap straight away. For a page that really matters, you can request indexing with the URL Inspection tool in Search Console.

9. Remove pages that don't earn their place

Old thin pages, near-duplicates and empty archives can be merged, redirected or removed (with a 404 or 410). Fewer low-value URLs leaves more of Google's attention for the pages that matter.

10. Look at your server logs

Server logs are the only place you can see every request Googlebot actually made. With them you can see:

  • Which pages Googlebot visits most, and whether those are the right ones.
  • Sections of your site it rarely visits.
  • Pages in your sitemap it never fetches.
  • Requests spent on redirects, 404s and parameter URLs.

For a small site this is rarely needed. For a large one it's the clearest view you'll get.

Tools for checking crawl budget

You don't need an expensive tool to start.

Free: Google Search Console

The Crawl Stats report (Settings, then Crawl stats) shows how many requests Googlebot makes per day, your average response time and the mix of status codes it gets back. The URL Inspection tool shows when Google last crawled a specific page. Start here.

Screaming Frog SEO Spider

I've used Screaming Frog. It crawls your site the way a search engine would and finds duplicate pages, redirect chains, broken links and orphan pages. The free version crawls up to 500 URLs, and a licence costs $279 a year to remove the limit. Log analysis is a separate product, the Screaming Frog Log File Analyser.

Other site audit tools

  • Sitebulb: a desktop site auditor that explains each issue and ranks what to fix first. Lite is $18 a month, or $180 a year.
  • Semrush: its Site Audit flags crawl problems like redirect chains, duplicate content and broken pages, alongside keyword research and backlinks. The SEO plan is $139.95 a month.
  • Ahrefs: Site Audit is included in its plans and covers the same basics.

For a side-by-side look with current prices, see my guide to the best technical SEO audit tools.

For very large sites

Sites with hundreds of thousands of pages or more often use enterprise platforms like Botify or Lumar, which combine crawling, server log analysis and search data. They're sold through sales demos rather than published prices, and they're built for large teams.

What the Crawl Stats report tells you

The Crawl Stats report in Search Console is the quickest health check:

  • Response time: if it's high or climbing, your server is limiting how much Google can crawl.
  • Status codes: most responses should be 200. A large share of redirects, 404s or server errors means wasted requests.
  • Crawl requests over time: a sudden drop can point to server problems or something blocking Googlebot.

For more on tracking SEO without paid tools, see my guide on how to track SEO progress.

The bottom line

For most small business sites, crawl budget isn't the thing holding you back. Run the simple test above. If new pages are indexed within a few days, focus on content and links instead.

If your site is large or changes fast, start with the basics: block filter and search URLs you don't need crawled, keep one version of every URL, shorten redirect chains, keep your sitemap clean and fix a slow server. Use Search Console's Crawl Stats to check, a crawler like Screaming Frog to find the problems, and server logs when you need to see exactly what Googlebot does.

For a full comparison of SEO tools at every price point, see my best SEO software roundup.

Sources

Tools in this guide

Read next

Joonas Rotko

Joonas Rotko

SEO & GEO professional, founder of That Marketing Buddy

I'm a Finnish SEO and GEO professional with 11+ years in digital marketing, including years as a co-owner of a digital marketing agency. On That Marketing Buddy I test and compare SEO and AI-visibility software for entrepreneurs, marketing professionals and agencies.

Hands-on tests

Get my newest hands-on tests of SEO and GEO tools

I test SEO and AI visibility tools on my own sites and send you what I found: real prices, how accurate the data was and what didn't work.

At most one email a week. Unsubscribe anytime. I use your email only for this. Privacy policy