How to use Crawl Stats in Google Search Console to optimise crawl budget

There’s a heck of a lot of useful stuff in Google Search Console, but are you using it as much as you could?

The Crawl Stats report in Google Search Console is one of the most underrated tools available to SEOs – if you’ve heard of it, that is! Tucked neatly away under the Settings tab, it’s often forgotten about – but if you want Google to spend its time wisely on your site, this is where you should be looking.

Whether you’re managing a large e-commerce store or a content-heavy site, understanding how Google is crawling your pages can help you make smarter decisions, reduce crawl waste, and improve your visibility in search engines.

Why crawl budget matters

For smaller sites, crawl budget probably isn’t an issue. But once your site has thousands of URLs, or auto-generates pages with parameters, filters, or variations, crawl budget becomes critical. E-commerce sites are one of the biggest culprits for this!

If Google is spending time crawling old 404s, empty pages, or pointless URL parameters, your most important content might be sitting in the queue, waiting to be discovered, or simply not being crawled at all. That’s not ideal if you’ve just updated a key category page or launched a new product line.

What to look for in the Crawl Stats Report

The Crawl Stats report tells you:

• How often Google is crawling your site
• Which pages it’s spending the most time on
• How successful those crawl attempts are

The key thing to watch out for is where crawl time is being spent. If a significant portion of Google’s crawl activity is being wasted on non-valuable pages, it’s time to act and put that time to better use.

Fixing crawl waste: 404 pages

One of the most common crawl issues is 404 errors. You might think once a page is gone, Google stops checking it – but nope. Google can (and will) keep trying to crawl a deleted URL for years.

What you should do:

• Out-of-stock product? Keep the URL live and show users the product is currently unavailable.
• Product discontinued? Redirect the URL to the most relevant alternative or to the category page.
• Nothing to redirect to? Use a 410 (Gone) status to tell Google the page has been permanently removed.

The goal is to send the clearest possible signal about the status of your content, so Google doesn’t waste time.

Are parameter URLs wasting your crawl budget?

You might notice a large number of parameter-based URLs in the “OK” section of the Crawl Stats report. These are usually URLs with things like session IDs, tracking tags, filter options, or sort orders. For example:

example.com/product?ref=instagram  

example.com/shoes?sort=latest

These often don’t offer any SEO value, and from Google’s perspective, they’re usually seen as duplicate or near-duplicate content.

Why it matters:

When Google is crawling these variations, it isn’t spending that time on your core content. That means:

• New or updated pages might not get crawled as fast.
• You could be wasting crawl budget on irrelevant or low-priority pages.
• Your internal link equity and crawl path get diluted.

How to fix it:

• Block them in robots.txt

 If these URLs don’t need to be crawled, disallow them:

Disallow: /*?sort=

Disallow: /*?ref=

• Use canonical tags

 Where parameters are necessary for users (like filters), make sure you’re using canonicals to point to the main version of the page.

• Review filter and sort UX

 If crawl bloat is a regular issue, look at how your filters and site structure are generating these URLs and whether they need to be crawlable at all.

A simple report, a big impact

The Crawl Stats report doesn’t give you a flashy dashboard or instant SEO wins, but it does give you something better – insight into how Google sees your site at scale.

By reviewing this report monthly and taking action on the pages that are wasting crawl time, you make it easier for Google to find and prioritise your best content.

All Articles