Facebook Pixel for LeadDigital

There is No Benchmark for Crawl Budget

My Post - 2020-02-17T175224.099.pngThere is no benchmark for what is considered an optimal crawl budget for websites, Google’s John Mueller says.

This was stated in response to a Reddit thread where an SEO asked whether there’s ideal percentage of pages that Googlebot should be crawling every day. The thought behind this question is content may be kept fresh if Googlebot crawls it more regularly.

Here is the question in full:

“While everyone talks about crawl budget, I haven’t found a specific cutoff or a range for this. Like what should be the minimum percentage of pages out of total (or total changes made everyday) should be crawled by GoogleBot everyday to keep the content fresh?

I understand that this can vary a lot because lot depends upon static/variable content on the website. I am just trying to understand how to benchmark crawl budget.”

In response, Mueller simply states: “There is no number.”

Therefore, if you’re looking to boost your crawl budget, there’s really no ideal number to be aiming for. However, that’s not to say there’s no benefit to optimizing a website’s crawl budget.

What is Crawl Budget?

Simply put, a crawl budget is the number of URLs Googlebot is able to crawl (based on site speed) and wants to crawl (based on user demand).

A higher crawl budget can help keep popular content fresh, and help prevent older content from becoming stale.

Factors Affecting Crawl Budget

One of the best ways to improve crawl budget is to limit the amount of low-value-add URLs on a website, such as:

  • Faceted navigation and session identifiers
  • On-site duplicate content
  • Soft error pages
  • Hacked pages
  • Infinite spaces and proxies
  • Low quality and spam content

These types of pages can steal crawl activity away from a site’s more important pages. – Read more

Facebook
Twitter
LinkedIn
Pinterest
best ad platforms

The 8 BEST Advertising Platforms for 2024

Sometimes, the allure of the new can sweep us off our feet. In the realm of advertising, this chase for the next big platform, the next untapped audience, it’s akin to a modern-day gold rush. We’re miners, sifting through the digital silt, hoping to find those glittering nuggets of audience

Read More »
Best franchise marketing

Best Digital Agency for Franchises: Top Picks for Boosting Your Business

Digital marketing has become an essential aspect of modern business, and franchises are no exception. With the rise of e-commerce and online shopping, franchises must have a robust online presence to remain competitive. However, creating and maintaining a digital marketing strategy can be challenging, especially for franchise owners who have

Read More »