Skip to Main Content

When the word budget is mentioned, the first thing that comes to mind are some finances, planning and allocation of expenses. Well, the budget can also have something to do with yours website, actually crawl budget. Yet, crawl budget it does not define a specific financial construction, but is more of a figurative definition. 

 

What is a crawl budget?

Crawl budget  is a term that tells you how many pages Google crawls your site in a given period of time. Google can crawl and index a varying number of pages, and your "budget" is generally determined by the size of your site, the "health" of your site (how many errors Google encountered) and the number of links to your website.

 

How to check the crawl budget?

The crawl budget of a particular site can be checked quite easily. The initial budget that Google gives to each new site is 1000 pages per day. And this is a number that is quite satisfactory for most smaller sites. 

screenshot from search console

Go to Google Search Console and in the option Settings choose Crawl Stats, where you will see the average number of pages crawled per day.

 

Crawl budget and SEO

Every day, Googlebots search millions of pages and change their ranking, adding them to the index or defining different page ranking penalties. Probably the best instance that talks about it what is it crawl budget is alone Google. What you need to do is to do the basic things correctly which Google requests for your pages and wait for them Google list and index.

There are many reasons why Google crawled and indexed a page. He may have found new links pointing to the site page, or it has been mentioned on a social network, or it may have been updated. XML Sitemap site… There is no way to list all the reasons why you should Google indexed the URL, but when it finds it has to, it adds it to the to-do list.

A site's pages must be indexed in order to be displayed in the SERP, and the mismatch between crawl budget and the number of pages on your site means that the pages cannot be displayed in the SERP. This means you won't be able to get organic traffic from these pages and unfortunately, you won't be able to rank for the keywords these pages are targeting. So if your page count exceeds crawl budget of your site, you will have pages on your site that are not indexed.

 

When can a crawl budget become a problem?

If you work with smaller websites, crawl budget it probably won't be something you have to worry about. According to Google-in, "Crawl budget is not something most publishers have to worry about. If a site has less than a few thousand URLs, most of the time it will be crawled efficiently."

However, if you work on large websites such as e-commerce shops, especially those that automatically generate pages based on URL parameters, you may want to prioritize activities that help Google-in order to understand what and when to list.

If you spend in vain crawl budget, search engines will not be able to index your website efficiently. They will spend time on parts of your site that don't matter, which can leave important parts of your site undiscovered. If they don't know about the pages, they won't crawl and index them, and you won't be able to drive visitors through search engines to them.

 

What URLs does Google index?

Sometimes it is necessary to make sure which pages of the site Google he visits and which he doesn't. The only "real" way to find out is by looking server logs your website. Depending on your hosting type, you may not always be able to download log files. However, if you think you need to work on optimization crawl budget-and since your site is big, you should get them.

 

How to optimize crawl budget?

Optimization crawl budget-a it can refer to its enlargement (ie. stating Googleto spend more time on your website) as well as targeting Google-and to "more wisely" spend the time he has already allocated to your website.

There are numerous factors that can negatively affect budget spending, so it is necessary to constantly and regularly monitor the website and correct potential errors. 

Work on website speed, check all redirects, robots.txt files, duplicate content i canonical links, and with large sites take care to prevent Google to publish the budget on pages where it should not be done. Wasting server resources on these types of pages will drain activity crawl budget-a from pages that actually have value, which can prevent or delay Google to discover your great content.

 

For more news and interesting stories from digital marketing visit our blog page or follow our Instagram profile.

Made by Nebojsa Radovanovic - SEO Expert @Digitizer