Deepcrawl is now Lumar. Read more.
DeepcrawlはLumarになりました。 詳細はこちら

Google times out after two minutes when crawling sitemaps

SEO and Digital Marketing Best Practices

Our dedicated DeepCrawl engineers are constantly building, testing and pushing SEO to its limits. So it seems only right to share their findings and help you fine-tune your strategy even further. Introducing our new series on SEO experiments: DeepCrawl Labs…

DeepCrawl Labs
 

The test

Sitemaps can sometimes take significant amounts of time to generate, and we wanted to test how Google reacted to sitemaps with differing response times.
 

The method

We set up multiple XML sitemaps with varying response/load delays to see if Google would be patient enough to wait for them. These ranged from 100 to 350 seconds.
 

The results

Sitemaps with a load time of up to 2 minutes/120 seconds were generally accepted without issues. Nothing over 200 seconds was ever indexed.

Sitemaps Load Time

Some slower sitemaps between 120-200 seconds showed a timeout error, but also showed that they were later accepted with submitted and indexed items.

Sitemaps Load Error
 

What this means for you

We noticed that when crawling sitemaps with longer loading times, the crawl speed was negatively affected. We therefore suggest aiming for a 60 second load time for your sitemap to be sure they aren’t causing any problems.

We also recommend using lots of smaller sitemaps, rather than one big one, as this can reduce the sitemap response times.

If your sitemaps are still taking too long to generate, it may be possible to pre-generate them periodically and cache them on the server.

For tips on creating sitemaps, conducting audits or advanced configuration, have a read of this guide.

Avatar image for Alec Bertram
Alec Bertram

Alec is the former Head of Product at Lumar. He has over 10 years' of experience in SEO.

Newsletter

Get the best digital marketing & SEO insights, straight to your inbox