Deepcrawl is now Lumar. Read more.
DeepcrawlはLumarになりました。 詳細はこちら


The Lumar website intelligence platform is used by SEO, engineering, marketing and digital operations teams to monitor the performance of their site’s technical health, and ensure a high-performing, revenue-driving website. 

The cornerstone of our platform is a website crawler, which crawls a site’s pages to collect data for sophisticated website analytics. 

We obey the Robots.txt protocol and will not crawl your site if you exclude the Lumar user-agent token, e.g.

User-agent: lumar
Disallow: /
User-agent: deepcrawl
Disallow: /

Our crawler requests primarily come from one of two IP addresses ( or or in some cases from a regional IP. You can find a full list of these IP addresses in JSON format here.

Please note that our user-agent may reference our previous company name (Deepcrawl) to avoid creating issues for our customers.

Please also note that we do not support the crawl-delay directive. Our aim is to match the way Google crawls as closely as possible, and Google does not support the crawl-delay directive. Crawl-delay also can make it difficult to support domain level crawl rate limits, which is why most dev ops use a bot management system to give them complete control.

If you have already excluded the Lumar user-agent but your site is still being crawled without your permission and you would like it to stop, then please contact us at


Get the best digital marketing & SEO insights, straight to your inbox