The Lumar website intelligence platform is used by SEO, engineering, marketing and digital operations teams to monitor the performance of their site’s technical health, and ensure a high-performing, revenue-driving website.
The cornerstone of our platform is a website crawler, which crawls a site’s pages to collect data for sophisticated website analytics.
We obey the Robots.txt protocol and will not crawl your site if you exclude the Lumar user-agent token, e.g.
User-agent: lumar Disallow: /
User-agent: deepcrawl Disallow: /
Our crawler requests primarily come from one of two IP addresses (52.5.118.182 or 52.86.188.211) or in some cases from a regional IP. You can find a full list of these IP addresses in JSON format here.
Please note that our user-agent may reference our previous company name (Deepcrawl) to avoid creating issues for our customers.
Please also note that we do not support the crawl-delay directive. Our aim is to match the way Google crawls as closely as possible, and Google does not support the crawl-delay directive. Crawl-delay also can make it difficult to support domain level crawl rate limits, which is why most dev ops use a bot management system to give them complete control.
If you have already excluded the Lumar user-agent but your site is still being crawled without your permission and you would like it to stop, then please contact us at support@lumar.io.