If you are using Cloudflare to improve your website’s security and performance, our crawler may be blocked by Cloudflare’s firewall, hence stopping you from performing a crawl on your website. You can lift this restriction by following the simple steps below.
1. Log into your Cloudflare account
2. On your left sidebar, click on Security, then click on WAF
3. On that page, click on the “Create firewall rule” button
4. Name your firewall rule and select either of the two options below:
A. For IP address whitelisting, you’ll need to enter the Lumar IP addresses for crawling. To do this, choose IP Source Address as the field, equals as the Operator and 52.5.118.182 as the Value. You will then need to click Or to add a second line for IP address 52.86.188.211.
B. If you want to crawl with a regional IP or custom proxy, add ‘deepcrawl’ as an allowed user agent. Please note that while we have changed our name to Lumar, our user agent still (for the time being) references ‘deepcrawl’ to avoid creating issues for existing customers. You therefore need to choose User Agent as the field, Contains as the Operator and enter deepcrawl as the Value.
5. Once that information is entered, select Allow from the Action dropdown and click on the “Deploy firewall rule” button
