If you are using Cloudflare to improve your website’s security and performance, our crawler may be blocked by Cloudflare’s firewall, hence stopping you from performing a crawl on your website. You can lift this restriction by following the simple steps below.
1. Log into your Cloudflare account
2. On your left sidebar, click on Security, then click on WAF
3. On that page, click on the “Create firewall rule” button
4. Name your firewall rule and select either of the two options below:
A. If you have a static IP address, we recommend choosing IP Address and entering your own external IP address
B. If you do not have a static address or want to crawl your website from different IP addresses/locations, please add Lumar as an allowed User Agent
Please note that while we have changed our name to Lumar, our user agent still (for the time being) references ‘deepcrawl’ to avoid creating issues for existing customers.
5. Once that information is entered, select Allow from the Action dropdown and click on the “Deploy firewall rule” button