Deepcrawl is now Lumar. Read more.
DeepcrawlはLumarになりました。 詳細はこちら

Crawling Multiple Domains Or Subdomains

It’s possible to configure exactly which domains and subdomains will be included in your crawl.

This is useful if you wish to filter in/out known subdomains or if you only want to crawl a very specific area of your website.

Any URLs that are not defined within the crawl are treated as external and will not be crawled and will therefore not appear in your reports.

By default, the domain listed in step 1 of the crawl setup defines the domain scope and crawl start point.

Make sure to choose the correct HTTP/HTTPS version.

dc2 crawl setup domain check

If you select the ‘Crawl subdomains’ option for Website crawl in step 2, Lumar will automatically include all URLs on a subdomain of your primary domain which are discovered during the crawl.

dc2 URL sources

To include additional domains or subdomains in your crawl, add them to the ‘Secondary Domains’ section under Scope in Advanced Settings.

You can now enter as many domain or subdomains as you like.

dc2 crawl setup secondary domains

You can also use the ‘Included Only URLs’ and ‘Excluded URLs’ filters in Advanced Settings, to further restrict any domains or subdomains. e.g. Filter out all HTTPS versions of websites.

dc2 include exclude https

For further information on positive and negative URL restrictions in your crawl set up, check out the Restricting a Crawl guide.

If your setup is getting complicated, feel free to contact us if you need any help with these features.

Avatar image for Tristan Pirouz
Tristan Pirouz

Marketing Strategist

Tristan is an SEO enthusiast, strategist, and the former Head of Marketing at Lumar.


Get the best digital marketing & SEO insights, straight to your inbox