Deepcrawl is now Lumar. Read more.
DeepcrawlはLumarになりました。 詳細はこちら

Lumar Protect for Accessibility

Lumar Product Guide header image showing accessibility test suites in Lumar Protect


Protect allows you to take a proactive approach to accessibility, and prevent issues being introduced to your site when new code is published. This saves huge amounts of time rectifying issues on the live site, while mitigating the risk of non-compliance. 

In Protect, you can configure test suites for a variety of tests, to run as new code is published. These can be flagged as warnings or failures and, if integrated to your CI/CD pipeline, failures will stop the build going live, allowing you to rectify issues before the site is updated.


Setting Up a Test Suite

Step 1 – Connecting your site

From the dashboard, click on the purple ‘New test suite’ button at the top right hand corner of the screen. 

Screenshot of the Lumar Protect dashboard with the 'New test suite'  button highlighted in the top right corner.

This button is also available on the ‘Test Suites’ page.

Screenshot of the Lumar Protect Test Suites page with the 'New test suite' button highlighted in the top right corner.

In Step 1, give your test suite a name, so you can understand the nature of the crawl once it’s run, and add in the domain/subdomain details. 

You can also add any necessary authentication in this step by expanding the ‘Authentication and Whitelisting’ section.

Screenshot of the Lumar Protect test suite set up process, showing the test suite name and domain, and authentication and whitelisting options including authentication, whitelisting and custom DNS.

You can choose from authentication using Username and Password, IP Whitelisting or Custom DNS, or a combination of these methods:

  • Authentication: If your site requires a username and password for the crawler to gain access, you can enter that information here.
  • Whitelisting: Lumar uses the IP address or by default. These are the IP addresses a network administrator would need so that Lumar can bypass any firewalls or blocks you may have in place.
  • Custom DNS Settings: These might be needed if you are testing a staging environment. The only caveat is that you cannot use this setting with JS rendering turned on.

Once you’ve filled in the relevant details, click ‘Save’ to move onto the next step. 

Step 2 – Choosing your crawl settings

In Step 2 you can choose your crawl settings. In the top section, choose the type of crawl. 

  • Web crawl, where the crawler will start at the domain/subdomain you specified in Step 1, and follow links to crawl your site. For example, you could choose to start your crawl at a specific subdomain to test an important part of your site.
  • List crawl, where you can upload a test or .CSV file with a list of specific URLs to crawl. You can use this option to upload a list of URLs built on specific templates to check for errors.

You can choose between these options, or select both. 

JavaScript rendering is enabled by default to make sure the crawler does not miss valuable code and content. For accessibility crawls, this setting cannot be disabled.

To set the crawl speed, use the slider to choose the appropriate crawl speed for your site. When choosing the crawl speed, you should consult with your web ops team to determine the best speed, based on your server capability. 

In the URL Limit field, choose the number of URLs that will be completed in the crawl. As Protect is focused on a specific purpose, you can choose a limit of up to 10,000 URLs. In Analyze (where the crawl scope is much broader) the limits are much higher. 

Finally, you can also add any custom extraction rules that you want to include.

Screenshot of Lumar Protect test suite set up step 2, showing the URL source options (web or list), JavaScript rendering options, crawl speed, URL limit and custom extractions.

At the bottom of this section, you can expand the ‘Advanced Settings’ section to access additional settings:

  • Start URLs: To start the crawl from somewhere other than the primary domain. 
  • Include and Exclude URLs: To limit your crawl to URLs that only include your specified paths, use regular expressions to match specific paths if required. Alternatively, you can add URL paths to exclude them from the crawl. When using this feature, note that exclude rules will take precedence over include rules. If you use Include or Exclude URLs, you must also include those URLs as Start URLs.
  • Robots.txt Overwrite: To replace the live robots.txt for the primary domain with an alternative. 
  • JavaScript Rendering Settings: Here you can add additional settings, including blocking ad or analytics scripts, blocking scripts URLs, injecting custom JS snippet, or including external JS resources.
  • Test Settings: These are Lumar’s recommended levels for settings such as maximum title width, thin or empty page thresholds, or duplicate precision values. If you need to amend these, you can do so here.

Step 3 – Choosing your tests

In Step 3 you can choose up to 100 tests to include in the test suite. When Protect crawls, it will run these tests to dictate the overall success of the build. You can choose from over 90 tests related to accessibility.

At the top of step 3 you can search for the particular tests you want to include, and check the box next to the name to add it to the test suite. For accessibility, you can also check the relevant accessibility standard (e.g. Level A or Level AA) to check for all issues related to that level. 

As you check the relevant options, you’ll see them populate on the right hand side of the screen.

Screenshot of Lumar Protect showing step 3 of setting up a test suite. This screen shows the relevant tests on the left hand side and selected tests on the right.

Once you’ve added tests to the test suite, you can then click on each one on the right hand side to choose the test settings:

  • Test severity. Choose whether the threshold breach will result in a fail (which stops the build) or a warning (where the build continues). 
  • Use smart thresholds. Automatically adjust thresholds based on the worst performance from the previous five crawls. Find out more about smart thresholds in Lumar Protect below. 
  • Failure threshold. Choose the number of URLs that will trigger the warning or failure in Lumar Protect.
  • Jira ticket creation. You can also integrate Protect with Jira, to automatically create a ticket when a Fail or Warning is triggered. Once integrated, click the slider to enable.

Step 4 – Set notifications

In Step 4 you can choose the relevant notifications for your test suites. For email, your own address will appear with notifications set for passes, passes with warnings and fails by default. If, for example, you don’t want to receive an email notification for passes, simply uncheck the box. You can also add in any other email addresses using the ‘New email’ button, and again select which notifications they will receive an email for.

Screenshot of Lumar Protect showing step 4 of the test suite setup, with options for notifications via email, Slack or Jira.

Protect can also deliver notifications via Slack. This can be easily set up using a webhook, to deliver notifications to one or more Slack channels. You can also integrate Protect with Jira to automatically create issues in your backlog for fails and warnings generated when test suites run. 

Step 5 – Running your test suite

In the final step you can choose how you run your test suite:

  • Run via CI/CD. With this method, when development push new elements from staging to live, Protect will run the test suites. If everything passes, publishing to the live site will complete. However, if any tests fail, it will be stopped and sent back to dev so they can fix the issues that didn’t pass. If you choose to run via CI/CD, you’ll find instructions by clicking ‘Show instructions’ under Connect to CI/CD. Instructions for the CI/CD admin can be found here.
  • Run manually. If you don’t want to connect Protect to your CI/CD pipeline, you can run the test suites manually. Simply go to that option and click ‘Run test suite’. 

You can also schedule the test suite to run at a later date (see below). If you do want to set a schedule, click on ‘Go to test suites’ in the bottom right corner of the screen.

Screenshot of Lumar Protect showing step 5 of the test suite setup, where users can connect Protect to the CI/CD pipeline.

Step 6 – Viewing crawl results

The results can be seen on the Protect home page once the crawl has completed. The Builds list shows top level information about each build (crawl) and the number of tests that passed or failed.

If integrated to your CI/CD pipeline, when all tests pass or give warnings then the build will pass. If one test fails, the build will fail.

Screenshot of the Lumar Protect dashboard showing test suite runs and which runs have warnings of failures.

Clicking into the test suite will take you to the build detail page which shows each test that passed, failed, or gave a warning. You can see more detail about the URLs in each test by opening the Lumar report in Analyze.

Screenshot of Lumar Protect test suite results, showing failures and passes for specific tests, and the option to open the report for more analysis.

Bonus Step – Scheduling a test suite run

After you’ve defined your test suite using the steps above, you can schedule your test suite to run at a particular time, or on a regular basis. Click on the ‘Scheduler’ icon (a clock) in the left hand navigation bar. You can then click ‘Add schedule’ in the top right corner.

Screenshot of Lumar Protect with the Scheduler icon in the left navigation, and 'Add schedule' button in the top right corner highlighted.

Simply give the schedule a title, select the date and time, set any repetition you require, and choose the test suites to include in the run. 

Screenshot of the Lumar Protect Scheduler modal, showing options to include a title, date, time, repetition and the test suites included in the schedule.

Once you fill in the details and click save, you’ll see the schedule in the Scheduler page. 

Screenshot of the Lumar Protect scheduler page, showing a weekly test run scheduled in the platform.

Smart Thresholds

What are Smart Thresholds?

Once enabled, Smart Thresholds use the data from the previous five test runs to automatically adjust thresholds based on the worst performance from those tests. 

For example, when you start running a test suite in Protect you need to either run it manually at least 5 times, or set it up on a schedule that runs at least 5 times. Smart Thresholds will then kick in after the 5th run. If the threshold was set at 200 broken pages and you fix some of those broken pages, bringing the total down to 120 Broken Pages, Smart Thresholds will change the threshold to 121 Broken Pages without any manual intervention on your part. 

As your site continues to improve, we keep the threshold in line with it so you’re aware of any negative spikes on the site. If your Broken Pages remain over the set threshold, you will continue to be notified that this test Failed unless you manually set the threshold higher. This is to ensure that a critical alert doesn’t get missed.

Websites are obviously ever-changing, so Smart Thresholds can ensure your thresholds will be kept updated based on how your website is changing over time, without manual intervention.

When to use Smart Thresholds

Obviously you can use Smart Thresholds whenever you like. Like any automation though, it’s important to make sure it’s used efficiently. Automation is a great time saver for repetitive tasks that don’t require complex thinking, but when some errors in your site can lead to a significant loss of traffic or even revenue, your expertise is a vital factor in deciding on appropriate thresholds. For this reason, you may want to retain manual settings for the most critical tests for your site. 

Fortunately, our Smart Thresholds can be enabled on a per test basis, so you may choose to enable it for those tests that result in a ‘warning’, but keep ‘fails’ that block the build manually set. 

Of course you can also switch Smart Thresholds on and off at any time. This means you can turn them on to get a baseline for the thresholds, and then switch them off once that is set.

How to set up Smart Thresholds

In Step 3, where you add in the tests that you want to include in the test suite, you have the option to choose Smart Thresholds. 

To get started, click on the ‘Use smart thresholds’ toggle at the top of the screen.

Screenshot of Lumar Protect test suite setup, showing the option to enable Smart Thresholds highlighted.

By default, ‘apply to all tests’ will be selected. If you’d like to use Smart Thresholds for the entire suite then you’re good to go. The Example platform will use data from the previous five crawls to automatically adjust the threshold.

Screenshot of the Lumar Protect Smart Thresholds options, with the option to apply Smart Thresholds to all tests enabled.

To enable Smart Thresholds for individual test suites, choose ‘Apply to individual tests’. You can then open the relevant tests to enable Smart Thresholds for only selected tests.

Screenshot of Lumar Protect test suite setup showing the option to use Smart Thresholds on individual tests enabled, and a test opened with Smart Thresholds enabled.

And if you change your mind, you can always disable Smart Thresholds whenever you like.

Avatar image for Andrew Levey
Andrew Levey

Andrew heads up Product Marketing at Lumar, and is enthusiastic about helping prospects and clients understand the value of—and get the most out of—the Lumar platform. Outside of work, he likes great TV and film, is a bit of an amateur photographer and sings with the London Gay Men's Chorus.


Get the best digital marketing & SEO insights, straight to your inbox