Protect – Setting Up a Test Suite

Lumar Protect (formerly known as Automation Hub) allows you to set tests that will automatically run when new code is released. These tests will warn you of errors in the new code and can automatically stop the release if any critical tests fail. 

In this guide we’ll walk you through setting up test suites using Protect.

 

Step 1 – Connecting Your Site

From the dashboard, click on the purple ‘New test suite’ button at the top right hand corner of the screen. 

This button is also available on the ‘Test suites’ page.

In Step 1 give your test suite a name, so you can understand the nature of the crawl once it’s run, and add in the domain/subdomain details. 

You can also add any necessary authentication in this step by expanding the ‘Authentication and Whitelisting’ section. You can choose from authentication using Username and Password, IP Whitelisting or Custom DNS, or a combination of these methods:

• Authentication: If your site requires a username and password for the crawler to gain access, you can enter that information here.
• Whitelisting: Lumar uses the IP address 52.5.118.182 or 52.86.188.211 by default. This are the IP addresses a network administrator would need so that Lumar can bypass any firewalls or blocks you may have in place.
• Custom DNS Settings: These might be needed if you are testing a staging environment. The only caveat is that you cannot use this setting with JS rendering turned on.

Once you have filled in the relevant details, click ‘Save’ to move onto the next step.

 

Step 2 – Choosing Your Crawl Settings

In Step 2 you can choose your crawl settings. In the top section, you can choose the type of crawl:

• Web crawl, where the crawler will start at the domain/subdomain you specified in Step 1, and follow links to crawl your site.
• List crawl, where you can upload a text or .CSV file with a list of specific URLs to crawl.

You can choose between these options, or select both.

JavaScript Rendering

JavaScript rendering is set to ‘Not required’ by default. In almost all cases you would want to tick the button for ‘Enable JavaScript Rendering’ so that the crawler does not miss valuable code and content.

Crawl Speed

Use the slider to choose the appropriate crawl speed for your site. When choosing the crawl speed, you should consult with your web ops team to determine the best speed, based on your server capability. 

URL Limit

Here you can choose the number of URLs that will be completed in the crawl. As Protect is focused on a specific purpose, you can choose a limit of up to 10,000 URLs. In Analyze (where the crawl scope is much broader) the limits are much higher. 

Finally, you can also add any custom extraction rules that you want to include.

At the bottom of this section, you can expand the ‘Advanced Settings’ section to access additional settings:

• Start URLs: To start the crawl from somewhere other than the primary domain. 
• Include and Exclude URLs: To limit your crawl to URLs that only include your specified paths, using regular expressions to match specific paths if required. Alternatively you can add URL paths to exclude them from the crawl. When using this feature, note that exclude rules will take precedence over include rules. If you use Include or Exclude URLs, you must also include those URLs as Start URLs.
• Robots.txt Overwrite: To replace the live robots.txt for the primary domain with an alternative. 
• JavaScript Rendering Settings: If you have enabled JS rendering for the crawl, you can add additional settings here including blocking ad or analytics scripts, blocking scripts URLs, injecting a custom JS snippet, or including external JS resources.
• Test Settings: These are Lumar’s recommended levels for settings such as maximum title width, thin or empty page thresholds, or duplicate precision values. If you need to amend these, you can do so here.

 

Step 3 – Choosing Your Tests

In Step 3 you can choose up to 100 tests to include in the test suite. When Protect crawls, it will run these tests to dictate the overall success of the build. You can choose from over 200 tests related to config, content, indexation, links, mobile, performance, source gap, structured data and more.

By default, the following tests are automatically selected and set to fail if greater than or equal to 10%:

• 5xx errors
• Broken pages (4xx errors)
• Max fetch time

You’ll see these tests have purple ticks next to them on the left hand side. If you want to remove any of these tests, simply uncheck the tickbox. To add tests to the test suites, you can scan through the list of tests and check the relevant boxes. You can also use the ‘Search’ and ‘Filter’ options at the top of the list, to find the tests you’re looking for.

Once you’ve added tests to the test suite, you can then click on each one on the right hand side to choose the test settings:

• Test severity. Choose whether the threshold breach will result in a fail (which stops the build) or a warning (where the build continues). 
• Use smart thresholds. Automatically adjust thresholds based on the worst performance from the previous five crawls. Find out more about smart thresholds in Lumar Protect. 
• Failure threshold type. Choose between a percentage of URLs or an absolute number.
• Failure threshold. Choose whether the failure threshold is greater than or equal to, or less than a particular percentage or absolute number. 
• Jira ticket creation. You can also integrate Protect with Jira, to automatically create a ticket when a Fail or Warning is triggered. Once integrated, click the slider to enable.

 

Step 4 – Set Notifications

In Step 4 you can choose the relevant notifications for your test suites. For email, your own address will appear with notifications set for passes, passes with warnings and fails by default. If, for example, you don’t want to receive an email notification for passes, simply uncheck the box. You can also add in any other email addresses using the ‘New email’ button, and again select which notifications they will receive an email for.

Protect can also deliver notifications via Slack. This can be easily set up using a webhook, to deliver notifications to one or more Slack channels. You can also integrate Protect with Jira to automatically create issues in your backlog for fails and warnings generated when test suites run. 

 

Step 5 – Running Your Test Suite

In the final step you can choose how you run your test suite:

• Run via CI/CD. With this method, when development push new elements from staging to live, Protect will run the test suites. If everything passes, publishing to the live site will complete. However, if any tests fail, it will be stopped and sent back to dev so they can fix the issues that didn’t pass. If you choose to run via CI/CD, you’ll find instructions by clicking ‘Show instructions’ under Connect to CI/CD. Instructions for the CI/CD admin can be found here.
• Run manually. If you don’t want to connect Protect to your CI/CD pipeline, you can run the test suites manually. Simply go to that option and click ‘Run test suite’. 

You can also schedule the test suite to run at a later date (see below). If you do want to set a schedule, click on ‘Go to test suites’ in the bottom right corner of the screen.

 

Step 6 – Viewing Crawl Results

The results can be seen on the Protect home page once the crawl has completed. The Builds list shows top level information about each build (crawl) and the number of tests that passed or failed.

If integrated to your CI/CD pipeline, when all tests pass or give warnings then the build will pass. If one test fails, the build will fail.

Clicking into the test suite will take you to the build detail page which shows each test that passed, failed, or gave a warning. You can see more detail about the URLs in each test by opening the Lumar report in Analyze.

 

Bonus Step – Scheduling a Test Suite Run

After you’ve defined your test suite using the steps above, you can schedule your test suite to run at a particular time, or on a regular basis. Click on the ‘Scheduler’ icon (a clock) in the left hand navigation bar. You can then click ‘Add schedule’ in the top right corner.

Simply give the schedule a title, select the date and time, set any repetition you require, and choose the test suites to include in the run. 

Once you have this filled in, click save and you’ll be able to see the details of your schedule in the ‘Scheduler’ dashboard.