Concepts

Test automation is one of the three main testing activities in Testmo, besides manual test cases and exploratory testing. We've designed Testmo so it's very easy to submit your test automation results without having to create or manage your automation test suites and cases first.

In this guide we are taking a detailed look at Testmo's test automation concepts and explain how automation runs, sources, threads and test results work in Testmo. Let's start with an overview of Testmo's test automation entities.

Testmo test automation entities

Sources

To track and compare test results over time, it's useful to link similar tests and results. In Testmo, your test runs of the same tests are linked to the same source (you can think of sources as your test suites). You specify the source of your test runs as an argument to the Testmo CLI tool when you submit your test results.

So if you have a test automation suite for your backend API tests, you would submit the relevant runs with a source of the name backend-api for example. Testmo would then create a new source in the specified project if it doesn't exist already, and automatically link your subsequent backend API runs to the same source. This way you can easily compare tests over time and group/filter your runs more easily.

It's important not to mix the same source name for different tests/suites. You need to specify a different source for your different automation suites. E.g. typical examples would be:

• backend-api

• mobile-android

• mobile-ios

• mobile-performance-ipad

• frontend-selenium

• frontend-admin-webdriver

• frontend-unit

• etc.

Test runs

Whenever you run your automated tests as part of your CI/CD pipeline, from your build server or manually from the command line, you can submit the test results to Testmo. Each time you submit your automation run to Testmo, a new test run is created in Testmo. If you submit results from different sources/automation suites, you would submit these as separate test runs.

Likewise, if you execute the same suite multiple times in your CI/CD pipeline against different configurations, such as different web browsers or different mobile platforms, you would submit these as separate test runs.

Each test run can optionally also be linked to a configuration or milestone so it's easier to track, report and group your runs. Please see the relevant documentation on how to link runs to configurations and milestones.

Threads

Within test automation runs, your tests are grouped into threads. A test run always has at least one thread and can have multiple threads if you execute your automation suite with parallel CI/CD test jobs.

Threads can be useful if you have set up your CI/CD workflow to run your tests in parallel to speed up test execution. In such cases, you would submit the tests for each parallel test job as a separate thread. This is useful so you can see the test times of the different test jobs as well as track the overall success state, console output and additional fields.

Parallel test execution with threads

If you just have a regular non-parallel test automation run, you do not need to create threads yourself. Our automation:run:submit command will handle everything automatically and create a single thread and add all results to it.

Tests & results

All the test results you submit are stored together with the relevant thread/run. You can either see the tests and results filtered for a specific thread, or you can see all results of the run at once.

Your tests are also grouped together into folders, usually based on the classes/sections of your test automation suite. Testmo's CLI tool identifies the classes and test names from your test result files automatically. It also generates a unique key based on the folder (class) and test name so it can link future results of the same test.

Fields, artifacts & links

It is possible to optionally add additional fields, artifacts and links to runs, threads and tests. This is useful if you would like to include relevant details such as version numbers, Git hashes or a link to your CI/CD pipeline. These details are then conveniently displayed in Testmo.

For individual tests, Testmo automatically detects and submits any custom fields for failed tests from your result files. You can learn more about submitting additional fields and links to threads and runs.