Testmo CLI
The Testmo CLI is a powerful command line tool to submit your test automation results to Testmo. It is the best way to collect and report your test results from your test automation workflow, CI/CD pipeline and build servers.
We designed the Testmo CLI to work with practically any test automation tool and framework so you can start using it and submitting your tests results quickly.
The Testmo CLI is distributed as an NPM package so it's very easy to install and deploy it on any system (e.g. Linux, Windows & macOS) and in any environment (build machines, dedicated servers, Docker containers, CI/CD processes etc.).
Examples for GitHub Actions, GitLab CI/CD, CircleCI, Jenkins and more
We also have full examples on how to integrate Testmo with various CI/CD tools and workflows. See our CI/CD integration page to learn more.

Design goals

The Testmo CLI is used for and supports various use cases and scenarios:
  • Quickly submit your test automation results from the command line
  • Easy integration with any CI/CD and build system such as GitHub, GitLab, Jenkins, CircleCI, Bitbucket and more
  • Works with any test automation tool and framework by using standard JUnit XML report files
  • Supports both basic and advanced workflows such as parallel testing jobs
  • Can optionally capture full test console output, record exit codes and measure test times
  • Supports additional custom fields, links and artifacts

Getting started

To get started with the Testmo CLI, you can simply install the @testmo/testmo-cli NPM package. Once you have installed the package, you can simply use the testmo tool from the command line. The following example shows how to install the CLI as a global tool on your system:
$ npm install -g @testmo/testmo-cli
$ testmo -h
Usage: testmo [options] [command]
Most systems and environments such as CI/CD containers should already be prepared to run npm and install packages. If your system is missing NPM/NodeJS, just follow their guide on how to install these tools.

How it works

Once you have installed the testmo tool, you can use it to submit your test automation results to your Testmo instance.
To submit your test results, you simply run your test automation tool with the option to export a result log in the standard JUnit XML file format. This file format is also used by many CI/CD tools and test reporting systems, so practically any test automation tool and framework supports this file format. And even in the unlikely case that your tool does not support this format directly, you can usually convert other formats to JUnit XML files easily.
After you have run your test automation tool and have generated the XML files, you use the testmo command to submit your results. You can simply pass one or multiple XML files and the tool automatically scans and detects all test results.
$ TESTMO_TOKEN=******** testmo automation:run:submit \
--instance https://<your-name>.testmo.net \
--project-id 1 \
--name "New test run" \
--source "backend-unit" \
--results reports/*.xml
You need an API key for your Testmo user (or for a separate API user) to authenticate and submit your test results. Learn more about generating API keys.
The testmo tool also supports launching your test automation tool directly to capture the console output, record exit codes and measure test times. See below for a full example.

Using the CLI on Windows

The Testmo CLI tool also fully works on Windows. After you install NPM and the Testmo CLI package as explained above, you can call testmo from the Windows cmd.exe command line like this (PowerShell uses yet another syntax, so use the following example with cmd.exe):
> set TESTMO_TOKEN=********
> testmo automation:run:submit ^
--instance https://<your-name>.testmo.net ^
--project-id 1 ^
--name "New test run" ^
--source "backend-unit" ^
--results reports/*.xml

Best practices and recipes

The testmo command line tool supports various scenarios and workflows. To make it easier to get started and use the best approach, we've included some concrete examples for typical use cases below. Also make sure to review the full command line reference for all options and features.

Submitting your first test run

To get familiar with using the CLI tool, start by submitting your first test run to Testmo. You just need a valid JUnit XML file with the test results generated by your test automation tool as well as a Testmo API key. If you are not yet familiar with Testmo's basic entities and terminology, please also review the automation overview article.
$ TESTMO_TOKEN=******** testmo automation:run:submit \
--instance https://<your-name>.testmo.net \
--project-id 1 \
--name "New test run" \
--source "backend-unit" \
--results reports/*.xml
The easiest way to submit your test results is to use the automation:run:submit command. This command accomplishes multiple things at once:
  • Collecting the test automation results from your XML file(s)
  • Adding a new test run in Testmo to the project you've specified
  • Adding a single new thread to the run (what's a thread?)
  • Submitting all test results to the thread
  • And finally completing the thread and test run
After the command has run, you can view the test run, its thread and all test results in Testmo. If the specified source didn't exist before, Testmo also automatically created it (learn more about grouping automation test runs into sources).
Finding result files recursively
The testmo command line tool also supports glob style file patterns when looking for XML result files. To use this capability, make sure to pass the pattern with quotes on the command line. For example, you can find all XML files in a directory recursively like this:
--results "reports/**/*.xml"

Record console output and test times

Often times it's also useful to capture the output of your test automation tool as well as the exit code. The testmo tool supports this by passing the full automation command line as a parameter. testmo then launches your automation tool, captures the output, records the full test time and finally, after you automation tool exits, collects the result files.
In order to launch your automation tool with testmo, simply pass it after specifying two dashes at the end of the command like this:
$ TESTMO_TOKEN=******** testmo automation:run:submit \
--instance https://<your-name>.testmo.net \
--project-id 1 \
--name "New test run" \
--source "backend-unit" \
--results reports/*.xml \
-- npm test <-- replace this line with your testing tool command line
^ space!
Note: it's important to include a space between the -- dashes and your automation tool command line. The full command and all options after the dashes are used to launch your automation tool.
Testmo and the CLI tool support custom fields, links and artifacts to be submitted as part of your test automation results. You can include these details for runs, threads and individual test results. For test results, the testmo tool automatically extracts any custom fields from your XML files. For threads and test runs you can submit additional details directly from the command line. Including such details can be useful in various situations:
  • Including a link to your CI/CD pipeline run
  • Including a link to the relevant Git commit
  • Linking to artifacts such as build files or screenshots
  • Including fields about the build environment
  • Adding version numbers, log output or build reports
The testmo command makes it very easy to include custom fields, artifacts and links for your threads and runs. To do this, you just specify a resource JSON file when creating, submitting or completing runs and threads. You could create such a JSON file yourself, but the testmo tool includes convenient commands to build this file from the command line:
$ testmo automation:resources:add-link \
--name "GitHub repository" \
--url "https://github.com/my-org/my-repo" \
--note "Our project's git repository with all code files and issues."
$ testmo automation:resources:add-field \
--type string \
--name "Version" \
--value "2.3.1-5fbcc8d0"
# For text, console and HTML fields, make sure to pipe the
# field value via stdin, e.g. from a file:
$ cat files.txt | testmo automation:resources:add-field \
--type console \
--name "File list"
By default the tool writes the fields and links to a file called testmo-resources.json in the current directory. You can then include these resources by specifying the file with the run and thread commands:
$ TESTMO_TOKEN=******** testmo automation:run:submit \
<other-options> \
--resources testmo-resources.json

Specifying fields in the XML files

For automation runs and threads, the easiest way to include additional fields is to use the above mentioned approach and submit the fields via resource files. If you would like to include additional fields for individual tests, the testmo CLI tool also supports property tags on a testcase level in XML files. Let's take a look at a simple example with just a single test case:
<testcase name="test">
<property name="language" value="English"/>
<property name="files" value="24"/>
<system-out>Standard output</system-out>
<system-err>Error output</system-err>
The testmo tool would parse this example as four fields (two fields from the properties list and two additional fields based on the output). Please note that the testmo tool would only transmit test-level fields for non-passing tests to improve performance and scalability.

Parallel test jobs in CI/CD pipelines

Testmo also fully supports advanced CI/CD workflows with parallel testing jobs. Testmo can capture the results for each parallel test job as part of a separate thread under a combined test run. You can then view the results of each thread separately as well as the entire test run at once. The following illustration shows how you would report parallel test jobs with the testmo tool from your CI/CD pipeline:
Example Testmo parallel testing strategy
So in order to integrate with and submit your parallel test jobs to Testmo, you would follow this approach:
  1. 1.
    Before you start your parallel test jobs, first call the automation:run:create command in your CI/CD pipeline. This creates a new test automation run in Testmo so you can add results. The command returns a test run ID. You need to store this ID and pass it to your parallel test jobs in your CI/CD pipeline, so you can add your results.
  2. 2.
    Then in each parallel test job you start your test automation tool to run your tests (you can either start it yourself or let the testmo tool launch your tests). You use the automation:run:submit-thread command to create a new thread and submit your results (and optionally launch your automation tool). You need to pass the above mentioned test run ID, so the new thread and results are added to the test run you created in step 1.
  3. 3.
    After all your parallel test jobs completed, mark the test run as completed by calling the automation:run:complete command. You again need to pass the test run ID created in step 1.
With each command, you can also optionally pass additional fields, links and artifacts (see above).
Examples for GitHub Actions, GitLab CI/CD, CircleCI and more
Ready to implement the Testmo integration with parallel test jobs in your CI/CD pipeline? We have ready-to-use examples, just see our CI/CD integration page to learn more.

Linking test runs to configurations and milestones

It's useful to link your test automation runs to configurations and milestones to make it easier to track, report and group your test runs. When you create a test automation run, either with the automation:run:submit or automation:run:create commands, you can specify a configuration and milestone.

Linking configurations

To link a test automation run to a configuration, you can either directly specify the configuration ID or use the configuration name:
  • You can specify a milestone ID with the --config-id option. IDs for configurations can be found in Testmo under Admin > Configurations.
  • Or you can specify (part of) a configuration name with the --config-name option and Testmo will try to find the relevant configuration based on its name.

Linking milestones

To link a run to a milestone, there are multiple options. You can specify the milestone ID, or specify the name, or use tags for automatic milestone linking:
  • You can specify the milestone ID with the --milestone-id option. The ID of a milestone can be found by clicking the small info icon next to the milestone name in the header section of the milestone's view page.
  • Alternatively you can specify the name of a milestone with the --milestone option. Testmo will then try to find the milestone based on the milestone name.
  • Or you can ask Testmo to automatically link test runs to milestones based on tags (with the --tags option). When you add or edit a milestone in Testmo, you can add one or more automation tags (e.g. latest, release-5 etc). Testmo will automatically link a run to the latest milestone that was created with a matching automation tag. This way you do not need to change your command line call (and thus your CI/CD config). You just add new milestones in Testmo with the same tag. New runs are then always automatically linked to the latest relevant milestone.

Handling test failures and exist codes

When you launch your test automation tool by passing it to the testmo command, our CLI tool automatically passes through the exit code by default. This way, when your test runs fail, the exit code is correctly reported to your CI/CD pipeline.
When one of your test runs fail, usually most CI/CD pipelines stop any subsequent jobs from running and mark the CI/CD pipeline as failed. With the Testmo integration it can be useful to still run the test-complete (or however you would like to call it) pipeline job on failure through to run the automation:run:complete job.
Some CI/CD tools such as GitHub Actions support such fine-grained pipeline configurations (see our example article). Others (such as CircleCI) don't support running specific jobs on failure. However, Testmo automatically detects this and completes test automation runs after a certain time without new results (this can be configured in Testmo under Admin > Automation).
Copy link
On this page
Design goals
Getting started
How it works
Using the CLI on Windows
Best practices and recipes
Submitting your first test run
Record console output and test times
Adding custom fields, links & artifacts
Parallel test jobs in CI/CD pipelines
Linking test runs to configurations and milestones
Handling test failures and exist codes