Testmo CLI

The Testmo CLI is a powerful command line tool to submit your test automation results to Testmo. It is the best way to collect and report your test results from your test automation workflow, CI/CD pipeline and build servers.

We designed the Testmo CLI to work with any test automation tool and framework so you can start using it and submitting your tests results in minutes.

The Testmo CLI tool is distributed as an NPM package so it's very easy to install and deploy it on any platform (e.g. Linux, Windows & macOS) and in any environment (build machines, dedicated servers, Docker containers, CI/CD pipelines etc.).

Quickstart guide

Send your first test automation results to Testmo in minutes by following our quickstart guide. Or continue below for all the details on our CLI tool.

Examples for GitHub, GitLab, CircleCI, Jenkins and more

We also have full examples on how to integrate Testmo with various CI/CD tools and workflows. See our CI/CD integration page to learn more.

Design goals

The Testmo CLI is used for and supports various use cases and scenarios:

Quickly submit your test automation results from the command line
Integrates with any CI/CD and build system such as GitHub, GitLab, Jenkins and more
Works with any test automation tool and framework by using standard JUnit XML report files
Supports both standard and advanced workflows such as parallel testing jobs
Can optionally capture full test console output, record exit codes and measure test times
Supports additional custom fields, links and artifacts

Installation & getting started

To get started with the Testmo CLI, you can simply install our official @testmo/testmo-cli NPM package. Once you have installed the package, you can simply use the testmo tool from the command line. The following example shows how to install (or update) the CLI as a global tool on your system:

$ npm install -g @testmo/testmo-cli
$ testmo -h

Usage: testmo [options] [command]
[...]

If you have already globally installed the Testmo CLI, you can update the package to the latest version as follows:

$ npm update -g @testmo/testmo-cli
$ testmo --version
1.x.x

Most systems and environments such as CI/CD containers usually already come with NPM installed. If your system is missing NPM/Node.js, just follow their guide on installing these tools.

Please note that you do not need to use Node.js/JavaScript yourself for your projects or as your automation programming language. We just use NPM to conveniently use & deploy the CLI tool on any platform. You can use Testmo with any platform, including Java, .NET, Python, JavaScript, C++ and many more.

Submitting your first test run

Once you have installed the testmo tool, you can use it to submit your test automation results to your Testmo instance.

To submit your test results, you simply run your test automation tool with the option to export the results to a standard JUnit-style XML file. This file format is a standard format used to exchange test results between testing tools and is also used by many CI/CD services. We also have many examples on how to generate such report files for many automation tools.

After you have run your test automation tool and have generated the XML files, you use the testmo command to submit your results. You can simply pass one or multiple XML files and the tool automatically scans and detects all test results and submits them to Testmo.

$ export TESTMO_TOKEN=********
$ testmo automation:run:submit \
  --instance https://<your-name>.testmo.net \
  --project-id 1 \
  --name "Test run" \
  --source "frontend" \
  --results results/*.xml

If you run this example on a single command line, make sure to remove the backspaces:

$ export TESTMO_TOKEN=********
$ testmo automation:run:submit --instance https://<your-name>.testmo.net --project-id 1 --name "Test run" --source "frontend" --results results/*.xml

On Windows, the command line uses a slightly different syntax to set a variable:

> set TESTMO_TOKEN=********
> testmo automation:run:submit --instance https://<your-name>.testmo.net --project-id 1 --name "Test run" --source "frontend" --results results/*.xml

You need an API key for your Testmo user (or for a separate API user) to authenticate and submit your test results. Learn more about generating API keys.

The testmo tool also supports launching your test automation tool directly to capture the console output, record exit codes and measure test times. See below for a full example.

Finding result files recursively

The testmo command line tool also supports glob style file patterns when looking for XML result files. To use this capability, make sure to pass the pattern with quotes on the command line. For example, you can find all XML files in a directory recursively like this:

--results "reports/**/*.xml"

Learn about Testmo automation concepts

To learn about the basic automation concepts in Testmo such as runs, sources and threads, take a look a our automation concepts guide.

Record console output and test times

Often times it's also useful to capture the output of your test automation tool as well as the exit code. The testmo tool supports this by passing the full automation command line as a parameter. testmo then launches your automation tool, captures the output, records the full test time and finally, after you automation tool exits, collects the result files.

In order to launch your automation tool with testmo, simply pass it after specifying two dashes at the end of the command like this:

$ export TESTMO_TOKEN=********
$ testmo automation:run:submit \
    --instance https://<your-name>.testmo.net \
    --project-id 1 \
    --name "New test run" \
    --source "backend-unit" \
    --results reports/*.xml \
    -- npm test <-- replace this line with your testing tool command line
      ^ space!

Note: it's important to include a space between the -- dashes and your automation tool command line. The full command and all options after the dashes are used to launch your automation tool.

Linking to configurations and milestones

It's useful to link your test automation runs to configurations and milestones to make it easier to track, report and group your test runs. When you create a test automation run, either with the automation:run:submit or automation:run:create commands, you can specify a configuration and milestone.

Linking configurations

To link a test automation run to a configuration, you can either directly specify the configuration ID or use the configuration name:

You can specify a milestone ID with the --config-id option. IDs for configurations can be found in Testmo under Admin > Configurations.
Or you can specify (part of) a configuration name with the --config option and Testmo will try to find the relevant configuration based on its name.

Linking milestones

To link a run to a milestone, there are multiple options. You can specify the milestone ID, or specify the name, or use tags for automatic milestone linking:

You can specify the milestone ID with the --milestone-id option. The ID of a milestone can be found by clicking the small info icon next to the milestone name in the header section of the milestone's view page.
Alternatively you can specify the name of a milestone with the --milestone option. Testmo will then try to find the milestone based on the milestone name.
Or you can ask Testmo to automatically link test runs to milestones based on tags (with the --tags option). When you add or edit a milestone in Testmo, you can add one or more automation tags (e.g. latest, release-5 etc). Testmo will automatically link a run to the latest milestone that was created with a matching automation tag. This way you do not need to change your command line call (and thus your CI/CD config). You just add new milestones in Testmo with the same tag. New runs are then always automatically linked to the latest relevant milestone.

Handling test failures and exist codes

When you launch your test automation tool by passing it to the testmo command, our CLI tool automatically passes through the exit code by default. This way, when your test runs fail, the exit code is correctly reported to your CI/CD pipeline.

When one of your test runs fail, usually most CI/CD pipelines stop any subsequent jobs from running and mark the CI/CD pipeline as failed. With the Testmo integration it can be useful to still run the test-complete (or however you would like to call it) pipeline job on failure through to run the automation:run:complete job.

Some CI/CD tools such as GitHub Actions support such fine-grained pipeline configurations (see our example article). Others (such as CircleCI) don't support running specific jobs on failure. However, Testmo automatically detects this and completes test automation runs after a certain time without new results (this can be configured in Testmo under Admin > Automation).

PreviousCI/CD Integration NextReference

Last updated 11 months ago