Testmo CLI

The Testmo CLI is a powerful command line tool to submit your test automation results to Testmo. It is the best way to collect and report your test results from your test automation workflow, CI/CD pipeline and build servers.
We designed the Testmo CLI to work with practically any test automation tool and framework so you can start using it and submitting your tests results quickly.
The Testmo CLI is distributed as an NPM package so it's very easy to install and deploy it on any system (e.g. Linux, Windows & macOS) and in any environment (build machines, dedicated servers, Docker containers, CI/CD processes etc.).
Examples for GitHub Actions, GitLab CI/CD, CircleCI, Jenkins and more
We also have full examples on how to integrate Testmo with various CI/CD tools and workflows. See our CI/CD integration page to learn more.
Design goals
The Testmo CLI is used for and supports various use cases and scenarios:
Quickly submit your test automation results from the command line
Easy integration with any CI/CD and build system such as GitHub, GitLab, Jenkins, CircleCI, Bitbucket and more
Works with any test automation tool and framework by using standard JUnit XML report files
Supports both basic and advanced workflows such as parallel testing jobs
Can optionally capture full test console output, record exit codes and measure test times
Supports additional custom fields, links and artifacts
Getting started
To get started with the Testmo CLI, you can simply install the @testmo/testmo-cli NPM package. Once you have installed the package, you can simply use the testmo tool from the command line. The following example shows how to install the CLI as a global tool on your system:
1
$ npm install -g @testmo/testmo-cli
2
$ testmo -h
3
​
4
Usage: testmo [options] [command]
5
[...]
Copied!
Most systems and environments such as CI/CD containers should already be prepared to run npm and install packages. If your system is missing NPM/NodeJS, just follow their guide on how to install these tools.
How it works
Once you have installed the testmo tool, you can use it to submit your test automation results to your Testmo instance.
To submit your test results, you simply run your test automation tool with the option to export a result log in the standard JUnit XML file format. This file format is also used by many CI/CD tools and test reporting systems, so practically any test automation tool and framework supports this file format. And even in the unlikely case that your tool does not support this format directly, you can usually convert other formats to JUnit XML files easily.
After you have run your test automation tool and have generated the XML files, you use the testmo command to submit your results. You can simply pass one or multiple XML files and the tool automatically scans and detects all test results.
1
$ TESTMO_TOKEN=******** testmo automation:run:submit \
2
    --instance https://<your-name>.testmo.net \
3
    --project-id 1 \
4
    --name "New test run" \
5
    --source "backend-unit" \
6
    --results reports/*.xml
Copied!
You need an API key for your Testmo user (or for a separate API user) to authenticate and submit your test results. Learn more about generating API keys.
The testmo tool also supports launching your test automation tool directly to capture the console output, record exit codes and measure test times. See below for a full example.
Best practices and recipes
The testmo command line tool supports various scenarios and workflows. To make it easier to get started and use the best approach, we've included some concrete examples for typical use cases below. Also make sure to review the full command line reference for all options and features.
Submitting your first test run
To get familiar with using the CLI tool, start by submitting your first test run to Testmo. You just need a valid JUnit XML file with the test results generated by your test automation tool as well as a Testmo API key. If you are not yet familiar with Testmo's basic entities and terminology, please also review the automation overview article.
1
$ TESTMO_TOKEN=******** testmo automation:run:submit \
2
    --instance https://<your-name>.testmo.net \
3
    --project-id 1 \
4
    --name "New test run" \
5
    --source "backend-unit" \
6
    --results reports/*.xml
Copied!
The easiest way to submit your test results is to use the automation:run:submit command. This command accomplishes multiple things at once:
Collecting the test automation results from your XML file(s)
Adding a new test run in Testmo to the project you've specified
Adding a single new thread to the run (what's a thread?)
Submitting all test results to the thread
And finally completing the thread and test run
After the command has run, you can view the test run, its thread and all test results in Testmo. If the specified source didn't exist before, Testmo also automatically created it (learn more about grouping automation test runs into sources).
Finding result files recursively
The testmo command line tool also supports glob style file patterns when looking for XML result files. To use this capability, make sure to pass the pattern with quotes on the command line. For example, you can find all XML files in a directory recursively like this:
--results "reports/**/*.xml"
Record console output and test times
Often times it's also useful to capture the output of your test automation tool as well as the exit code. The testmo tool supports this by passing the full automation command line as a parameter. testmo then launches your automation tool, captures the output, records the full test time and finally, after you automation tool exits, collects the result files.
In order to launch your automation tool with testmo, simply pass it after specifying two dashes at the end of the command like this:
1
$ TESTMO_TOKEN=******** testmo automation:run:submit \
2
    --instance https://<your-name>.testmo.net \
3
    --project-id 1 \
4
    --name "New test run" \
5
    --source "backend-unit" \
6
    --results reports/*.xml \
7
    -- npm test <-- replace this line with your testing tool command line
8
      ^ space!
Copied!
Note: it's important to include a space between the -- dashes and your automation tool command line. The full command and all options after the dashes are used to launch your automation tool.
Adding custom fields, links & artifacts
Testmo and the CLI tool support custom fields, links and artifacts to be submitted as part of your test automation results. You can include these details for runs, threads and individual test results. For test results, the testmo tool automatically extracts any custom fields from your XML files. For threads and test runs you can submit additional details directly from the command line. Including such details can be useful in various situations:
Including a link to your CI/CD pipeline run
Including a link to the relevant Git commit
Linking to artifacts such as build files or screenshots
Including fields about the build environment
Adding version numbers, log output or build reports
The testmo command makes it very easy to include custom fields, artifacts and links for your threads and runs. To do this, you just specify a resource JSON file when creating, submitting or completing runs and threads. You could create such a JSON file yourself, but the testmo tool includes convenient commands to build this file from the command line:
1
$ testmo automation:resources:add-link \
2
  --name "GitHub repository" \
3
  --url "https://github.com/my-org/my-repo" \
4
  --note "Our project's git repository with all code files and issues."
5
​
6
$ testmo automation:resources:add-field \
7
  --type string \
8
  --name "Version" \
9
  --value "2.3.1-5fbcc8d0"
10
​
11
# For text, console and HTML fields, make sure to pipe the
12
# field value via stdin, e.g. from a file:
13
​
14
$ cat files.txt | testmo automation:resources:add-field \
15
  --type console \
16
  --name "File list"
Copied!
By default the tool writes the fields and links to a file called testmo-resources.json in the current directory. You can then include these resources by specifying the file with the run and thread commands:
1
$ TESTMO_TOKEN=******** testmo automation:run:submit \
2
      <other-options> \
3
      --resources testmo-resources.json
Copied!
Parallel test jobs in CI/CD pipelines
Testmo also fully supports advanced CI/CD workflows with parallel testing jobs. Testmo can capture the results for each parallel test job as part of a separate thread under a combined test run. You can then view the results of each thread separately as well as the entire test run at once. The following illustration shows how you would report parallel test jobs with the testmo tool from your CI/CD pipeline:
Example Testmo parallel testing strategy
So in order to integrate with and submit your parallel test jobs to Testmo, you would follow this approach:
1.Before you start your parallel test jobs, first call the automation:run:create command in your CI/CD pipeline. This creates a new test automation run in Testmo so you can add results. The command returns a test run ID. You need to store this ID and pass it to your parallel test jobs in your CI/CD pipeline, so you can add your results.
2.Then in each parallel test job you start your test automation tool to run your tests (you can either start it yourself or let the testmo tool launch your tests). You use the automation:run:submit-thread command to create a new thread and submit your results (and optionally launch your automation tool). You need to pass the above mentioned test run ID, so the new thread and results are added to the test run you created in step 1.
3.After all your parallel test jobs completed, mark the test run as completed by calling the automation:run:complete command. You again need to pass the test run ID created in step 1.
With each command, you can also optionally pass additional fields, links and artifacts (see above).
Examples for GitHub Actions, GitLab CI/CD, CircleCI and more
Ready to implement the Testmo integration with parallel test jobs in your CI/CD pipeline? We have ready-to-use examples, just see our CI/CD integration page to learn more.
Linking test runs to configurations and milestones
It's useful to link your test automation runs to configurations and milestones to make it easier to track, report and group your test runs. When you create a test automation run, either with the automation:run:submit or automation:run:create commands, you can specify a configuration and milestone.
Linking configurations
To link a test automation run to a configuration, you can either directly specify the configuration ID or use the configuration name:
You can specify a milestone ID with the --config-id option. IDs for configurations can be found in Testmo under Admin > Configurations.
Or you can specify (part of) a configuration name with the --config-name option and Testmo will try to find the relevant configuration based on its name.
Linking milestones
To link a run to a milestone, there are multiple options. You can specify the milestone ID, or specify the name, or use tags for automatic milestone linking:
You can specify the milestone ID with the --milestone-id option. The ID of a milestone can be found by clicking the small info icon next to the milestone name in the header section of the milestone's view page.
Alternatively you can specify the name of a milestone with the --milestone option. Testmo will then try to find the milestone based on the milestone name.
Or you can ask Testmo to automatically link test runs to milestones based on tags (with the --tags option). When you add or edit a milestone in Testmo, you can add one or more automation tags (e.g. latest, release-5 etc). Testmo will automatically link a run to the latest milestone that was created with a matching automation tag. This way you do not need to change your command line call (and thus your CI/CD config). You just add new milestones in Testmo with the same tag. New runs are then always automatically linked to the latest relevant milestone.
Handling test failures and exist codes
When you launch your test automation tool by passing it to the testmo command, our CLI tool automatically passes through the exit code by default. This way, when your test runs fail, the exit code is correctly reported to your CI/CD pipeline.
When one of your test runs fail, usually most CI/CD pipelines stop any subsequent jobs from running and mark the CI/CD pipeline as failed. With the Testmo integration it can be useful to still run the test-complete (or however you would like to call it) pipeline job on failure through to run the automation:run:complete job.
Some CI/CD tools such as GitHub Actions support such fine-grained pipeline configurations (see our example article). Others (such as CircleCI) don't support running specific jobs on failure. However, Testmo automatically detects this and completes test automation runs after a certain time without new results (this can be configured in Testmo under Admin > Automation).
Automation - Previous
CI/CD Integration
Next - Automation
Reference
Last modified 11d ago
Copy link
Contents
Design goals
Getting started
How it works
Best practices and recipes
Submitting your first test run
Record console output and test times
Adding custom fields, links & artifacts
Parallel test jobs in CI/CD pipelines
Linking test runs to configurations and milestones
Handling test failures and exist codes