Testmo CLI

The Testmo CLI is a powerful command line tool to submit your test automation results to Testmo. It is the best way to collect and report your test results from your test automation workflow, CI/CD pipeline and build servers.
We designed the Testmo CLI to work with any test automation tool and framework so you can start using it and submitting your tests results in minutes.
The Testmo CLI tool is distributed as an NPM package so it's very easy to install and deploy it on any platform (e.g. Linux, Windows & macOS) and in any environment (build machines, dedicated servers, Docker containers, CI/CD pipelines etc.).
Quickstart guide
Send your first test automation results to Testmo in minutes by following our quickstart guide. Or continue below for all the details on our CLI tool.
Examples for GitHub, GitLab, CircleCI, Jenkins and more
We also have full examples on how to integrate Testmo with various CI/CD tools and workflows. See our CI/CD integration page to learn more.
Design goals
The Testmo CLI is used for and supports various use cases and scenarios:
Quickly submit your test automation results from the command line
Integrates with any CI/CD and build system such as GitHub, GitLab, Jenkins and more
Works with any test automation tool and framework by using standard JUnit XML report files
Supports both standard and advanced workflows such as parallel testing jobs
Can optionally capture full test console output, record exit codes and measure test times
Supports additional custom fields, links and artifacts
Installation & getting started
To get started with the Testmo CLI, you can simply install our official @testmo/testmo-cli NPM package. Once you have installed the package, you can simply use the testmo tool from the command line. The following example shows how to install the CLI as a global tool on your system:
$ npm install -g @testmo/testmo-cli
$ testmo -h
​
Usage: testmo [options] [command]
[...]
Most systems and environments such as CI/CD containers usually already come with NPM installed. If your system is missing NPM/NodeJS, just follow their guide on installing these tools.
Please note that you do not need to use NodeJS/JavaScript yourself for your projects or as your automation programming language. We just use NPM to conveniently use & deploy the CLI tool on any platform. You can use Testmo with any platform, including Java, .NET, Python, JavaScript, C++ and many more.
Submitting your first test run
Once you have installed the testmo tool, you can use it to submit your test automation results to your Testmo instance.
To submit your test results, you simply run your test automation tool with the option to export the results to a standard JUnit-style XML file. This file format is a standard format used to exchange test results between testing tools and is also used by many CI/CD services. We also have many examples on how to generate such report files for many automation tools.
After you have run your test automation tool and have generated the XML files, you use the testmo command to submit your results. You can simply pass one or multiple XML files and the tool automatically scans and detects all test results and submits them to Testmo.
Full example
Single line
Windows
$ export TESTMO_TOKEN=********
$ testmo automation:run:submit \
  --instance https://<your-name>.testmo.net \
  --project-id 1 \
  --name 'Test run' \
  --source 'frontend' \
  --results results/*.xml
If you run this example on a single command line, make sure to remove the backspaces:
$ export TESTMO_TOKEN=********
$ testmo automation:run:submit --instance https://<your-name>.testmo.net --project-id 1 --name 'Test run' --source 'frontend' --results results/*.xml
On Windows, the command line uses a slightly different syntax to set a variable:
> set TESTMO_TOKEN=********
> testmo automation:run:submit --instance https://<your-name>.testmo.net --project-id 1 --name 'Test run' --source 'frontend' --results results/*.xml
You need an API key for your Testmo user (or for a separate API user) to authenticate and submit your test results. Learn more about generating API keys.
The testmo tool also supports launching your test automation tool directly to capture the console output, record exit codes and measure test times. See below for a full example.
Finding result files recursively
The testmo command line tool also supports glob style file patterns when looking for XML result files. To use this capability, make sure to pass the pattern with quotes on the command line. For example, you can find all XML files in a directory recursively like this:
--results "reports/**/*.xml"
Learn about Testmo automation concepts
To learn about the basic automation concepts in Testmo such as runs, sources and threads, take a look a our automation concepts guide.
Record console output and test times
Often times it's also useful to capture the output of your test automation tool as well as the exit code. The testmo tool supports this by passing the full automation command line as a parameter. testmo then launches your automation tool, captures the output, records the full test time and finally, after you automation tool exits, collects the result files.
In order to launch your automation tool with testmo, simply pass it after specifying two dashes at the end of the command like this:
$ export TESTMO_TOKEN=********
$ testmo automation:run:submit \
    --instance https://<your-name>.testmo.net \
    --project-id 1 \
    --name "New test run" \
    --source "backend-unit" \
    --results reports/*.xml \
    -- npm test <-- replace this line with your testing tool command line
      ^ space!
Note: it's important to include a space between the -- dashes and your automation tool command line. The full command and all options after the dashes are used to launch your automation tool.
Custom fields, links & artifacts
Testmo and the CLI tool support custom fields, links and artifacts to be submitted as part of your test automation results. You can include these details for runs, threads and individual test results. For test results, the testmo tool automatically extracts any custom fields from your XML files. For threads and test runs you can submit additional details directly from the command line. Including such details can be useful in various situations:
Including a link to your CI/CD pipeline run
Including a link to the relevant Git commit
Linking to artifacts such as build files or screenshots
Including fields about the build environment
Adding version numbers, log output or build reports
The testmo command makes it very easy to include custom fields, artifacts and links for your threads and runs. To do this, you just specify a resource JSON file when creating, submitting or completing runs and threads. You could create such a JSON file yourself, but the testmo tool includes convenient commands to build this file from the command line:
$ testmo automation:resources:add-link \
  --name "GitHub repository" \
  --url "https://github.com/my-org/my-repo" \
  --note "Our project's git repository with all code files and issues."
​
$ testmo automation:resources:add-field \
  --type string \
  --name "Version" \
  --value "2.3.1-5fbcc8d0"
​
# For text, console and HTML fields, make sure to pipe the
# field value via stdin, e.g. from a file:
​
$ cat files.txt | testmo automation:resources:add-field \
  --type console \
  --name "File list"
By default the tool writes the fields and links to a file called testmo-resources.json in the current directory. You can then include these resources by specifying the file with the run and thread commands:
$ export TESTMO_TOKEN=********
$ testmo automation:run:submit \
    <other-options> \
    --resources testmo-resources.json
Specifying fields in the XML files
For automation runs and threads, the easiest way to include additional fields is to use the above mentioned approach and submit the fields via resource files. If you would like to include additional fields for individual tests, the testmo CLI tool also supports property tags on a testcase level in XML files. Let's take a look at a simple example with just a single test case:
<testsuites>
    <testcase name="test">
        <properties>
          <property name="language" value="English"/>
          <property name="files" value="24"/>
        </properties>
        <system-out>Standard output</system-out>
        <system-err>Error output</system-err>
    </testcase>
</testsuites>
The testmo tool would parse this example as four fields (two fields from the properties list and two additional fields based on the output). Please note that the testmo tool would only transmit test-level fields for non-passing tests to improve performance and scalability.
Parallel test jobs in CI/CD pipelines
Testmo also fully supports advanced CI/CD workflows with parallel testing jobs. Testmo can capture the results for each parallel test job as part of a separate thread under a combined test run. You can then view the results of each thread separately as well as the entire test run at once. The following illustration shows how you would report parallel test jobs with the testmo tool from your CI/CD pipeline:
Example Testmo parallel testing strategy
So in order to integrate with and submit your parallel test jobs to Testmo, you would follow this approach:
1.Before you start your parallel test jobs, first call the automation:run:create command in your CI/CD pipeline. This creates a new test automation run in Testmo so you can add results. The command returns a test run ID. You need to store this ID and pass it to your parallel test jobs in your CI/CD pipeline, so you can add your results.
2.Then in each parallel test job you start your test automation tool to run your tests (you can either start it yourself or let the testmo tool launch your tests). You use the automation:run:submit-thread command to create a new thread and submit your results (and optionally launch your automation tool). You need to pass the above mentioned test run ID, so the new thread and results are added to the test run you created in step 1.
3.After all your parallel test jobs completed, mark the test run as completed by calling the automation:run:complete command. You again need to pass the test run ID created in step 1.
With each command, you can also optionally pass additional fields, links and artifacts (see above).
Examples for GitHub Actions, GitLab CI/CD, CircleCI and more
Ready to implement the Testmo integration with parallel test jobs in your CI/CD pipeline? We have ready-to-use examples, just see our CI/CD integration page to learn more.
Linking to configurations and milestones
It's useful to link your test automation runs to configurations and milestones to make it easier to track, report and group your test runs. When you create a test automation run, either with the automation:run:submit or automation:run:create commands, you can specify a configuration and milestone.
Linking configurations
To link a test automation run to a configuration, you can either directly specify the configuration ID or use the configuration name:
You can specify a milestone ID with the --config-id option. IDs for configurations can be found in Testmo under Admin > Configurations.
Or you can specify (part of) a configuration name with the --config-name option and Testmo will try to find the relevant configuration based on its name.
Linking milestones
To link a run to a milestone, there are multiple options. You can specify the milestone ID, or specify the name, or use tags for automatic milestone linking:
You can specify the milestone ID with the --milestone-id option. The ID of a milestone can be found by clicking the small info icon next to the milestone name in the header section of the milestone's view page.
Alternatively you can specify the name of a milestone with the --milestone option. Testmo will then try to find the milestone based on the milestone name.
Or you can ask Testmo to automatically link test runs to milestones based on tags (with the --tags option). When you add or edit a milestone in Testmo, you can add one or more automation tags (e.g. latest, release-5 etc). Testmo will automatically link a run to the latest milestone that was created with a matching automation tag. This way you do not need to change your command line call (and thus your CI/CD config). You just add new milestones in Testmo with the same tag. New runs are then always automatically linked to the latest relevant milestone.
Handling test failures and exist codes
When you launch your test automation tool by passing it to the testmo command, our CLI tool automatically passes through the exit code by default. This way, when your test runs fail, the exit code is correctly reported to your CI/CD pipeline.
When one of your test runs fail, usually most CI/CD pipelines stop any subsequent jobs from running and mark the CI/CD pipeline as failed. With the Testmo integration it can be useful to still run the test-complete (or however you would like to call it) pipeline job on failure through to run the automation:run:complete job.
Some CI/CD tools such as GitHub Actions support such fine-grained pipeline configurations (see our example article). Others (such as CircleCI) don't support running specific jobs on failure. However, Testmo automatically detects this and completes test automation runs after a certain time without new results (this can be configured in Testmo under Admin > Automation).
Last modified 13d ago