Reference
The testmo command line tool provides various commands to make it easier to submit and work with your test automation suites and CI/CD pipelines and implement different use cases. Make sure to review the available options and settings to learn about all its capabilities. If you are missing specific options or features, make sure to get in touch to provide feedback.
$ testmo help
Usage: testmo [options] [command]
Options:
-V, --version Output the version number
--ansi Force ANSI console output (colors,
animations)
--no-ansi Do not use ANSI console output (colors,
animations)
--debug Output additional debug information
-h, --help Display help for command
Commands:
automation:resources:add-artifact Adds a new artifact to the local resource
file to be submitted with an automation run
automation:resources:add-field Adds a new field to the local resource file
to be submitted with an automation run
automation:resources:add-link Adds a new link to the local resource file
to be submitted with an automation run
automation:run:complete Marks a test automation run as completed
automation:run:create Creates a new automation run for parallel
execution
automation:run:submit Creates a new test automation run, submits
test results and completes the run in a
single step
automation:run:submit-thread Creates a new thread for an existing run,
submits test results and completes the
thread
help Display help for command
Submitting entire test runs
To create a new test run, submit all test results and complete the run in a single command, use the automation:run:submit command. This is best used for simple test automation workflows without parallel tests.
$ testmo help automation:run:submit
Usage: testmo automation:run:submit [options] -- [executable]
Creates a new test automation run, submits test results and completes the run
in a single step
Options:
--instance <url> Required: The full address of your Testmo instance
(https://***.testmo.net)
--project-id <id> Required: The ID of the project this run is added to
--name <name> Required: The name of the newly created test run
--source <source> Required: The name of the source for this run and
future runs of this same source/suite
--results <files...> Required: Result log files (in JUnit XML format). If
you pass a single quoted parameter, it is evaluated
by Node's `glob` to resolve files (allowing recursive
search, see example below)
--tags <tags...> Tags for the new run. Tags can also be used to
automatically link this run to milestones
--config <name> Automatically find an existing configuration for this
run based on the config name
--config-id <id> Or specify a configuration by using the config's ID
--milestone <name> Automatically find an existing milestone for this run
based on the milestone name
--milestone-id <id> Or specify a milestone by using the milestone's ID
--resources <file> An optional JSON file with custom fields, links and
artifacts (can be built with the automation:resources
commands)
--results-rewriter <file> An optional JavaScript file to rewrite & transform
tests before sending them to Testmo
--ignore-parse-error Ignore result file parsing errors and upload
remaining successfully parsed files (default: false)
--exec-suppress Suppress standard output (stdout) of passed executable
(default: false)
--exec-stop-error Stop and don't create a run if the passed executable
failed (exit code not 0) (default: false)
--ansi Force ANSI console output (colors, animations)
--no-ansi Do not use ANSI console output (colors, animations)
--debug Output additional debug information
-h, --help Display help for command
Details:
Use this command to create a new test automation run, submit test results and
complete the run in a single step. That is, use this command if you don't run
your tests in multiple parallel threads. There are two typical ways to use
this command:
- Run this command after your tests finished and after your test result
reports were generated. This command then uploads your test results.
$ testmo automation:run:submit [options]
OR
- You can alternatively pass the executable/command line to run your tests
to this command. The command then also captures the output of your tests
and measures the exact time. The output and time are then also sent to
Testmo.
$ testmo automation:run:submit \
[options] -- <your-test-run-command> [options]
In both scenarios the command expects that your test results are generated and
reported as JUnit XML files. Practically any test automation tool can report
or convert test results to this format. The list of report files can be
specified with the --results option:
testmo automation:run:submit [options] --results reports/*.xml
It is also possible to use recursive search by specifying a quoted single
option to be evaluated by 'glob' (don't forget the quotes):
testmo automation:run:submit [options] --results "reports/**/*.xml"
The command automatically detects if any found files look like a valid JUnit
XML report and ignores other files. By default, detected JUnit XML files that
fail to parse throw an error.
Make sure to set the API authentication token as TESTMO_TOKEN environment
variable. You can either add an authentication token for your user in your
profile settings in Testmo. Or your Testmo admin can add a separate API user
just for authentication.
Examples:
You can create a test automation run and submit test results after your
test run:
$ TESTMO_TOKEN=******** testmo automation:run:submit \
--instance https://<your-name>.testmo.net \
--project-id 1 \
--name "New test run" \
--source "backend-unit" \
--results reports/*.xml
You can additionally pass the executable/command line for your test automation
tool as the last parameter after the special -- option. The command then
executes your tool and captures the full output and measures the execution
time. Both are included in the result test automation run in Testmo (note the
space after --):
$ TESTMO_TOKEN=******** testmo automation:run:submit \
--instance https://<your-name>.testmo.net \
--project-id 1 \
--name "New test run" \
--source "backend-unit" \
--results reports/*.xml \
-- npm test <-- replace this line with your testing tool command line
^ space!
To also include any additional resources previously stored in a file with the
automation:resources:* commands, such as custom fields, links and artifacts,
specify the resource file with the --resources option:
$ TESTMO_TOKEN=******** testmo automation:run:submit \
<other-options> \
--resources testmo-resources.jsonCreating and completing test runs
If you would like to create multiple threads for e.g. parallel tests, you would first create a test run, then submit the threads (see below) and finally complete the run. You would use the automation:run:create and automation:run:complete commands for this.
$ testmo help automation:run:create
Usage: testmo automation:run:create [options]
Creates a new automation run for parallel execution
Options:
--instance <url> Required: The full address of your Testmo instance
(https://***.testmo.net)
--project-id <id> Required: The ID of the project this run is added to
--name <name> Required: The name of the newly created test run
--source <source> Required: The name of the source for this run and future
runs of this same source/suite
--tags <tags...> Tags for the new run. Tags can also be used to
automatically link this run to milestones
--config <name> Automatically find an existing configuration for this
run based on the config name
--config-id <id> Or specify a configuration by using the config's ID
--milestone <name> Automatically find an existing milestone for this run
based on the milestone name
--milestone-id <id> Or specify a milestone by using the milestone's ID
--resources <file> An optional JSON file with custom fields, links and
artifacts (can be built with the automation:resources
commands)
--ansi Force ANSI console output (colors, animations)
--no-ansi Do not use ANSI console output (colors, animations)
--debug Output additional debug information
-h, --help Display help for command
Details:
Use this command to create a new automation run you intend to use for parallel
execution, e.g. as part of parallel CI tasks. If successful, this command
outputs the test automation run ID.
You will need to store this ID and use it for subsequent calls to
automation:run:submit-thread and automation:run:complete. You would usually
use these commands for parallel execution of automated tests:
automation:run:create <-- Create the run before your tests start
| automation:run:submit-thread <-- Submit results in your parallel CI tasks
| automation:run:submit-thread <-- Submit results in your parallel CI tasks
| automation:run:submit-thread <-- Submit results in your parallel CI tasks
automation:run:complete <-- Mark run as completed at the end of
your CI
If you just want to create an automation run and submit results without
parallel execution, just use automation:run:submit without calling
create/complete (not submit-thread!).
Make sure to set the API authentication token as TESTMO_TOKEN environment
variable. You can either add an authentication token for your user in your
profile settings in Testmo. Or your Testmo admin can add a separate API user
just for authentication.
Examples:
The following example creates a new automation run:
$ TESTMO_TOKEN=******** testmo automation:run:create \
--instance https://<your-name>.testmo.net \
--project-id 1 \
--name "New test run" \
--source "backend-unit"
To also include any additional resources previously stored in a file with the
automation:resources:* commands, such as custom fields, links and artifacts,
specify the resource file with the --resources option:
$ TESTMO_TOKEN=******** testmo automation:run:create \
<other-options> \
--resources testmo-resources.json$ testmo help automation:run:complete
Usage: testmo automation:run:complete [options]
Marks a test automation run as completed
Options:
--instance <url> Required: The full address of your Testmo instance
(https://***.testmo.net)
--run-id <id> Required: The ID of the run to be completed (returned
by automation:run:create)
--no-measure-elapsed By default Testmo measures the run time by looking at
the time difference between create and complete. This
option disables this behavior. The test run time is then
based on the longest running thread
--resources <file> An optional JSON file with custom fields, links and
artifacts (can be built with the automation:resources
commands)
--ansi Force ANSI console output (colors, animations)
--no-ansi Do not use ANSI console output (colors, animations)
--debug Output additional debug information
-h, --help Display help for command
Details:
Use this command to mark a test automation run you previously started with
automation:run:create as completed. Once the test run has been completed, it
cannot receive additional threads and test results.
You should call this command after all your CI test tasks have been completed
for this run and after your last call to automation:run:submit-thread.
Please note that you do not need to complete a run you have started with
automation:run:submit. It is automatically completed after the results have
been uploaded.
By default Testmo measures the run time by looking at the time difference
between create and complete. In most situations this is the recommended
approach as it results in accurate automation time measurement. You can
use the --no-measure-elapsed option to disable this behavior. The time of
the test run is then based on the longest running thread.
Make sure to set the API authentication token as TESTMO_TOKEN environment
variable. You can either add an authentication token for your user in your
profile settings in Testmo. Or your Testmo admin can add a separate API user
just for authentication.
Examples:
The following example completes a running automation run:
$ TESTMO_TOKEN=******** testmo automation:run:complete \
--instance https://<your-name>.testmo.net \
--run-id 1
To also include any additional resources previously stored in a file with the
automation:resources:* commands, such as custom fields, links and artifacts,
specify the resource file with the --resources option:
$ TESTMO_TOKEN=******** testmo automation:run:complete \
<other-options> \
--resources testmo-resources.jsonSubmitting threads for runs
If you create test runs with the above automation:run:create command, you can submit individual threads and test results with the automation:run:submit-thread command. You would usually use this approach for more advanced CI/CD pipeline workflows with parallel test jobs.
$ testmo help automation:run:submit-thread
Usage: testmo automation:run:submit-thread [options] -- [executable]
Creates a new thread for an existing run, submits test results and completes
the thread
Options:
--instance <url> Required: The full address of your Testmo instance
(https://***.testmo.net)
--run-id <id> Required: The ID of the run this new thread should
be added to (returned by automation:run:create)
--results <files...> Required: Result log files (in JUnit XML format).
If you pass a single quoted parameter, it is
evaluated by Node's `glob` to resolve files
(allowing recursive search, see example below)
--run-resources <file> An optional JSON file with custom fields, links and
artifacts (can be built with the
automation:resources commands) for the existing run
--thread-resources <file> An optional JSON file with custom fields and
artifacts (can be built with the
automation:resources commands) for the new thread
--results-rewriter <file> An optional JavaScript file to rewrite & transform
tests before sending them to Testmo
--ignore-parse-error Ignore result file parsing errors and upload
remaining successfully parsed files
(default: false)
--exec-suppress Suppress standard output (stdout) of passed
executable (default: false)
--exec-stop-error Stop and don't create a run if the passed
executable failed (exit code not 0)
(default: false)
--ansi Force ANSI console output (colors, animations)
--no-ansi Do not use ANSI console output (colors, animations)
--debug Output additional debug information
-h, --help Display help for command
Details:
Use this command to add a new thread to an existing running test automation
run, submit test results and complete the thread in a single step. You would
use this command to submit the results of a single (parallel) CI task to an
active run.
If you just want to create a test run with a single thread (i.e. a
non-parallel single CI task), you would just use the automation:run:submit
task instead.
There are two typical ways to use this command:
- Run this command in your parallel CI task after your tests finished and
after your test result reports were generated. This command then uploads
your test results.
$ testmo automation:run:submit-thread [options]
OR
- You can alternatively pass the executable/command line to run your tests to
this command. The command then also captures the output of your tests and
measures the exact time. The output and time are then also sent to Testmo.
$ testmo automation:run:submit-thread \
[options] -- <your-test-run-command> [options]
In both scenarios the command expects that your test results are generated
and reported as JUnit XML files. Practically any test automation tool can
report or convert test results to this format. The list of report files can
be specified with the --results option:
testmo automation:run:submit-thread [options] --results reports/*.xml
It is also possible to use recursive search by specifying a quoted single
option to be evaluated by 'glob' (don't forget the quotes):
testmo automation:run:submit-thread [options] --results "reports/**/*.xml"
The command automatically detects if any found files look like a valid JUnit
XML report and ignores other files. By default, detected JUnit XML files that
fail to parse throw an error.
Make sure to set the API authentication token as TESTMO_TOKEN environment
variable. You can either add an authentication token for your user in your
profile settings in Testmo. Or your Testmo admin can add a separate API user
just for authentication.
Examples:
You can create a new thread and submit test results after your test run:
$ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
--instance https://<your-name>.testmo.net \
--run-id 1 \
--results reports/*.xml
You can additionally pass the executable/command line for your test automation
tool as the last parameter after the special -- option. The command then
executes your tool and captures the full output and measures the execution
time. Both are included in the result test automation run in Testmo (note the
space after --):
$ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
--instance https://<your-name>.testmo.net \
--run-id 1 \
--results reports/*.xml \
-- npm test <-- replace this line with your testing tool command line
^ space!
To also include any additional resources previously stored in a file with the
automation:resources:* commands, such as custom fields, links and artifacts,
specify the resource file. You can add resources to the run and/or thread by
using the --run-resources and --thread-resources option. Note that links
are only support for runs and are ignored for threads.
$ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
<other-options> \
--run-resources testmo-run-resources.json \
--thread-resources testmo-thread-resources.jsonAdding fields, links and artifacts
You can use the following commands to manage a testmo-resources.json file to add fields, links and artifacts. You can pass this file to the other commands to submit these resources to runs and threads.
$ testmo help automation:resources:add-artifact
Usage: testmo automation:resources:add-artifact [options]
Adds a new artifact to the local resource file to be submitted with an
automation run
Options:
--name <name> Required: The name (e.g. filename) displayed for this
artifact
--url <link> Required: The URL to access this artifact; please make
sure the URL is only accessible for your team for
security reasons
--note <text> An additional note displayed for the artifact
--file <file> If you specify a local file, the command also collects
the file size and mime type (the file is not uploaded
to Testmo though
--resources <file> The JSON file this artifact is written to
(default: "testmo-resources.json")
--ansi Force ANSI console output (colors, animations)
--no-ansi Do not use ANSI console output (colors, animations)
--debug Output additional debug information
-h, --help Display help for command
Details:
You can include additional resources such as custom fields, links and
artifacts with automation runs. To make it easier to submit these resources,
use the automation:resources:* commands to prepare a local JSON file.
When you then call one of the automation:run:* commands, you can specify
the resources file with the --resources option. All previously written
fields, links and artifacts in this file are then submitted with the run.
By default this command adds a new artifact to a local testmo-resources.json
file if no other filename is specified. If the file does not exist yet,
it is automatically created.
What's an artifact? In this context, an artifact is any file, result or
resource generated by your build, CI or test system you would like to
keep. The files are not stored in Testmo. You need to store the file
separately, e.g. in your CI system or cloud storage. With Testmo's
automation artifact, you can include a link to the file to make it easy
to access from your automation run.
In addition to the name and URL, you can optionally specify the local file
with this command. If you do this, the file size and mime type is
automatically collected as well (the file is not uploaded to Testmo though).
Examples:
You can write a new artifact to the resource file like this:
$ testmo automation:resources:add-artifact \
--name "" \
--url "https://d28c7a82n.s3.amazonaws.com/artifacts/report.xml" \
--note "Test result report"$ testmo help automation:resources:add-field
Usage: testmo automation:resources:add-field [options]
Adds a new field to the local resource file to be submitted with an
automation run
Options:
--type <type> Required: Either string, url, text, console, html
--name <name> Required: The name of this field
--value <string|url> Value for string or URL fields; use stdin for text,
console or HTML
--resources <file> The JSON file this field is written to
(default: "testmo-resources.json")
--ansi Force ANSI console output (colors, animations)
--no-ansi Do not use ANSI console output (colors, animations)
--debug Output additional debug information
-h, --help Display help for command
Details:
You can include additional resources such as custom fields, links and
artifacts with automation runs. To make it easier to submit these resources,
use the automation:resources:* commands to prepare a local JSON file.
When you then call one of the automation:run:* commands, you can specify
the resources file with the --resources option. All previously written
fields, links and artifacts in this file are then submitted with the run.
By default this command adds a new field to a local testmo-resources.json
file if no other filename is specified. If the file does not exist yet,
it is automatically created.
Note: you can either add short string/URL fields or longer fields (displayed
in Testmo as text, console output or rendered HTML). Values for short
string/URL fields need to be specified with the --value option. For text,
console and HTML fields, pass the field value via stdin (see second example
below).
Examples:
You can write a new string/URL fields to the resource file like this:
$ testmo automation:resources:add-field \
--type string \
--name "Version" \
--value "2.3.1-5fbcc8d0"
For text, console and HTML fields, make sure to pipe the field value via
stdin, e.g. from a file:
$ cat files.txt | testmo automation:resources:add-field \
--type console \
--name "File list"$ testmo help automation:resources:add-link
Usage: testmo automation:resources:add-link [options]
Adds a new link to the local resource file to be submitted with an
automation run
Options:
--name <name> Required: The name displayed for this link
--url <link> Required: The URL for this link
--note <text> An additional note displayed for the link
--resources <file> The JSON file this link is written to
(default: "testmo-resources.json")
--ansi Force ANSI console output (colors, animations)
--no-ansi Do not use ANSI console output (colors, animations)
--debug Output additional debug information
-h, --help Display help for command
Details:
You can include additional resources such as custom fields, links and
artifacts with automation runs. To make it easier to submit these resources,
use the automation:resources:* commands to prepare a local JSON file.
When you then call one of the automation:run:* commands, you can specify
the resources file with the --resources option. All previously written
fields, links and artifacts in this file are then submitted with the run.
By default this command adds a new link to a local testmo-resources.json
file if no other filename is specified. If the file does not exist yet,
it is automatically created.
Examples:
You can write a new link to the resource file like this:
$ testmo automation:resources:add-link \
--name "GitHub repository" \
--url "https://github.com/testmo-app/repo" \
--note "Our project's git repository with all code files and issues."Last updated