Reference
The
testmo command line tool provides various commands to make it easier to submit and work with your test automation suites and CI/CD pipelines and implement different use cases. Make sure to review the available options and settings to learn about all its capabilities. If you are missing specific options or features, make sure to get in touch to provide feedback.1
$ testmo help
2
​
3
Usage: testmo [options] [command]
4
​
5
Options:
6
-V, --version Output the version number
7
--ansi Force ANSI console output (colors,
8
animations)
9
--no-ansi Do not use ANSI console output (colors,
10
animations)
11
--debug Output additional debug information
12
-h, --help Display help for command
13
​
14
Commands:
15
automation:resources:add-artifact Adds a new artifact to the local resource
16
file to be submitted with an automation run
17
automation:resources:add-field Adds a new field to the local resource file
18
to be submitted with an automation run
19
automation:resources:add-link Adds a new link to the local resource file
20
to be submitted with an automation run
21
automation:run:complete Marks a test automation run as completed
22
automation:run:create Creates a new automation run for parallel
23
execution
24
automation:run:submit Creates a new test automation run, submits
25
test results and completes the run in a
26
single step
27
automation:run:submit-thread Creates a new thread for an existing run,
28
submits test results and completes the
29
thread
30
help Display help for command
Copied!
Submitting entire test runs
To create a new test run, submit all test results and complete the run in a single command, use the
automation:run:submit command. This is best used for simple test automation workflows without parallel tests.1
$ testmo help automation:run:submit
2
​
3
Usage: testmo automation:run:submit [options] -- [executable]
4
​
5
Creates a new test automation run, submits test results and completes the run
6
in a single step
7
​
8
Options:
9
--instance <url> Required: The full address of your Testmo instance
10
(https://***.testmo.net)
11
--project-id <id> Required: The ID of the project this run is added to
12
--name <name> Required: The name of the newly created test run
13
--source <source> Required: The name of the source for this run and future
14
runs of this same source/suite
15
--results <files...> Required: Result log files (in JUnit XML format). If you
16
pass a single quoted parameter, it is evaluated by
17
Node's `glob` to resolve files (allowing recursive
18
search, see example below)
19
--tags <tags...> Tags for the new run. Tags can also be used to
20
automatically link this run to milestones
21
--config <name> Automatically find an existing configuration for this
22
run based on the config name
23
--config-id <id> Or specify a configuration by using the config's ID
24
--milestone <name> Automatically find an existing milestone for this run
25
based on the milestone name
26
--milestone-id <id> Or specify a milestone by using the milestone's ID
27
--resources <file> An optional JSON file with custom fields, links and
28
artifacts (can be built with the automation:resources
29
commands)
30
--ignore-parse-error Ignore result file parsing errors and upload remaining
31
successfully parsed files (default: false)
32
--exec-suppress Suppress standard output (stdout) of passed executable
33
(default: false)
34
--exec-stop-error Stop and don't create a run if the passed executable
35
failed (exit code not 0) (default: false)
36
--ansi Force ANSI console output (colors, animations)
37
--no-ansi Do not use ANSI console output (colors, animations)
38
--debug Output additional debug information
39
-h, --help Display help for command
40
​
41
Details:
42
Use this command to create a new test automation run, submit test results and
43
complete the run in a single step. That is, use this command if you don't run
44
your tests in multiple parallel threads. There are two typical ways to use
45
this command:
46
​
47
- Run this command after your tests finished and after your test result
48
reports were generated. This command then uploads your test results.
49
​
50
$ testmo automation:run:submit [options]
51
​
52
OR
53
​
54
- You can alternatively pass the executable/command line to run your tests
55
to this command. The command then also captures the output of your tests
56
and measures the exact time. The output and time are then also sent to
57
Testmo.
58
​
59
$ testmo automation:run:submit \
60
[options] -- <your-test-run-command> [options]
61
​
62
In both scenarios the command expects that your test results are generated and
63
reported as JUnit XML files. Practically any test automation tool can report
64
or convert test results to this format. The list of report files can be
65
specified with the --results option:
66
​
67
testmo automation:run:submit [options] --results reports/*.xml
68
​
69
It is also possible to use recursive search by specifying a quoted single
70
option to be evaluated by 'glob' (don't forget the quotes):
71
​
72
testmo automation:run:submit [options] --results "reports/**/*.xml"
73
​
74
The command automatically detects if any found files look like a valid JUnit
75
XML report and ignores other files. By default, detected JUnit XML files that
76
fail to parse throw an error.
77
​
78
Make sure to set the API authentication token as TESTMO_TOKEN environment
79
variable. You can either add an authentication token for your user in your
80
profile settings in Testmo. Or your Testmo admin can add a separate API user
81
just for authentication.
82
​
83
Examples:
84
You can create a test automation run and submit test results after your
85
test run:
86
​
87
$ TESTMO_TOKEN=******** testmo automation:run:submit \
88
--instance https://<your-name>.testmo.net \
89
--project-id 1 \
90
--name "New test run" \
91
--source "backend-unit" \
92
--results reports/*.xml
93
​
94
You can additionally pass the executable/command line for your test automation
95
tool as the last parameter after the special -- option. The command then
96
executes your tool and captures the full output and measures the execution
97
time. Both are included in the result test automation run in Testmo (note the
98
space after --):
99
​
100
$ TESTMO_TOKEN=******** testmo automation:run:submit \
101
--instance https://<your-name>.testmo.net \
102
--project-id 1 \
103
--name "New test run" \
104
--source "backend-unit" \
105
--results reports/*.xml \
106
-- npm test <-- replace this line with your testing tool command line
107
^ space!
108
​
109
To also include any additional resources previously stored in a file with the
110
automation:resources:* commands, such as custom fields, links and artifacts,
111
specify the resource file with the --resources option:
112
​
113
$ TESTMO_TOKEN=******** testmo automation:run:submit \
114
<other-options> \
115
--resources testmo-resources.json
Copied!
Creating and completing test runs
If you would like to create multiple threads for e.g. parallel tests, you would first create a test run, then submit the threads (see below) and finally complete the run. You would use the
automation:run:create and automation:run:complete commands for this.1
$ testmo help automation:run:create
2
​
3
Usage: testmo automation:run:create [options]
4
​
5
Creates a new automation run for parallel execution
6
​
7
Options:
8
--instance <url> Required: The full address of your Testmo instance
9
(https://***.testmo.net)
10
--project-id <id> Required: The ID of the project this run is added to
11
--name <name> Required: The name of the newly created test run
12
--source <source> Required: The name of the source for this run and future
13
runs of this same source/suite
14
--tags <tags...> Tags for the new run. Tags can also be used to
15
automatically link this run to milestones
16
--config <name> Automatically find an existing configuration for this
17
run based on the config name
18
--config-id <id> Or specify a configuration by using the config's ID
19
--milestone <name> Automatically find an existing milestone for this run
20
based on the milestone name
21
--milestone-id <id> Or specify a milestone by using the milestone's ID
22
--resources <file> An optional JSON file with custom fields, links and
23
artifacts (can be built with the automation:resources
24
commands)
25
--ansi Force ANSI console output (colors, animations)
26
--no-ansi Do not use ANSI console output (colors, animations)
27
--debug Output additional debug information
28
-h, --help Display help for command
29
​
30
Details:
31
Use this command to create a new automation run you intend to use for parallel
32
execution, e.g. as part of parallel CI tasks. If successful, this command
33
outputs the test automation run ID.
34
​
35
You will need to store this ID and use it for subsequent calls to
36
automation:run:submit-thread and automation:run:complete. You would usually
37
use these commands for parallel execution of automated tests:
38
​
39
automation:run:create <-- Create the run before your tests start
40
| automation:run:submit-thread <-- Submit results in your parallel CI tasks
41
| automation:run:submit-thread <-- Submit results in your parallel CI tasks
42
| automation:run:submit-thread <-- Submit results in your parallel CI tasks
43
automation:run:complete <-- Mark run as completed at the end of
44
your CI
45
​
46
If you just want to create an automation run and submit results without
47
parallel execution, just use automation:run:submit without calling
48
create/complete (not submit-thread!).
49
​
50
Make sure to set the API authentication token as TESTMO_TOKEN environment
51
variable. You can either add an authentication token for your user in your
52
profile settings in Testmo. Or your Testmo admin can add a separate API user
53
just for authentication.
54
​
55
Examples:
56
The following example creates a new automation run:
57
​
58
$ TESTMO_TOKEN=******** testmo automation:run:create \
59
--instance https://<your-name>.testmo.net \
60
--project-id 1 \
61
--name "New test run" \
62
--source "backend-unit"
63
​
64
To also include any additional resources previously stored in a file with the
65
automation:resources:* commands, such as custom fields, links and artifacts,
66
specify the resource file with the --resources option:
67
​
68
$ TESTMO_TOKEN=******** testmo automation:run:create \
69
<other-options> \
70
--resources testmo-resources.json
Copied!
1
$ testmo help automation:run:complete
2
​
3
Usage: testmo automation:run:complete [options]
4
​
5
Marks a test automation run as completed
6
​
7
Options:
8
--instance <url> Required: The full address of your Testmo instance
9
(https://***.testmo.net)
10
--run-id <id> Required: The ID of the run to be completed (returned
11
by automation:run:create)
12
--no-measure-elapsed By default Testmo measures the run time by looking at
13
the time difference between create and complete. This
14
option disables this behavior. The test run time is then
15
based on the longest running thread
16
--resources <file> An optional JSON file with custom fields, links and
17
artifacts (can be built with the automation:resources
18
commands)
19
--ansi Force ANSI console output (colors, animations)
20
--no-ansi Do not use ANSI console output (colors, animations)
21
--debug Output additional debug information
22
-h, --help Display help for command
23
​
24
Details:
25
Use this command to mark a test automation run you previously started with
26
automation:run:create as completed. Once the test run has been completed, it
27
cannot receive additional threads and test results.
28
​
29
You should call this command after all your CI test tasks have been completed
30
for this run and after your last call to automation:run:submit-thread.
31
​
32
Please note that you do not need to complete a run you have started with
33
automation:run:submit. It is automatically completed after the results have
34
been uploaded.
35
​
36
By default Testmo measures the run time by looking at the time difference
37
between create and complete. In most situations this is the recommended
38
approach as it results in accurate automation time measurement. You can
39
use the --no-measure-elapsed option to disable this behavior. The time of
40
the test run is then based on the longest running thread.
41
​
42
Make sure to set the API authentication token as TESTMO_TOKEN environment
43
variable. You can either add an authentication token for your user in your
44
profile settings in Testmo. Or your Testmo admin can add a separate API user
45
just for authentication.
46
​
47
Examples:
48
The following example completes a running automation run:
49
​
50
$ TESTMO_TOKEN=******** testmo automation:run:complete \
51
--instance https://<your-name>.testmo.net \
52
--run-id 1
53
​
54
To also include any additional resources previously stored in a file with the
55
automation:resources:* commands, such as custom fields, links and artifacts,
56
specify the resource file with the --resources option:
57
​
58
$ TESTMO_TOKEN=******** testmo automation:run:complete \
59
<other-options> \
60
--resources testmo-resources.json
Copied!
Submitting threads for runs
If you create test runs with the above
automation:run:create command, you can submit individual threads and test results with the automation:run:submit-thread command. You would usually use this approach for more advanced CI/CD pipeline workflows with parallel test jobs.1
$ testmo help automation:run:submit-thread
2
​
3
Usage: testmo automation:run:submit-thread [options] -- [executable]
4
​
5
Creates a new thread for an existing run, submits test results and completes
6
the thread
7
​
8
Options:
9
--instance <url> Required: The full address of your Testmo instance
10
(https://***.testmo.net)
11
--run-id <id> Required: The ID of the run this new thread should
12
be added to (returned by automation:run:create
13
--results <files...> Required: Result log files (in JUnit XML format).
14
If you pass a single quoted parameter, it is
15
evaluated by Node's `glob` to resolve files
16
(allowing recursive search, see example below)
17
--run-resources <file> An optional JSON file with custom fields, links and
18
artifacts (can be built with the
19
automation:resources commands) for the existing run
20
--thread-resources <file> An optional JSON file with custom fields and
21
artifacts (can be built with the
22
automation:resources commands) for the new thread
23
--ignore-parse-error Ignore result file parsing errors and upload
24
remaining successfully parsed files
25
(default: false)
26
--exec-suppress Suppress standard output (stdout) of passed
27
executable (default: false)
28
--exec-stop-error Stop and don't create a run if the passed
29
executable failed (exit code not 0)
30
(default: false)
31
--ansi Force ANSI console output (colors, animations)
32
--no-ansi Do not use ANSI console output (colors, animations)
33
--debug Output additional debug information
34
-h, --help Display help for command
35
​
36
Details:
37
Use this command to add a new thread to an existing running test automation
38
run, submit test results and complete the thread in a single step. You would
39
use this command to submit the results of a single (parallel) CI task to an
40
active run.
41
​
42
If you just want to create a test run with a single thread (i.e. a
43
non-parallel single CI task), you would just use the automation:run:submit
44
task instead.
45
​
46
There are two typical ways to use this command:
47
​
48
- Run this command in your parallel CI task after your tests finished and
49
after your test result reports were generated. This command then uploads
50
your test results.
51
​
52
$ testmo automation:run:submit-thread [options]
53
​
54
OR
55
​
56
- You can alternatively pass the executable/command line to run your tests to
57
this command. The command then also captures the output of your tests and
58
measures the exact time. The output and time are then also sent to Testmo.
59
​
60
$ testmo automation:run:submit-thread \
61
[options] -- <your-test-run-command> [options]
62
​
63
In both scenarios the command expects that your test results are generated
64
and reported as JUnit XML files. Practically any test automation tool can
65
report or convert test results to this format. The list of report files can
66
be specified with the --results option:
67
​
68
testmo automation:run:submit-thread [options] --results reports/*.xml
69
​
70
It is also possible to use recursive search by specifying a quoted single
71
option to be evaluated by 'glob' (don't forget the quotes):
72
​
73
testmo automation:run:submit-thread [options] --results "reports/**/*.xml"
74
​
75
The command automatically detects if any found files look like a valid JUnit
76
XML report and ignores other files. By default, detected JUnit XML files that
77
fail to parse throw an error.
78
​
79
Make sure to set the API authentication token as TESTMO_TOKEN environment
80
variable. You can either add an authentication token for your user in your
81
profile settings in Testmo. Or your Testmo admin can add a separate API user
82
just for authentication.
83
​
84
Examples:
85
You can create a new thread and submit test results after your test run:
86
​
87
$ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
88
--instance https://<your-name>.testmo.net \
89
--run-id 1 \
90
--results reports/*.xml
91
​
92
You can additionally pass the executable/command line for your test automation
93
tool as the last parameter after the special -- option. The command then
94
executes your tool and captures the full output and measures the execution
95
time. Both are included in the result test automation run in Testmo (note the
96
space after --):
97
​
98
$ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
99
--instance https://<your-name>.testmo.net \
100
--run-id 1 \
101
--results reports/*.xml \
102
-- npm test <-- replace this line with your testing tool command line
103
^ space!
104
​
105
To also include any additional resources previously stored in a file with the
106
automation:resources:* commands, such as custom fields, links and artifacts,
107
specify the resource file. You can add resources to the run and/or thread by
108
using the --run-resources and --thread-resources option. Note that links
109
are only support for runs and are ignored for threads.
110
​
111
$ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
112
<other-options> \
113
--run-resources testmo-run-resources.json \
114
--thread-resources testmo-thread-resources.json
Copied!
Adding fields, links and artifacts
You can use the following commands to manage a
testmo-resources.json file to add fields, links and artifacts. You can pass this file to the other commands to submit these resources to runs and threads.1
$ testmo help automation:resources:add-artifact
2
​
3
Usage: testmo automation:resources:add-artifact [options]
4
​
5
Adds a new artifact to the local resource file to be submitted
6
with an automation run
7
​
8
Options:
9
--name <name> Required: The name (e.g. filename) displayed for this
10
artifact
11
--url <link> Required: The URL to access this artifact; please make
12
sure the URL is only accessible for your team for
13
security reasons
14
--note <text> An additional note displayed for the artifact
15
--file <file> If you specify a local file, the command also collects
16
the file size and mime type (the file is not uploaded
17
to Testmo though
18
--resources <file> The JSON file this link is written to
19
(default: "testmo-resources.json")
20
--ansi Force ANSI console output (colors, animations)
21
--no-ansi Do not use ANSI console output (colors, animations)
22
--debug Output additional debug information
23
-h, --help Display help for command
24
​
25
Details:
26
You can include additional resources such as custom fields, links and
27
artifacts with automation runs. To make it easier to submit these resources,
28
use the automation:resources:* commands to prepare a local JSON file.
29
​
30
When you then call one of the automation:run:* commands, you can specify
31
the resources file with the --resources option. All previously written
32
fields, links and artifacts in this file are then submitted with the run.
33
​
34
By default this command adds a new artifact to a local testmo-resources.json
35
file if no other filename is specified. If the file does not exist yet,
36
it is automatically created.
37
​
38
What's an artifact? In this context, an artifact is any file, result or
39
resource generated by your build, CI or test system you would like to
40
keep. The files are not stored in Testmo. You need to store the file
41
separately, e.g. in your CI system or cloud storage. With Testmo's
42
automation artifact, you can include a link to the file to make it easy
43
to access from your automation run.
44
​
45
In addition to the name and URL, you can optionally specify the local file
46
with this command. If you do this, the file size and mime type is
47
automatically collected as well (the file is not uploaded to Testmo though).
48
​
49
Examples:
50
You can write a new artifact to the resource file like this:
51
​
52
$ testmo automation:resources:add-artifact \
53
--name "" \
54
--url "https://d28c7a82n.s3.amazonaws.com/artifacts/report.xml" \
55
--note "Test result report"
Copied!
1
$ testmo help automation:resources:add-field
2
​
3
Usage: testmo automation:resources:add-field [options]
4
​
5
Adds a new field to the local resource file to be submitted with an
6
automation run
7
​
8
Options:
9
--type <type> Required: Either string, url, text, console, html
10
--name <name> Required: The name of this field
11
--value <string|url> Value for string or URL fields; use stdin for text,
12
console or HTML
13
--resources <file> The JSON file this field is written to
14
(default: "testmo-resources.json")
15
--ansi Force ANSI console output (colors, animations)
16
--no-ansi Do not use ANSI console output (colors, animations)
17
--debug Output additional debug information
18
-h, --help Display help for command
19
​
20
Details:
21
You can include additional resources such as custom fields, links and
22
artifacts with automation runs. To make it easier to submit these resources,
23
use the automation:resources:* commands to prepare a local JSON file.
24
​
25
When you then call one of the automation:run:* commands, you can specify
26
the resources file with the --resources option. All previously written
27
fields, links and artifacts in this file are then submitted with the run.
28
​
29
By default this command adds a new field to a local testmo-resources.json
30
file if no other filename is specified. If the file does not exist yet,
31
it is automatically created.
32
​
33
Note: you can either add short string/URL fields or longer fields (displayed
34
in Testmo as text, console output or rendered HTML). Values for short
35
string/URL fields need to be specified with the --value option. For text,
36
console and HTML fields, pass the field value via stdin (see second example
37
below).
38
​
39
Examples:
40
You can write a new string/URL fields to the resource file like this:
41
​
42
$ testmo automation:resources:add-field \
43
--type string \
44
--name "Version" \
45
--value "2.3.1-5fbcc8d0"
46
​
47
For text, console and HTML fields, make sure to pipe the field value via
48
stdin, e.g. from a file:
49
​
50
$ cat files.txt | testmo automation:resources:add-field \
51
--type console \
52
--name "File list"
Copied!
1
$ testmo help automation:resources:add-link
2
​
3
Usage: testmo automation:resources:add-link [options]
4
​
5
Adds a new link to the local resource file to be submitted with an
6
automation run
7
​
8
Options:
9
--name <name> Required: The name displayed for this link
10
--url <link> Required: The URL for this link
11
--note <text> An additional note displayed for the link
12
--resources <file> The JSON file this link is written to
13
(default: "testmo-resources.json")
14
--ansi Force ANSI console output (colors, animations)
15
--no-ansi Do not use ANSI console output (colors, animations)
16
--debug Output additional debug information
17
-h, --help Display help for command
18
​
19
Details:
20
You can include additional resources such as custom fields, links and
21
artifacts with automation runs. To make it easier to submit these resources,
22
use the automation:resources:* commands to prepare a local JSON file.
23
​
24
When you then call one of the automation:run:* commands, you can specify
25
the resources file with the --resources option. All previously written
26
fields, links and artifacts in this file are then submitted with the run.
27
​
28
By default this command adds a new link to a local testmo-resources.json
29
file if no other filename is specified. If the file does not exist yet,
30
it is automatically created.
31
​
32
Examples:
33
You can write a new link to the resource file like this:
34
​
35
$ testmo automation:resources:add-link \
36
--name "GitHub repository" \
37
--url "https://github.com/testmo-app/repo" \
38
--note "Our project's git repository with all code files and issues."
Copied!
Last modified 1mo ago