Reference

The testmo command line tool provides various commands to make it easier to submit and work with your test automation suites and CI/CD pipelines and implement different use cases. Make sure to review the available options and settings to learn about all its capabilities. If you are missing specific options or features, make sure to get in touch to provide feedback.
1
$ testmo help
2
​
3
Usage: testmo [options] [command]
4
​
5
Options:
6
  -V, --version                      Output the version number
7
  --ansi                             Force ANSI console output (colors,
8
                                     animations)
9
  --no-ansi                          Do not use ANSI console output (colors,
10
                                     animations)
11
  --debug                            Output additional debug information
12
  -h, --help                         Display help for command
13
​
14
Commands:
15
  automation:resources:add-artifact  Adds a new artifact to the local resource
16
                                     file to be submitted with an automation run
17
  automation:resources:add-field     Adds a new field to the local resource file
18
                                     to be submitted with an automation run
19
  automation:resources:add-link      Adds a new link to the local resource file
20
                                     to be submitted with an automation run
21
  automation:run:complete            Marks a test automation run as completed
22
  automation:run:create              Creates a new automation run for parallel
23
                                     execution
24
  automation:run:submit              Creates a new test automation run, submits
25
                                     test results and completes the run in a
26
                                     single step
27
  automation:run:submit-thread       Creates a new thread for an existing run,
28
                                     submits test results and completes the
29
                                     thread
30
  help                               Display help for command
Copied!
Submitting entire test runs
To create a new test run, submit all test results and complete the run in a single command, use the automation:run:submit command. This is best used for simple test automation workflows without parallel tests.
1
$ testmo help automation:run:submit
2
​
3
Usage: testmo automation:run:submit [options] -- [executable]
4
​
5
Creates a new test automation run, submits test results and completes the run
6
in a single step
7
​
8
Options:
9
  --instance <url>      Required: The full address of your Testmo instance
10
                        (https://***.testmo.net)
11
  --project-id <id>     Required: The ID of the project this run is added to
12
  --name <name>         Required: The name of the newly created test run
13
  --source <source>     Required: The name of the source for this run and future
14
                        runs of this same source/suite
15
  --results <files...>  Required: Result log files (in JUnit XML format). If you
16
                        pass a single quoted parameter, it is evaluated by
17
                        Node's `glob` to resolve files (allowing recursive
18
                        search, see example below)
19
  --tags <tags...>      Tags for the new run. Tags can also be used to
20
                        automatically link this run to milestones
21
  --config <name>       Automatically find an existing configuration for this
22
                        run based on the config name
23
  --config-id <id>      Or specify a configuration by using the config's ID
24
  --milestone <name>    Automatically find an existing milestone for this run
25
                        based on the milestone name
26
  --milestone-id <id>   Or specify a milestone by using the milestone's ID
27
  --resources <file>    An optional JSON file with custom fields, links and
28
                        artifacts (can be built with the automation:resources
29
                        commands)
30
  --ignore-parse-error  Ignore result file parsing errors and upload remaining
31
                        successfully parsed files (default: false)
32
  --exec-suppress       Suppress standard output (stdout) of passed executable
33
                        (default: false)
34
  --exec-stop-error     Stop and don't create a run if the passed executable
35
                        failed (exit code not 0) (default: false)
36
  --ansi                Force ANSI console output (colors, animations)
37
  --no-ansi             Do not use ANSI console output (colors, animations)
38
  --debug               Output additional debug information
39
  -h, --help            Display help for command
40
​
41
Details:
42
  Use this command to create a new test automation run, submit test results and
43
  complete the run in a single step. That is, use this command if you don't run
44
  your tests in multiple parallel threads. There are two typical ways to use
45
  this command:
46
​
47
  - Run this command after your tests finished and after your test result
48
    reports were generated. This command then uploads your test results.
49
​
50
    $ testmo automation:run:submit [options]
51
​
52
  OR
53
​
54
  - You can alternatively pass the executable/command line to run your tests
55
    to this command. The command then also captures the output of your tests
56
    and measures the exact time. The output and time are then also sent to
57
    Testmo.
58
​
59
    $ testmo automation:run:submit \
60
      [options] -- <your-test-run-command> [options]
61
​
62
  In both scenarios the command expects that your test results are generated and
63
  reported as JUnit XML files. Practically any test automation tool can report
64
  or convert test results to this format. The list of report files can be
65
  specified with the --results option:
66
​
67
  testmo automation:run:submit [options] --results reports/*.xml
68
​
69
  It is also possible to use recursive search by specifying a quoted single
70
  option to be evaluated by 'glob' (don't forget the quotes):
71
​
72
  testmo automation:run:submit [options] --results "reports/**/*.xml"
73
​
74
  The command automatically detects if any found files look like a valid JUnit
75
  XML report and ignores other files. By default, detected JUnit XML files that
76
  fail to parse throw an error.
77
​
78
  Make sure to set the API authentication token as TESTMO_TOKEN environment
79
  variable. You can either add an authentication token for your user in your
80
  profile settings in Testmo. Or your Testmo admin can add a separate API user
81
  just for authentication.
82
​
83
Examples:
84
  You can create a test automation run and submit test results after your
85
  test run:
86
​
87
  $ TESTMO_TOKEN=******** testmo automation:run:submit \
88
      --instance https://<your-name>.testmo.net \
89
      --project-id 1 \
90
      --name "New test run" \
91
      --source "backend-unit" \
92
      --results reports/*.xml
93
​
94
  You can additionally pass the executable/command line for your test automation
95
  tool as the last parameter after the special -- option. The command then
96
  executes your tool and captures the full output and measures the execution
97
  time. Both are included in the result test automation run in Testmo (note the
98
  space after --):
99
​
100
  $ TESTMO_TOKEN=******** testmo automation:run:submit \
101
      --instance https://<your-name>.testmo.net \
102
      --project-id 1 \
103
      --name "New test run" \
104
      --source "backend-unit" \
105
      --results reports/*.xml \
106
      -- npm test <-- replace this line with your testing tool command line
107
        ^ space!
108
​
109
  To also include any additional resources previously stored in a file with the
110
  automation:resources:* commands, such as custom fields, links and artifacts,
111
  specify the resource file with the --resources option:
112
​
113
  $ TESTMO_TOKEN=******** testmo automation:run:submit \
114
      <other-options> \
115
      --resources testmo-resources.json
Copied!
Creating and completing test runs
If you would like to create multiple threads for e.g. parallel tests, you would first create a test run, then submit the threads (see below) and finally complete the run. You would use the automation:run:create and automation:run:complete commands for this.
1
$ testmo help automation:run:create
2
​
3
Usage: testmo automation:run:create [options]
4
​
5
Creates a new automation run for parallel execution
6
​
7
Options:
8
  --instance <url>     Required: The full address of your Testmo instance
9
                       (https://***.testmo.net)
10
  --project-id <id>    Required: The ID of the project this run is added to
11
  --name <name>        Required: The name of the newly created test run
12
  --source <source>    Required: The name of the source for this run and future
13
                       runs of this same source/suite
14
  --tags <tags...>     Tags for the new run. Tags can also be used to
15
                       automatically link this run to milestones
16
  --config <name>      Automatically find an existing configuration for this
17
                       run based on the config name
18
  --config-id <id>     Or specify a configuration by using the config's ID
19
  --milestone <name>   Automatically find an existing milestone for this run
20
                       based on the milestone name
21
  --milestone-id <id>  Or specify a milestone by using the milestone's ID
22
  --resources <file>   An optional JSON file with custom fields, links and
23
                       artifacts (can be built with the automation:resources
24
                       commands)
25
  --ansi               Force ANSI console output (colors, animations)
26
  --no-ansi            Do not use ANSI console output (colors, animations)
27
  --debug              Output additional debug information
28
  -h, --help           Display help for command
29
​
30
Details:
31
  Use this command to create a new automation run you intend to use for parallel
32
  execution, e.g. as part of parallel CI tasks. If successful, this command
33
  outputs the test automation run ID.
34
​
35
  You will need to store this ID and use it for subsequent calls to
36
  automation:run:submit-thread and automation:run:complete. You would usually
37
  use these commands for parallel execution of automated tests:
38
​
39
  automation:run:create            <-- Create the run before your tests start
40
   | automation:run:submit-thread  <-- Submit results in your parallel CI tasks
41
   | automation:run:submit-thread  <-- Submit results in your parallel CI tasks
42
   | automation:run:submit-thread  <-- Submit results in your parallel CI tasks
43
  automation:run:complete          <-- Mark run as completed at the end of
44
                                       your CI
45
​
46
  If you just want to create an automation run and submit results without
47
  parallel execution, just use automation:run:submit without calling
48
  create/complete (not submit-thread!).
49
​
50
  Make sure to set the API authentication token as TESTMO_TOKEN environment
51
  variable. You can either add an authentication token for your user in your
52
  profile settings in Testmo. Or your Testmo admin can add a separate API user
53
  just for authentication.
54
​
55
Examples:
56
  The following example creates a new automation run:
57
​
58
  $ TESTMO_TOKEN=******** testmo automation:run:create \
59
      --instance https://<your-name>.testmo.net \
60
      --project-id 1 \
61
      --name "New test run" \
62
      --source "backend-unit"
63
​
64
  To also include any additional resources previously stored in a file with the
65
  automation:resources:* commands, such as custom fields, links and artifacts,
66
  specify the resource file with the --resources option:
67
​
68
  $ TESTMO_TOKEN=******** testmo automation:run:create \
69
      <other-options> \
70
      --resources testmo-resources.json
Copied!
1
$ testmo help automation:run:complete
2
​
3
Usage: testmo automation:run:complete [options]
4
​
5
Marks a test automation run as completed
6
​
7
Options:
8
  --instance <url>      Required: The full address of your Testmo instance
9
                        (https://***.testmo.net)
10
  --run-id <id>         Required: The ID of the run to be completed (returned
11
                        by automation:run:create)
12
  --no-measure-elapsed  By default Testmo measures the run time by looking at
13
                        the time difference between create and complete. This
14
                        option disables this behavior. The test run time is then
15
                        based on the longest running thread
16
  --resources <file>    An optional JSON file with custom fields, links and
17
                        artifacts (can be built with the automation:resources
18
                        commands)
19
  --ansi                Force ANSI console output (colors, animations)
20
  --no-ansi             Do not use ANSI console output (colors, animations)
21
  --debug               Output additional debug information
22
  -h, --help            Display help for command
23
​
24
Details:
25
  Use this command to mark a test automation run you previously started with
26
  automation:run:create as completed. Once the test run has been completed, it
27
  cannot receive additional threads and test results.
28
​
29
  You should call this command after all your CI test tasks have been completed
30
  for this run and after your last call to automation:run:submit-thread.
31
​
32
  Please note that you do not need to complete a run you have started with
33
  automation:run:submit. It is automatically completed after the results have
34
  been uploaded.
35
​
36
  By default Testmo measures the run time by looking at the time difference
37
  between create and complete. In most situations this is the recommended
38
  approach as it results in accurate automation time measurement. You can
39
  use the --no-measure-elapsed option to disable this behavior. The time of
40
  the test run is then based on the longest running thread.
41
​
42
  Make sure to set the API authentication token as TESTMO_TOKEN environment
43
  variable. You can either add an authentication token for your user in your
44
  profile settings in Testmo. Or your Testmo admin can add a separate API user
45
  just for authentication.
46
​
47
Examples:
48
  The following example completes a running automation run:
49
​
50
  $ TESTMO_TOKEN=******** testmo automation:run:complete \
51
      --instance https://<your-name>.testmo.net \
52
      --run-id 1
53
​
54
  To also include any additional resources previously stored in a file with the
55
  automation:resources:* commands, such as custom fields, links and artifacts,
56
  specify the resource file with the --resources option:
57
​
58
  $ TESTMO_TOKEN=******** testmo automation:run:complete \
59
      <other-options> \
60
      --resources testmo-resources.json
Copied!
Submitting threads for runs
If you create test runs with the above automation:run:create command, you can submit individual threads and test results with the automation:run:submit-thread command. You would usually use this approach for more advanced CI/CD pipeline workflows with parallel test jobs.
1
$ testmo help automation:run:submit-thread
2
​
3
Usage: testmo automation:run:submit-thread [options] -- [executable]
4
​
5
Creates a new thread for an existing run, submits test results and completes
6
the thread
7
​
8
Options:
9
  --instance <url>           Required: The full address of your Testmo instance
10
                             (https://***.testmo.net)
11
  --run-id <id>              Required: The ID of the run this new thread should
12
                             be added to (returned by automation:run:create
13
  --results <files...>       Required: Result log files (in JUnit XML format).
14
                             If you pass a single quoted parameter, it is
15
                             evaluated by Node's `glob` to resolve files
16
                             (allowing recursive search, see example below)
17
  --run-resources <file>     An optional JSON file with custom fields, links and
18
                             artifacts (can be built with the
19
                             automation:resources commands) for the existing run
20
  --thread-resources <file>  An optional JSON file with custom fields and
21
                             artifacts (can be built with the
22
                             automation:resources commands) for the new thread
23
  --ignore-parse-error       Ignore result file parsing errors and upload
24
                             remaining successfully parsed files
25
                             (default: false)
26
  --exec-suppress            Suppress standard output (stdout) of passed
27
                             executable (default: false)
28
  --exec-stop-error          Stop and don't create a run if the passed
29
                             executable failed (exit code not 0)
30
                             (default: false)
31
  --ansi                     Force ANSI console output (colors, animations)
32
  --no-ansi                  Do not use ANSI console output (colors, animations)
33
  --debug                    Output additional debug information
34
  -h, --help                 Display help for command
35
​
36
Details:
37
  Use this command to add a new thread to an existing running test automation
38
  run, submit test results and complete the thread in a single step. You would
39
  use this command to submit the results of a single (parallel) CI task to an
40
  active run.
41
​
42
  If you just want to create a test run with a single thread (i.e. a
43
  non-parallel single CI task), you would just use the automation:run:submit
44
  task instead.
45
​
46
  There are two typical ways to use this command:
47
​
48
  - Run this command in your parallel CI task after your tests finished and
49
    after your test result reports were generated. This command then uploads
50
    your test results.
51
​
52
    $ testmo automation:run:submit-thread [options]
53
​
54
  OR
55
​
56
  - You can alternatively pass the executable/command line to run your tests to
57
    this command. The command then also captures the output of your tests and
58
    measures the exact time. The output and time are then also sent to Testmo.
59
​
60
    $ testmo automation:run:submit-thread \
61
        [options] -- <your-test-run-command> [options]
62
​
63
  In both scenarios the command expects that your test results are generated
64
  and reported as JUnit XML files. Practically any test automation tool can
65
  report or convert test results to this format. The list of report files can
66
  be specified with the --results option:
67
​
68
  testmo automation:run:submit-thread [options] --results reports/*.xml
69
​
70
  It is also possible to use recursive search by specifying a quoted single
71
  option to be evaluated by 'glob' (don't forget the quotes):
72
​
73
  testmo automation:run:submit-thread [options] --results "reports/**/*.xml"
74
​
75
  The command automatically detects if any found files look like a valid JUnit
76
  XML report and ignores other files. By default, detected JUnit XML files that
77
  fail to parse throw an error.
78
​
79
  Make sure to set the API authentication token as TESTMO_TOKEN environment
80
  variable. You can either add an authentication token for your user in your
81
  profile settings in Testmo. Or your Testmo admin can add a separate API user
82
  just for authentication.
83
​
84
Examples:
85
  You can create a new thread and submit test results after your test run:
86
​
87
  $ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
88
      --instance https://<your-name>.testmo.net \
89
      --run-id 1 \
90
      --results reports/*.xml
91
​
92
  You can additionally pass the executable/command line for your test automation
93
  tool as the last parameter after the special -- option. The command then
94
  executes your tool and captures the full output and measures the execution
95
  time. Both are included in the result test automation run in Testmo (note the
96
  space after --):
97
​
98
  $ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
99
      --instance https://<your-name>.testmo.net \
100
      --run-id 1 \
101
      --results reports/*.xml \
102
      -- npm test <-- replace this line with your testing tool command line
103
        ^ space!
104
​
105
  To also include any additional resources previously stored in a file with the
106
  automation:resources:* commands, such as custom fields, links and artifacts,
107
  specify the resource file. You can add resources to the run and/or thread by
108
  using the --run-resources and --thread-resources option. Note that links
109
  are only support for runs and are ignored for threads.
110
​
111
  $ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
112
      <other-options> \
113
      --run-resources testmo-run-resources.json \
114
      --thread-resources testmo-thread-resources.json
Copied!
Adding fields, links and artifacts
You can use the following commands to manage a testmo-resources.json file to add fields, links and artifacts. You can pass this file to the other commands to submit these resources to runs and threads.
1
$ testmo help automation:resources:add-artifact
2
​
3
Usage: testmo automation:resources:add-artifact [options]
4
​
5
Adds a new artifact to the local resource file to be submitted
6
with an automation run
7
​
8
Options:
9
  --name <name>       Required: The name (e.g. filename) displayed for this
10
                      artifact
11
  --url <link>        Required: The URL to access this artifact; please make
12
                      sure the URL is only accessible for your team for
13
                      security reasons
14
  --note <text>       An additional note displayed for the artifact
15
  --file <file>       If you specify a local file, the command also collects
16
                      the file size and mime type (the file is not uploaded
17
                      to Testmo though
18
  --resources <file>  The JSON file this link is written to
19
                      (default: "testmo-resources.json")
20
  --ansi              Force ANSI console output (colors, animations)
21
  --no-ansi           Do not use ANSI console output (colors, animations)
22
  --debug             Output additional debug information
23
  -h, --help          Display help for command
24
​
25
Details:
26
  You can include additional resources such as custom fields, links and
27
  artifacts with automation runs. To make it easier to submit these resources,
28
  use the automation:resources:* commands to prepare a local JSON file.
29
​
30
  When you then call one of the automation:run:* commands, you can specify
31
  the resources file with the --resources option. All previously written
32
  fields, links and artifacts in this file are then submitted with the run.
33
​
34
  By default this command adds a new artifact to a local testmo-resources.json
35
  file if no other filename is specified. If the file does not exist yet,
36
  it is automatically created.
37
​
38
  What's an artifact? In this context, an artifact is any file, result or
39
  resource generated by your build, CI or test system you would like to
40
  keep. The files are not stored in Testmo. You need to store the file
41
  separately, e.g. in your CI system or cloud storage. With Testmo's
42
  automation artifact, you can include a link to the file to make it easy
43
  to access from your automation run.
44
​
45
  In addition to the name and URL, you can optionally specify the local file
46
  with this command. If you do this, the file size and mime type is
47
  automatically collected as well (the file is not uploaded to Testmo though).
48
​
49
Examples:
50
  You can write a new artifact to the resource file like this:
51
​
52
  $ testmo automation:resources:add-artifact \
53
      --name "" \
54
      --url "https://d28c7a82n.s3.amazonaws.com/artifacts/report.xml" \
55
      --note "Test result report"
Copied!
1
$ testmo help automation:resources:add-field
2
​
3
Usage: testmo automation:resources:add-field [options]
4
​
5
Adds a new field to the local resource file to be submitted with an
6
automation run
7
​
8
Options:
9
  --type <type>         Required: Either string, url, text, console, html
10
  --name <name>         Required: The name of this field
11
  --value <string|url>  Value for string or URL fields; use stdin for text,
12
                        console or HTML
13
  --resources <file>    The JSON file this field is written to
14
                        (default:  "testmo-resources.json")
15
  --ansi                Force ANSI console output (colors, animations)
16
  --no-ansi             Do not use ANSI console output (colors, animations)
17
  --debug               Output additional debug information
18
  -h, --help            Display help for command
19
​
20
Details:
21
  You can include additional resources such as custom fields, links and
22
  artifacts with automation runs. To make it easier to submit these resources,
23
  use the automation:resources:* commands to prepare a local JSON file.
24
​
25
  When you then call one of the automation:run:* commands, you can specify
26
  the resources file with the --resources option. All previously written
27
  fields, links and artifacts in this file are then submitted with the run.
28
​
29
  By default this command adds a new field to a local testmo-resources.json
30
  file if no other filename is specified. If the file does not exist yet,
31
  it is automatically created.
32
​
33
  Note: you can either add short string/URL fields or longer fields (displayed
34
  in Testmo as text, console output or rendered HTML). Values for short
35
  string/URL fields need to be specified with the --value option. For text,
36
  console and HTML fields, pass the field value via stdin (see second example
37
  below).
38
​
39
Examples:
40
  You can write a new string/URL fields to the resource file like this:
41
​
42
  $ testmo automation:resources:add-field \
43
      --type string \
44
      --name "Version" \
45
      --value "2.3.1-5fbcc8d0"
46
​
47
  For text, console and HTML fields, make sure to pipe the field value via
48
  stdin, e.g. from a file:
49
​
50
  $ cat files.txt | testmo automation:resources:add-field \
51
      --type console \
52
      --name "File list"
Copied!
1
$ testmo help automation:resources:add-link
2
​
3
Usage: testmo automation:resources:add-link [options]
4
​
5
Adds a new link to the local resource file to be submitted with an
6
automation run
7
​
8
Options:
9
  --name <name>       Required: The name displayed for this link
10
  --url <link>        Required: The URL for this link
11
  --note <text>       An additional note displayed for the link
12
  --resources <file>  The JSON file this link is written to
13
                      (default: "testmo-resources.json")
14
  --ansi              Force ANSI console output (colors, animations)
15
  --no-ansi           Do not use ANSI console output (colors, animations)
16
  --debug             Output additional debug information
17
  -h, --help          Display help for command
18
​
19
Details:
20
  You can include additional resources such as custom fields, links and
21
  artifacts with automation runs. To make it easier to submit these resources,
22
  use the automation:resources:* commands to prepare a local JSON file.
23
​
24
  When you then call one of the automation:run:* commands, you can specify
25
  the resources file with the --resources option. All previously written
26
  fields, links and artifacts in this file are then submitted with the run.
27
​
28
  By default this command adds a new link to a local testmo-resources.json
29
  file if no other filename is specified. If the file does not exist yet,
30
  it is automatically created.
31
​
32
Examples:
33
  You can write a new link to the resource file like this:
34
​
35
  $ testmo automation:resources:add-link \
36
      --name "GitHub repository" \
37
      --url "https://github.com/testmo-app/repo" \
38
      --note "Our project's git repository with all code files and issues."
Copied!
Automation - Previous
Testmo CLI
Next - Administration
Overview
Last modified 6mo ago
Copy link
Contents
Submitting entire test runs
Creating and completing test runs
Submitting threads for runs
Adding fields, links and artifacts