Reference

The testmo command line tool provides various commands to make it easier to submit and work with your test automation suites and CI/CD pipelines and implement different use cases. Make sure to review the available options and settings to learn about all its capabilities. If you are missing specific options or features, make sure to get in touch to provide feedback.
$ testmo help
​
Usage: testmo [options] [command]
​
Options:
  -V, --version                      Output the version number
  --ansi                             Force ANSI console output (colors,
                                     animations)
  --no-ansi                          Do not use ANSI console output (colors,
                                     animations)
  --debug                            Output additional debug information
  -h, --help                         Display help for command
​
Commands:
  automation:resources:add-artifact  Adds a new artifact to the local resource
                                     file to be submitted with an automation run
  automation:resources:add-field     Adds a new field to the local resource file
                                     to be submitted with an automation run
  automation:resources:add-link      Adds a new link to the local resource file
                                     to be submitted with an automation run
  automation:run:complete            Marks a test automation run as completed
  automation:run:create              Creates a new automation run for parallel
                                     execution
  automation:run:submit              Creates a new test automation run, submits
                                     test results and completes the run in a
                                     single step
  automation:run:submit-thread       Creates a new thread for an existing run,
                                     submits test results and completes the
                                     thread
  help                               Display help for command
Submitting entire test runs
To create a new test run, submit all test results and complete the run in a single command, use the automation:run:submit command. This is best used for simple test automation workflows without parallel tests.
$ testmo help automation:run:submit
​
Usage: testmo automation:run:submit [options] -- [executable]
​
Creates a new test automation run, submits test results and completes the run
in a single step
​
Options:
  --instance <url>      Required: The full address of your Testmo instance
                        (https://***.testmo.net)
  --project-id <id>     Required: The ID of the project this run is added to
  --name <name>         Required: The name of the newly created test run
  --source <source>     Required: The name of the source for this run and future
                        runs of this same source/suite
  --results <files...>  Required: Result log files (in JUnit XML format). If you
                        pass a single quoted parameter, it is evaluated by
                        Node's `glob` to resolve files (allowing recursive
                        search, see example below)
  --tags <tags...>      Tags for the new run. Tags can also be used to
                        automatically link this run to milestones
  --config <name>       Automatically find an existing configuration for this
                        run based on the config name
  --config-id <id>      Or specify a configuration by using the config's ID
  --milestone <name>    Automatically find an existing milestone for this run
                        based on the milestone name
  --milestone-id <id>   Or specify a milestone by using the milestone's ID
  --resources <file>    An optional JSON file with custom fields, links and
                        artifacts (can be built with the automation:resources
                        commands)
  --ignore-parse-error  Ignore result file parsing errors and upload remaining
                        successfully parsed files (default: false)
  --exec-suppress       Suppress standard output (stdout) of passed executable
                        (default: false)
  --exec-stop-error     Stop and don't create a run if the passed executable
                        failed (exit code not 0) (default: false)
  --ansi                Force ANSI console output (colors, animations)
  --no-ansi             Do not use ANSI console output (colors, animations)
  --debug               Output additional debug information
  -h, --help            Display help for command
​
Details:
  Use this command to create a new test automation run, submit test results and
  complete the run in a single step. That is, use this command if you don't run
  your tests in multiple parallel threads. There are two typical ways to use
  this command:
​
  - Run this command after your tests finished and after your test result
    reports were generated. This command then uploads your test results.
​
    $ testmo automation:run:submit [options]
​
  OR
​
  - You can alternatively pass the executable/command line to run your tests
    to this command. The command then also captures the output of your tests
    and measures the exact time. The output and time are then also sent to
    Testmo.
​
    $ testmo automation:run:submit \
      [options] -- <your-test-run-command> [options]
​
  In both scenarios the command expects that your test results are generated and
  reported as JUnit XML files. Practically any test automation tool can report
  or convert test results to this format. The list of report files can be
  specified with the --results option:
​
  testmo automation:run:submit [options] --results reports/*.xml
​
  It is also possible to use recursive search by specifying a quoted single
  option to be evaluated by 'glob' (don't forget the quotes):
​
  testmo automation:run:submit [options] --results "reports/**/*.xml"
​
  The command automatically detects if any found files look like a valid JUnit
  XML report and ignores other files. By default, detected JUnit XML files that
  fail to parse throw an error.
​
  Make sure to set the API authentication token as TESTMO_TOKEN environment
  variable. You can either add an authentication token for your user in your
  profile settings in Testmo. Or your Testmo admin can add a separate API user
  just for authentication.
​
Examples:
  You can create a test automation run and submit test results after your
  test run:
​
  $ TESTMO_TOKEN=******** testmo automation:run:submit \
      --instance https://<your-name>.testmo.net \
      --project-id 1 \
      --name "New test run" \
      --source "backend-unit" \
      --results reports/*.xml
​
  You can additionally pass the executable/command line for your test automation
  tool as the last parameter after the special -- option. The command then
  executes your tool and captures the full output and measures the execution
  time. Both are included in the result test automation run in Testmo (note the
  space after --):
​
  $ TESTMO_TOKEN=******** testmo automation:run:submit \
      --instance https://<your-name>.testmo.net \
      --project-id 1 \
      --name "New test run" \
      --source "backend-unit" \
      --results reports/*.xml \
      -- npm test <-- replace this line with your testing tool command line
        ^ space!
​
  To also include any additional resources previously stored in a file with the
  automation:resources:* commands, such as custom fields, links and artifacts,
  specify the resource file with the --resources option:
​
  $ TESTMO_TOKEN=******** testmo automation:run:submit \
      <other-options> \
      --resources testmo-resources.json
Creating and completing test runs
If you would like to create multiple threads for e.g. parallel tests, you would first create a test run, then submit the threads (see below) and finally complete the run. You would use the automation:run:create and automation:run:complete commands for this.
$ testmo help automation:run:create
​
Usage: testmo automation:run:create [options]
​
Creates a new automation run for parallel execution
​
Options:
  --instance <url>     Required: The full address of your Testmo instance
                       (https://***.testmo.net)
  --project-id <id>    Required: The ID of the project this run is added to
  --name <name>        Required: The name of the newly created test run
  --source <source>    Required: The name of the source for this run and future
                       runs of this same source/suite
  --tags <tags...>     Tags for the new run. Tags can also be used to
                       automatically link this run to milestones
  --config <name>      Automatically find an existing configuration for this
                       run based on the config name
  --config-id <id>     Or specify a configuration by using the config's ID
  --milestone <name>   Automatically find an existing milestone for this run
                       based on the milestone name
  --milestone-id <id>  Or specify a milestone by using the milestone's ID
  --resources <file>   An optional JSON file with custom fields, links and
                       artifacts (can be built with the automation:resources
                       commands)
  --ansi               Force ANSI console output (colors, animations)
  --no-ansi            Do not use ANSI console output (colors, animations)
  --debug              Output additional debug information
  -h, --help           Display help for command
​
Details:
  Use this command to create a new automation run you intend to use for parallel
  execution, e.g. as part of parallel CI tasks. If successful, this command
  outputs the test automation run ID.
​
  You will need to store this ID and use it for subsequent calls to
  automation:run:submit-thread and automation:run:complete. You would usually
  use these commands for parallel execution of automated tests:
​
  automation:run:create            <-- Create the run before your tests start
   | automation:run:submit-thread  <-- Submit results in your parallel CI tasks
   | automation:run:submit-thread  <-- Submit results in your parallel CI tasks
   | automation:run:submit-thread  <-- Submit results in your parallel CI tasks
  automation:run:complete          <-- Mark run as completed at the end of
                                       your CI
​
  If you just want to create an automation run and submit results without
  parallel execution, just use automation:run:submit without calling
  create/complete (not submit-thread!).
​
  Make sure to set the API authentication token as TESTMO_TOKEN environment
  variable. You can either add an authentication token for your user in your
  profile settings in Testmo. Or your Testmo admin can add a separate API user
  just for authentication.
​
Examples:
  The following example creates a new automation run:
​
  $ TESTMO_TOKEN=******** testmo automation:run:create \
      --instance https://<your-name>.testmo.net \
      --project-id 1 \
      --name "New test run" \
      --source "backend-unit"
​
  To also include any additional resources previously stored in a file with the
  automation:resources:* commands, such as custom fields, links and artifacts,
  specify the resource file with the --resources option:
​
  $ TESTMO_TOKEN=******** testmo automation:run:create \
      <other-options> \
      --resources testmo-resources.json
$ testmo help automation:run:complete
​
Usage: testmo automation:run:complete [options]
​
Marks a test automation run as completed
​
Options:
  --instance <url>      Required: The full address of your Testmo instance
                        (https://***.testmo.net)
  --run-id <id>         Required: The ID of the run to be completed (returned
                        by automation:run:create)
  --no-measure-elapsed  By default Testmo measures the run time by looking at
                        the time difference between create and complete. This
                        option disables this behavior. The test run time is then
                        based on the longest running thread
  --resources <file>    An optional JSON file with custom fields, links and
                        artifacts (can be built with the automation:resources
                        commands)
  --ansi                Force ANSI console output (colors, animations)
  --no-ansi             Do not use ANSI console output (colors, animations)
  --debug               Output additional debug information
  -h, --help            Display help for command
​
Details:
  Use this command to mark a test automation run you previously started with
  automation:run:create as completed. Once the test run has been completed, it
  cannot receive additional threads and test results.
​
  You should call this command after all your CI test tasks have been completed
  for this run and after your last call to automation:run:submit-thread.
​
  Please note that you do not need to complete a run you have started with
  automation:run:submit. It is automatically completed after the results have
  been uploaded.
​
  By default Testmo measures the run time by looking at the time difference
  between create and complete. In most situations this is the recommended
  approach as it results in accurate automation time measurement. You can
  use the --no-measure-elapsed option to disable this behavior. The time of
  the test run is then based on the longest running thread.
​
  Make sure to set the API authentication token as TESTMO_TOKEN environment
  variable. You can either add an authentication token for your user in your
  profile settings in Testmo. Or your Testmo admin can add a separate API user
  just for authentication.
​
Examples:
  The following example completes a running automation run:
​
  $ TESTMO_TOKEN=******** testmo automation:run:complete \
      --instance https://<your-name>.testmo.net \
      --run-id 1
​
  To also include any additional resources previously stored in a file with the
  automation:resources:* commands, such as custom fields, links and artifacts,
  specify the resource file with the --resources option:
​
  $ TESTMO_TOKEN=******** testmo automation:run:complete \
      <other-options> \
      --resources testmo-resources.json
Submitting threads for runs
If you create test runs with the above automation:run:create command, you can submit individual threads and test results with the automation:run:submit-thread command. You would usually use this approach for more advanced CI/CD pipeline workflows with parallel test jobs.
$ testmo help automation:run:submit-thread
​
Usage: testmo automation:run:submit-thread [options] -- [executable]
​
Creates a new thread for an existing run, submits test results and completes
the thread
​
Options:
  --instance <url>           Required: The full address of your Testmo instance
                             (https://***.testmo.net)
  --run-id <id>              Required: The ID of the run this new thread should
                             be added to (returned by automation:run:create
  --results <files...>       Required: Result log files (in JUnit XML format).
                             If you pass a single quoted parameter, it is
                             evaluated by Node's `glob` to resolve files
                             (allowing recursive search, see example below)
  --run-resources <file>     An optional JSON file with custom fields, links and
                             artifacts (can be built with the
                             automation:resources commands) for the existing run
  --thread-resources <file>  An optional JSON file with custom fields and
                             artifacts (can be built with the
                             automation:resources commands) for the new thread
  --ignore-parse-error       Ignore result file parsing errors and upload
                             remaining successfully parsed files
                             (default: false)
  --exec-suppress            Suppress standard output (stdout) of passed
                             executable (default: false)
  --exec-stop-error          Stop and don't create a run if the passed
                             executable failed (exit code not 0)
                             (default: false)
  --ansi                     Force ANSI console output (colors, animations)
  --no-ansi                  Do not use ANSI console output (colors, animations)
  --debug                    Output additional debug information
  -h, --help                 Display help for command
​
Details:
  Use this command to add a new thread to an existing running test automation
  run, submit test results and complete the thread in a single step. You would
  use this command to submit the results of a single (parallel) CI task to an
  active run.
​
  If you just want to create a test run with a single thread (i.e. a
  non-parallel single CI task), you would just use the automation:run:submit
  task instead.
​
  There are two typical ways to use this command:
​
  - Run this command in your parallel CI task after your tests finished and
    after your test result reports were generated. This command then uploads
    your test results.
​
    $ testmo automation:run:submit-thread [options]
​
  OR
​
  - You can alternatively pass the executable/command line to run your tests to
    this command. The command then also captures the output of your tests and
    measures the exact time. The output and time are then also sent to Testmo.
​
    $ testmo automation:run:submit-thread \
        [options] -- <your-test-run-command> [options]
​
  In both scenarios the command expects that your test results are generated
  and reported as JUnit XML files. Practically any test automation tool can
  report or convert test results to this format. The list of report files can
  be specified with the --results option:
​
  testmo automation:run:submit-thread [options] --results reports/*.xml
​
  It is also possible to use recursive search by specifying a quoted single
  option to be evaluated by 'glob' (don't forget the quotes):
​
  testmo automation:run:submit-thread [options] --results "reports/**/*.xml"
​
  The command automatically detects if any found files look like a valid JUnit
  XML report and ignores other files. By default, detected JUnit XML files that
  fail to parse throw an error.
​
  Make sure to set the API authentication token as TESTMO_TOKEN environment
  variable. You can either add an authentication token for your user in your
  profile settings in Testmo. Or your Testmo admin can add a separate API user
  just for authentication.
​
Examples:
  You can create a new thread and submit test results after your test run:
​
  $ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
      --instance https://<your-name>.testmo.net \
      --run-id 1 \
      --results reports/*.xml
​
  You can additionally pass the executable/command line for your test automation
  tool as the last parameter after the special -- option. The command then
  executes your tool and captures the full output and measures the execution
  time. Both are included in the result test automation run in Testmo (note the
  space after --):
​
  $ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
      --instance https://<your-name>.testmo.net \
      --run-id 1 \
      --results reports/*.xml \
      -- npm test <-- replace this line with your testing tool command line
        ^ space!
​
  To also include any additional resources previously stored in a file with the
  automation:resources:* commands, such as custom fields, links and artifacts,
  specify the resource file. You can add resources to the run and/or thread by
  using the --run-resources and --thread-resources option. Note that links
  are only support for runs and are ignored for threads.
​
  $ TESTMO_TOKEN=******** testmo automation:run:submit-thread \
      <other-options> \
      --run-resources testmo-run-resources.json \
      --thread-resources testmo-thread-resources.json
Adding fields, links and artifacts
You can use the following commands to manage a testmo-resources.json file to add fields, links and artifacts. You can pass this file to the other commands to submit these resources to runs and threads.
$ testmo help automation:resources:add-artifact
​
Usage: testmo automation:resources:add-artifact [options]
​
Adds a new artifact to the local resource file to be submitted
with an automation run
​
Options:
  --name <name>       Required: The name (e.g. filename) displayed for this
                      artifact
  --url <link>        Required: The URL to access this artifact; please make
                      sure the URL is only accessible for your team for
                      security reasons
  --note <text>       An additional note displayed for the artifact
  --file <file>       If you specify a local file, the command also collects
                      the file size and mime type (the file is not uploaded
                      to Testmo though
  --resources <file>  The JSON file this link is written to
                      (default: "testmo-resources.json")
  --ansi              Force ANSI console output (colors, animations)
  --no-ansi           Do not use ANSI console output (colors, animations)
  --debug             Output additional debug information
  -h, --help          Display help for command
​
Details:
  You can include additional resources such as custom fields, links and
  artifacts with automation runs. To make it easier to submit these resources,
  use the automation:resources:* commands to prepare a local JSON file.
​
  When you then call one of the automation:run:* commands, you can specify
  the resources file with the --resources option. All previously written
  fields, links and artifacts in this file are then submitted with the run.
​
  By default this command adds a new artifact to a local testmo-resources.json
  file if no other filename is specified. If the file does not exist yet,
  it is automatically created.
​
  What's an artifact? In this context, an artifact is any file, result or
  resource generated by your build, CI or test system you would like to
  keep. The files are not stored in Testmo. You need to store the file
  separately, e.g. in your CI system or cloud storage. With Testmo's
  automation artifact, you can include a link to the file to make it easy
  to access from your automation run.
​
  In addition to the name and URL, you can optionally specify the local file
  with this command. If you do this, the file size and mime type is
  automatically collected as well (the file is not uploaded to Testmo though).
​
Examples:
  You can write a new artifact to the resource file like this:
​
  $ testmo automation:resources:add-artifact \
      --name "" \
      --url "https://d28c7a82n.s3.amazonaws.com/artifacts/report.xml" \
      --note "Test result report"
$ testmo help automation:resources:add-field
​
Usage: testmo automation:resources:add-field [options]
​
Adds a new field to the local resource file to be submitted with an
automation run
​
Options:
  --type <type>         Required: Either string, url, text, console, html
  --name <name>         Required: The name of this field
  --value <string|url>  Value for string or URL fields; use stdin for text,
                        console or HTML
  --resources <file>    The JSON file this field is written to
                        (default:  "testmo-resources.json")
  --ansi                Force ANSI console output (colors, animations)
  --no-ansi             Do not use ANSI console output (colors, animations)
  --debug               Output additional debug information
  -h, --help            Display help for command
​
Details:
  You can include additional resources such as custom fields, links and
  artifacts with automation runs. To make it easier to submit these resources,
  use the automation:resources:* commands to prepare a local JSON file.
​
  When you then call one of the automation:run:* commands, you can specify
  the resources file with the --resources option. All previously written
  fields, links and artifacts in this file are then submitted with the run.
​
  By default this command adds a new field to a local testmo-resources.json
  file if no other filename is specified. If the file does not exist yet,
  it is automatically created.
​
  Note: you can either add short string/URL fields or longer fields (displayed
  in Testmo as text, console output or rendered HTML). Values for short
  string/URL fields need to be specified with the --value option. For text,
  console and HTML fields, pass the field value via stdin (see second example
  below).
​
Examples:
  You can write a new string/URL fields to the resource file like this:
​
  $ testmo automation:resources:add-field \
      --type string \
      --name "Version" \
      --value "2.3.1-5fbcc8d0"
​
  For text, console and HTML fields, make sure to pipe the field value via
  stdin, e.g. from a file:
​
  $ cat files.txt | testmo automation:resources:add-field \
      --type console \
      --name "File list"
$ testmo help automation:resources:add-link
​
Usage: testmo automation:resources:add-link [options]
​
Adds a new link to the local resource file to be submitted with an
automation run
​
Options:
  --name <name>       Required: The name displayed for this link
  --url <link>        Required: The URL for this link
  --note <text>       An additional note displayed for the link
  --resources <file>  The JSON file this link is written to
                      (default: "testmo-resources.json")
  --ansi              Force ANSI console output (colors, animations)
  --no-ansi           Do not use ANSI console output (colors, animations)
  --debug             Output additional debug information
  -h, --help          Display help for command
​
Details:
  You can include additional resources such as custom fields, links and
  artifacts with automation runs. To make it easier to submit these resources,
  use the automation:resources:* commands to prepare a local JSON file.
​
  When you then call one of the automation:run:* commands, you can specify
  the resources file with the --resources option. All previously written
  fields, links and artifacts in this file are then submitted with the run.
​
  By default this command adds a new link to a local testmo-resources.json
  file if no other filename is specified. If the file does not exist yet,
  it is automatically created.
​
Examples:
  You can write a new link to the resource file like this:
​
  $ testmo automation:resources:add-link \
      --name "GitHub repository" \
      --url "https://github.com/testmo-app/repo" \
      --note "Our project's git repository with all code files and issues."
Automation - Previous
Testmo CLI
Next - Administration
Overview
Last modified 9mo ago
Copy link
On this page
Submitting entire test runs
Creating and completing test runs
Submitting threads for runs
Adding fields, links and artifacts