Automation Runs

The automation/runs namespace provides API methods for automation runs and threads. You can use these methods to retrieve automation runs and threads for a project, create new runs and threads, submit test results (including custom fields, artifacts & links) and finally mark runs and threads as completed.

GET `/projects/{project_id}/automation/runs`

Returns all automation runs for a project.

This method uses pagination so you might need to request additional pages to retrieve all automation runs.

Request

"page": {
    "type": "integer",
    "format": "int64",
    "description": "Number of page (default: first page)."
},
"per_page": {
    "type": "integer",
    "format": "int64",
    "enum": [
        15,
        25,
        50,
        100
    ],
    "description": "Maximum number of items to return per page (default: 100)."
},
"sort": {
    "type": "string",
    "enum": [
        "automation_runs:created_at"
    ],
    "description": "Sort field for the list of automation runs."
},
"order": {
    "type": "string",
    "enum": [
        "asc",
        "desc"
    ],
    "description": "Sort order (ascending or descending)."
},
"config_id": {
    "type": "string",
    "description": "Comma-separated list of configurations to filter by."
},
"created_after": {
    "type": "string",
    "format": "date-time",
    "description": "Limit result to automation runs created after (in ISO8601 format and UTC time zone)."
},
"created_before": {
    "type": "string",
    "format": "date-time",
    "description": "Limit result to automation runs created before (in ISO8601 format and UTC time zone)."
},
"created_by": {
    "type": "string",
    "description": "Comma-separated list of users to filter by."
},
"milestone_id": {
    "type": "string",
    "description": "Comma-separated list of milestones to filter by."
},
"source_id": {
    "type": "string",
    "description": "Comma-separated list of automation sources to filter by."
},
"status": {
    "type": "string",
    "description": "Comma-separated list of statuses to filter by. Use: `2` for success, `3` for failure, `4` for running."
},
"tags": {
    "type": "string",
    "description": "Comma-separated list of tags to filter by."
},
"expands": {
    "type": "string",
    "description": "Comma-separated list of expands to return."
}

This method supports the following expands so you can automatically include additional information for referenced objects:

automation_sources
configs
milestones
statuses
users

Response

GET /api/v1/projects/1/automation/runs

200 OK
{
    "page": 1,
    "prev_page": null,
    "next_page": 2,
    "last_page": 2,
    "per_page": 100,
    "total": 150,
    "result": [
        {
            "id": 1,
            "project_id": 1,
            "name": "Run 1",
            "status": 2,
            ..
            "is_completed": false,
            "untested_count": 0,
            "status1_count": 90,
            "status2_count": 10,
            "status3_count": 0,
            "status4_count": 0,
            "status5_count": 0,
            "status6_count": 0,
            "status7_count": 0,
            "status8_count": 0,
            "status9_count": 0,
            "status10_count": 0,
            "status11_count": 0,
            "status12_count": 0,
            "status13_count": 0,
            "status14_count": 0,
            "status15_count": 0,
            "status16_count": 0,
            "status17_count": 0,
            "status18_count": 0,
            "status19_count": 0,
            "status20_count": 0,
            "status21_count": 0,
            "status22_count": 0,
            "status23_count": 0,
            "status24_count": 0,
            "success_count": 90,
            "failure_count": 10,
            "completed_count": 100,
            "total_count": 100,
            "thread_count": 8,
            "thread_active_count": 2,
            "thread_completed_count": 6,
            "created_at": "..",
            "created_by": 2,
            "updated_at": null,
            "updated_by": null,
            "completed_at": null,
            "completed_by": null
        },
        {
            "id": 2,
            "project_id": 1,
            "name": "Run 2",
            "status": 3,
            ..
            "is_completed": true,
            "untested_count": 1,
            "status1_count": 20,
            "status2_count": 5,
            "status3_count": 0,
            "status4_count": 0,
            "status5_count": 0,
            "status6_count": 0,
            "status7_count": 0,
            "status8_count": 0,
            "status9_count": 0,
            "status10_count": 0,
            "status11_count": 0,
            "status12_count": 0,
            "status13_count": 0,
            "status14_count": 0,
            "status15_count": 0,
            "status16_count": 0,
            "status17_count": 0,
            "status18_count": 0,
            "status19_count": 0,
            "status20_count": 0,
            "status21_count": 0,
            "status22_count": 0,
            "status23_count": 0,
            "status24_count": 0,
            "success_count": 20,
            "failure_count": 5,
            "completed_count": 25,
            "total_count": 26,
            "thread_count": 8,
            "thread_active_count": 2,
            "thread_completed_count": 6,
            "created_at": "..",
            "created_by": 2,
            "updated_at": null,
            "updated_by": null,
            "completed_at": "..",
            "completed_by": 1
        },
        ..
    ],
    "expands": {
        ..
    }
}

Examples

// Get latest 100 automation runs for project with ID 5
GET /api/v1/projects/5/automation/runs

// Get second result page (pagination)
GET /api/v1/projects/5/automation/runs?page=2

// Get latest 100 automation runs
GET /api/v1/projects/5/automation/runs

// Get latest 100 failed automation runs
GET /api/v1/projects/5/automation/runs?state=3

// Get latest automation runs created after a certain date & time
GET /api/v1/projects/5/automation/runs?created_after=2023-02-15T00:00:00.000Z

// Get automation runs and include expands
GET /api/v1/projects/5/automation/runs?expands=automation_sources,configs,users

Status codes

200 400 401 403 422 (details)

GET `/automation/runs/{automation_run_id}`

Returns a single automation run.

Request

"expands": {
    "type": "string",
    "description": "Comma-separated list of expands to return."
}

This method supports the following expands so you can automatically include additional information for referenced objects:

automation_sources
configs
milestones
statuses
users

Response

GET /api/v1/automation/runs/1

200 OK
{
    "result": {
        "id": 1,
        "project_id": 1,
        "name": "Run 1",
        "status": 2,
        ..
        "is_completed": false,
        "untested_count": 0,
        "status1_count": 90,
        "status2_count": 10,
        "status3_count": 0,
        "status4_count": 0,
        "status5_count": 0,
        "status6_count": 0,
        "status7_count": 0,
        "status8_count": 0,
        "status9_count": 0,
        "status10_count": 0,
        "status11_count": 0,
        "status12_count": 0,
        "status13_count": 0,
        "status14_count": 0,
        "status15_count": 0,
        "status16_count": 0,
        "status17_count": 0,
        "status18_count": 0,
        "status19_count": 0,
        "status20_count": 0,
        "status21_count": 0,
        "status22_count": 0,
        "status23_count": 0,
        "status24_count": 0,
        "success_count": 90,
        "failure_count": 10,
        "completed_count": 100,
        "total_count": 100,
        "thread_count": 8,
        "thread_active_count": 2,
        "thread_completed_count": 6,
        "created_at": "..",
        "created_by": 2,
        "updated_at": null,
        "updated_by": null,
        "completed_at": null,
        "completed_by": null
    },
    "expands": {
        ..
    }
}

Examples

// Get the automation run with ID 5
GET /api/v1/automation/runs/5

// Get a automation run and include expands
GET /api/v1/automation/runs/1?expands=automation_sources,configs,users

Status codes

200 400 401 403 404 422 (details)

POST `/projects/{project_id}/automation/runs`

Creates a new automation run in a target project in preparation for adding threads and test results.

Request

"name": {
    "type": "string",
    "description": "Name of the new automation run."
},
"source": {
    "type": "string",
    "description": "Name of the source for the new automation run. If this source does not already exist in the target project, Testmo automatically creates a new one. It is recommended to keep source names short. Good examples are `backend`, `frontend` or `mobile-iphone`."
},
"config": {
    "type": "string",
    "description": "Name of the configuration for the new automation run."
},
"config_id": {
    "type": "integer",
    "format": "int64",
    "description": "ID of the configuration for the new automation run. If both `config` and `config_id` are specified, `config_id` is given precedence."
},
"milestone": {
    "type": "string",
    "description": "Name of the milestone for the new automation run."
},
"milestone_id": {
    "type": "integer",
    "format": "int64",
    "description": "ID of the milestone for the new automation run. If both `milestone` and `milestone_id` are specified, `milestone_id` is given precedence."
},
"tags": {
    "type": "array",
    "description": "List of tags for the new automation run. If a milestone in the same project has one or more matching automation tags, the new automation run is automatically linked to that milestone (unless `milestone` or `milestone_id` is also specified).",
    "items": {
        "type": "string"
    }
},
"artifacts": {
    "type": "array",
    "description": "List of externally stored test artifacts to link to the new automation run (such as log files, screenshots or test data).",
    "items": {
        "type": "object",
        "required": [
            "name",
            "url"
        ],
        "properties": {
            "name": {
                "type": "string",
                "description": "Name or file name of the test artifact."
            },
            "note": {
                "type": "string",
                "description": "Short note or summary of the test artifact with a maximum of 80 characters."
            },
            "url": {
                "type": "string",
                "description": "Link to an external resource to download the test artifact."
            },
            "mime_type": {
                "type": "string",
                "description": "MIME type of the test artifact (for example, `image/png`, `text/plain` or `application/octet-stream`)."
            },
            "size": {
                "type": "integer",
                "format": "int64",
                "description": "File size of the test artifact in bytes."
            }
        }
    }
},
"fields": {
    "type": "array",
    "description": "List of fields to attach to the new automation run (such as environment variables, errors or terminal output).",
    "items": {
        "type": "object",
        "required": [
            "type",
            "name"
        ],
        "properties": {
            "type": {
                "type": "integer",
                "format": "int8",
                "enum": [
                    1,
                    2,
                    3,
                    4,
                    5
                ],
                "description": "Type of the field. Use: `1` for regular strings, `2` for plain text, `3` for HTML text, `4` for text to display in a terminal/console frame (with a monospaced font), `5` for URLs."
            },
            "name": {
                "type": "string",
                "description": "Name of the field."
            },
            "value": {
                "type": "string",
                "description": "Value of the field (interpreted as defined by `type`)."
            },
            "meta": {
                "type": "object",
                "additionalProperties": {
                    "type": "string"
                },
                "description": "Meta fields to attach to the field (to store extra information with a field)."
            },
            "is_highlight": {
                "type": "boolean",
                "description": "Defines if the field (or related meta fields) are highlighted in the user interface."
            }
        }
    }
},
"links": {
    "type": "array",
    "description": "List of links to attach to the new automation run (such as a link back to the build in the CI tool that triggered the tests).",
    "items": {
        "type": "object",
        "required": [
            "name",
            "url"
        ],
        "properties": {
            "name": {
                "type": "string",
                "description": "Name or file name of the link."
            },
            "note": {
                "type": "string",
                "description": "Short note or summary of the link with a maximum of 80 characters."
            },
            "url": {
                "type": "string",
                "description": "Link to an external resource or website."
            }
        }
    }
}

Example

POST /api/v1/projects/1/automation/runs
{
    "name": "Run 1",
    "source": "frontend",
    "tags": [
        "tag-a",
        "tag-b"
    ],
    "artifacts": [
        {
            "name": "Artifact 1.png",
            "url": "https://example.com/artifact1.png",
            "mime_type": "image/png",
            "size": 1048576
        }
    ],
    "fields": [
        {
            "name": "Field 1",
            "type": 4,
            "value": ".."
        }
    ],
    "links": [
        {
            "name": "Link 1",
            "url": "https://example.com"
        }
    ]
}

Returns the ID of the created automation run on success:

201 Created
{
    "id": 1
}

Status codes

201 400 401 403 404 422 (details)

POST `/automation/runs/{automation_run_id}/append`

Appends test artifacts, fields or links to an existing automation run.

Request

"artifacts": {
    "type": "array",
    "description": "List of externally stored test artifacts to link to the automation run (such as log files, screenshots or test data).",
    "items": {
        "type": "object",
        "required": [
            "name",
            "url"
        ],
        "properties": {
            "name": {
                "type": "string",
                "description": "Name or file name of the test artifact."
            },
            "note": {
                "type": "string",
                "description": "Short note or summary of the test artifact with a maximum of 80 characters."
            },
            "url": {
                "type": "string",
                "description": "Link to an external resource to download the test artifact."
            },
            "mime_type": {
                "type": "string",
                "description": "MIME type of the test artifact (for example, `image/png`, `text/plain` or `application/octet-stream`)."
            },
            "size": {
                "type": "integer",
                "format": "int64",
                "description": "File size of the test artifact in bytes."
            }
        }
    }
},
"fields": {
    "type": "array",
    "description": "List of fields to attach to the automation run (such as environment variables, errors or terminal output).",
    "items": {
        "type": "object",
        "required": [
            "type",
            "name"
        ],
        "properties": {
            "type": {
                "type": "integer",
                "format": "int8",
                "enum": [
                    1,
                    2,
                    3,
                    4,
                    5
                ],
                "description": "Type of the field. Use: `1` for regular strings, `2` for plain text, `3` for HTML text, `4` for text to display in a terminal/console frame (with a monospaced font), `5` for URLs."
            },
            "name": {
                "type": "string",
                "description": "Name of the field."
            },
            "value": {
                "type": "string",
                "description": "Value of the field (interpreted as defined by `type`)."
            },
            "meta": {
                "type": "object",
                "additionalProperties": {
                    "type": "string"
                },
                "description": "Meta fields to attach to the field (to store extra information with a field)."
            },
            "is_highlight": {
                "type": "boolean",
                "description": "Defines if the field (or related meta fields) are highlighted in the user interface."
            }
        }
    }
},
"links": {
    "type": "array",
    "description": "List of links to attach to the automation run (such as a link back to the build in the CI tool that triggered the tests).",
    "items": {
        "type": "object",
        "required": [
            "name",
            "url"
        ],
        "properties": {
            "name": {
                "type": "string",
                "description": "Name or file name of the link."
            },
            "note": {
                "type": "string",
                "description": "Short note or summary of the link with a maximum of 80 characters."
            },
            "url": {
                "type": "string",
                "description": "Link to an external resource or website."
            }
        }
    }
}

Example

POST /api/v1/automation/runs/1/append
{
    "artifacts": [
        {
            "name": "Artifact 1.png",
            "url": "https://example.com/artifact1.png",
            "mime_type": "image/png",
            "size": 1048576
        }
    ],
    "fields": [
        {
            "name": "Field 1",
            "type": 4,
            "value": ".."
        }
    ],
    "links": [
        {
            "name": "Link 1",
            "url": "https://example.com"
        }
    ]
}

204 No Content

Status codes

204 400 401 403 404 422 (details)

POST `/automation/runs/{automation_run_id}/complete`

Marks an automation run and its threads as completed and closes it for new threads and test results.

Request

"measure_elapsed": {
    "type": "boolean",
    "description": "Defines if the execution time of the automation run should automatically be set to the elapsed time between the creation and completion date of the run. This is useful in scenarios where the overall execution time of the run is independent from the execution times of its threads."
}

Example

POST /api/v1/automation/runs/1/complete
{
    "measure_elapsed": true
}

204 No Content

Status codes

204 400 401 403 404 422 (details)

POST `/automation/runs/{automation_run_id}/threads`

Creates a new thread in an automation run in preparation for adding test results.

Request

"elapsed_observed": {
    "type": "integer",
    "format": "int64",
    "description": "Observed overall elapsed (execution time) of the thread in microseconds."
},
"elapsed_computed": {
    "type": "integer",
    "format": "int64",
    "description": "Computed overall elapsed (execution time) of the thread in microseconds. The _observed_ elapsed is usually derived from measuring the execution time of tests whereas the _computed_ elapsed is calculated from result log files."
},
"artifacts": {
    "type": "array",
    "description": "List of externally stored test artifacts to link to the new thread (such as log files, screenshots or test data).",
    "items": {
        "type": "object",
        "required": [
            "name",
            "url"
        ],
        "properties": {
            "name": {
                "type": "string",
                "description": "Name or file name of the test artifact."
            },
            "note": {
                "type": "string",
                "description": "Short note or summary of the test artifact with a maximum of 80 characters."
            },
            "url": {
                "type": "string",
                "description": "Link to an external resource to download the test artifact."
            },
            "mime_type": {
                "type": "string",
                "description": "MIME type of the test artifact (for example, `image/png`, `text/plain` or `application/octet-stream`)."
            },
            "size": {
                "type": "integer",
                "format": "int64",
                "description": "File size of the test artifact in bytes."
            }
        }
    }
},
"fields": {
    "type": "array",
    "description": "List of fields to attach to the new thread (such as environment variables, errors or terminal output).",
    "items": {
        "type": "object",
        "required": [
            "type",
            "name"
        ],
        "properties": {
            "type": {
                "type": "integer",
                "format": "int8",
                "enum": [
                    1,
                    2,
                    3,
                    4,
                    5
                ],
                "description": "Type of the field. Use: `1` for regular strings, `2` for plain text, `3` for HTML text, `4` for text to display in a terminal/console frame (with a monospaced font), `5` for URLs."
            },
            "name": {
                "type": "string",
                "description": "Name of the field."
            },
            "value": {
                "type": "string",
                "description": "Value of the field (interpreted as defined by `type`)."
            },
            "meta": {
                "type": "object",
                "additionalProperties": {
                    "type": "string"
                },
                "description": "Meta fields to attach to the field (to store extra information with a field)."
            },
            "is_highlight": {
                "type": "boolean",
                "description": "Defines if the field (or related meta fields) are highlighted in the user interface."
            }
        }
    }
}

Example

POST /api/v1/automation/runs/1/threads
{
    "elapsed_observed": 600000000,
    "artifacts": [
        {
            "name": "Artifact 1.png",
            "url": "https://example.com/artifact1.png",
            "mime_type": "image/png",
            "size": 1048576
        }
    ],
    "fields": [
        {
            "name": "Field 1",
            "type": 4,
            "value": ".."
        }
    ]
}

Returns the ID of the created thread on success:

201 Created
{
    "id": 1
}

Status codes

201 400 401 403 404 422 (details)

POST `/automation/runs/threads/{automation_run_thread_id}/append`

Appends test artifacts, fields or test results to an existing thread in an automation run.

Request

"elapsed_observed": {
    "type": "integer",
    "format": "int64",
    "description": "Partial observed elapsed (execution time) in microseconds to add to the overall observed time of the thread."
},
"elapsed_computed": {
    "type": "integer",
    "format": "int64",
    "description": "Partial computed elapsed (execution time) in microseconds to add to the overall computed time of the thread. The _observed_ elapsed is usually derived from measuring the execution time of tests whereas the _computed_ elapsed is calculated from result log files."
},
"artifacts": {
    "type": "array",
    "description": "List of externally stored test artifacts to link to the thread (such as log files, screenshots or test data).",
    "items": {
        "type": "object",
        "required": [
            "name",
            "url"
        ],
        "properties": {
            "name": {
                "type": "string",
                "description": "Name or file name of the test artifact."
            },
            "note": {
                "type": "string",
                "description": "Short note or summary of the test artifact with a maximum of 80 characters."
            },
            "url": {
                "type": "string",
                "description": "Link to an external resource to download the test artifact."
            },
            "mime_type": {
                "type": "string",
                "description": "MIME type of the test artifact (for example, `image/png`, `text/plain` or `application/octet-stream`)."
            },
            "size": {
                "type": "integer",
                "format": "int64",
                "description": "File size of the test artifact in bytes."
            }
        }
    }
},
"fields": {
    "type": "array",
    "description": "List of fields to attach to the thread (such as environment variables, errors or terminal output).",
    "items": {
        "type": "object",
        "required": [
            "type",
            "name"
        ],
        "properties": {
            "type": {
                "type": "integer",
                "format": "int8",
                "enum": [
                    1,
                    2,
                    3,
                    4,
                    5
                ],
                "description": "Type of the field. Use: `1` for regular strings, `2` for plain text, `3` for HTML text, `4` for text to display in a terminal/console frame (with a monospaced font), `5` for URLs."
            },
            "name": {
                "type": "string",
                "description": "Name of the field."
            },
            "value": {
                "type": "string",
                "description": "Value of the field (interpreted as defined by `type`)."
            },
            "meta": {
                "type": "object",
                "additionalProperties": {
                    "type": "string"
                },
                "description": "Meta fields to attach to the field (to store extra information with a field)."
            },
            "is_highlight": {
                "type": "boolean",
                "description": "Defines if the field (or related meta fields) are highlighted in the user interface."
            }
        }
    }
},
"tests": {
    "type": "array",
    "description": "List of tests to add to the thread.",
    "items": {
        "type": "object",
        "required": [
            "key",
            "name",
            "status",
            "folder"
        ],
        "properties": {
            "key": {
                "type": "string",
                "pattern": "^[a-z0-9_]{1,64}$",
                "description": "Key used to identify tests across multiple automation runs (in the context of the same source). Limited to a maximum of 64 lowercase alphanumeric (a-z, 0-9) characters."
            },
            "name": {
                "type": "string",
                "description": "Name of the test."
            },
            "status": {
                "type": "string",
                "pattern": "^[a-z0-9_]+$",
                "description": "Alias of the status for the result of the test (for example, `failed` or `passed`). The status aliases can be configured in Testmo's admin area."
            },
            "folder": {
                "type": "string",
                "description": "Fully qualified name of the target folder of the test. Folders can be used to group related tests and usually map to class or type names as defined in the test automation suite."
            },
            "elapsed": {
                "type": "integer",
                "format": "int64",
                "description": "Overall execution time (observed or computed) of the test in microseconds."
            },
            "file": {
                "type": "string",
                "description": "Name of the file (with or without full path) the test is located in."
            },
            "line": {
                "type": "integer",
                "description": "Line number of the test in the given `file`."
            },
            "assertions": {
                "type": "integer",
                "description": "Number of assertions in the test."
            },
            "artifacts": {
                "type": "array",
                "description": "List of externally stored test artifacts to link to the test (such as log files, screenshots or test data).",
                "items": {
                    "type": "object",
                    "required": [
                        "name",
                        "url"
                    ],
                    "properties": {
                        "name": {
                            "type": "string",
                            "description": "Name or file name of the test artifact."
                        },
                        "note": {
                            "type": "string",
                            "description": "Short note or summary of the test artifact with a maximum of 80 characters."
                        },
                        "url": {
                            "type": "string",
                            "description": "Link to an external resource to download the test artifact."
                        },
                        "mime_type": {
                            "type": "string",
                            "description": "MIME type of the test artifact (for example, `image/png`, `text/plain` or `application/octet-stream`)."
                        },
                        "size": {
                            "type": "integer",
                            "format": "int64",
                            "description": "File size of the test artifact in bytes."
                        }
                    }
                }
            },
            "fields": {
                "type": "array",
                "description": "List of fields to attach to the test (such as environment variables, errors or terminal output).",
                "items": {
                    "type": "object",
                    "required": [
                        "type",
                        "name"
                    ],
                    "properties": {
                        "type": {
                            "type": "integer",
                            "format": "int8",
                            "enum": [
                                1,
                                2,
                                3,
                                4,
                                5
                            ],
                            "description": "Type of the field. Use: `1` for regular strings, `2` for plain text, `3` for HTML text, `4` for text to display in a terminal/console frame (with a monospaced font), `5` for URLs."
                        },
                        "name": {
                            "type": "string",
                            "description": "Name of the field."
                        },
                        "value": {
                            "type": "string",
                            "description": "Value of the field (interpreted as defined by `type`)."
                        },
                        "meta": {
                            "type": "object",
                            "additionalProperties": {

GET /projects/{project_id}/automation/runs

Request

Response

Examples

Status codes

GET /automation/runs/{automation_run_id}

Request

Response

Examples

Status codes

POST /projects/{project_id}/automation/runs

Request

Example

Status codes

POST /automation/runs/{automation_run_id}/append

Request

Example

Status codes

POST /automation/runs/{automation_run_id}/complete

Request

Example

Status codes

POST /automation/runs/{automation_run_id}/threads

Request

Example

Status codes

POST /automation/runs/threads/{automation_run_thread_id}/append

Request

GET `/projects/{project_id}/automation/runs`

GET `/automation/runs/{automation_run_id}`

POST `/projects/{project_id}/automation/runs`

POST `/automation/runs/{automation_run_id}/append`

POST `/automation/runs/{automation_run_id}/complete`

POST `/automation/runs/{automation_run_id}/threads`

POST `/automation/runs/threads/{automation_run_thread_id}/append`