Issue Trackers
Enterprise auth
CircleCI
Testmo comes with full CircleCI integration so you can submit your automated test results from your CircleCI pipeline to Testmo to implement CircleCI test management. This only requires a few lines of code and is easy to set up. Any test automation tool, framework and platform can be used.
You can integrate simple CircleCI pipelines (such as running just a single test automation suite) as well as complex workflows with multiple test suites (e.g. frontend, backend, API etc.) and even parallel test jobs with multiple threads.
Regardless of your specific configuration and workflow, the basic concept of integrating your CircleCI pipeline with Testmo is always the same:
- 1.You execute your automated tests as part of your CircleCI pipeline. It doesn't matter if you run a single test suite, multiple test suites or even parallel test jobs.
- 2.You configure your automated tests to generate a result file using the standard JUnit XML file format. This file format has become the de facto standard format to exchange test results between testing tools. So practically any test automation tool and framework either supports this format directly or can convert to this format.
- 3.You then use the Testmo CLI tool to easily submit the results to Testmo, enabling you to track your test suites over time, identify problematic test cases and improve testing performance.
Further links, references and guides
Basic CircleCI & Testmo workflow
The following CircleCI configuration shows how to run a simple test automation run as part of your CI workflow and submit the test results to Testmo. In this example we are using a test suite with the Mocha/Chai (NodeJS/Javascript) testing framework. Remember that you can use any test automation tool and framework with Testmo and we are just using this framework for our example.
We are using the Testmo CLI tool here as it makes it very easy to run our tests and submit all results with a single command line call (using the
automation:run:submit command).We are also sending a few additional fields to Testmo here, such as a link back to the CircleCI run as well as the Git commit hash. These details are then displayed in Testmo along with the test results, console output, exit code etc.
You can find the full code for this project on GitHub in the example project repository. You can also find a detailed explanation of this project along with more code samples and screenshots in our reference guide:
1
# .circleci/config.yml
2
version: 2.1
3
​
4
jobs:
5
build:
6
docker:
7
- image: cimg/node:current
8
steps:
9
- checkout
10
​
11
# Restore previous node package cache if there is one
12
- restore_cache:
13
key: node-{{ checksum "package-lock.json" }}
14
​
15
# Install or update packages
16
- run: npm ci
17
​
18
# Save cache
19
- save_cache:
20
key: node-{{ checksum "package-lock.json" }}
21
paths:
22
- ~/.npm
23
​
24
# Store project files for the next job
25
- persist_to_workspace:
26
root: ~/
27
paths:
28
- project
29
​
30
test:
31
docker:
32
- image: cimg/node:current
33
​
34
steps:
35
# Start with the initial build
36
- attach_workspace:
37
at: ~/
38
​
39
# Install Testmo CLI tool
40
- run: npm install --no-save @testmo/testmo-cli
41
​
42
# Optionally add a couple of fields such as the git hash and link to the build
43
- run:
44
command: |
45
npx testmo automation:resources:add-field --name git --type string \
46
--value ${CIRCLE_SHA1:0:7} --resources resources.json
47
npx testmo automation:resources:add-link --name build \
48
--url $CIRCLE_BUILD_URL --resources resources.json
49
​
50
# Run tests and report results to Testmo
51
- run:
52
command: |
53
npx testmo automation:run:submit \
54
--instance $TESTMO_URL \
55
--project-id 1 \
56
--name "Mocha test run" \
57
--source "unit-tests" \
58
--resources resources.json \
59
--results results/*.xml \
60
-- npm run mocha-junit # Note space after --
61
​
62
# Store test artifacts and results
63
- store_artifacts:
64
path: results
65
destination: test_results
66
- store_test_results:
67
path: results
68
​
69
workflows:
70
version: 2
71
full:
72
jobs:
73
- build
74
- test:
75
requires:
76
- build
77
context:
78
- testmo
Copied!
Advanced CircleCI workflows
We also support advanced CircleCI workflows. For example, you can submit test results for many test suites from a CircleCI workflow and Testmo also fully supports parallel test runs.
- Running multiple test suites in a build If you have multiple test suites for your project (such as frontend, backend, API, mobile tests etc.) you can simply submit these as separate test runs from the same CircleCI pipeline. You would just use multiple calls to
automation:run:submitto submit each test suite separately. Important: if you have different test suites, you need to use different source names for the test runs in Testmo. This way Testmo can differentiate where the test results are coming from. Learn more about test automation sources. - Using parallel test jobs with CircleCI Testmo also fully supports tracking test runs with multiple parallel testing jobs. So if you have configured your CircleCI pipeline to run multiple parallel jobs to speed up your test execution, you can also submit each job as a separate thread of the same run to Testmo. You can learn more about implementing this in the following example.
CircleCI parallel testing workflow
Testmo fully supports CircleCI workflows with parallel test steps, so you can submit the results as separate threads of the same run to Testmo. You can then see the results (and console output, exit codes & testing times) of each thread separately, as well as the overall results and statistics for the entire test run in Testmo. The following illustration shows how to implement the workflow with CircleCI.
Example parallel testing workflow
In this workflow we use separate jobs to set up our test run (
test-setup), then submit a thread for each testing job of the run (test), and finally mark the test run as completed (test-complete).We have a detailed guide with example code, screenshots and explanations on implementing this for CircleCI. You can also find a complete example project repository on GitHub. You can find the full guide here:
1
# .circleci/config.yml
2
version: 2.1
3
​
4
jobs:
5
build:
6
docker:
7
- image: cimg/node:current
8
steps:
9
- checkout
10
​
11
# Restore previous node package cache if there is one
12
- restore_cache:
13
key: node-{{ checksum "package-lock.json" }}
14
​
15
# Install or update packages
16
- run: npm ci
17
​
18
# Save cache
19
- save_cache:
20
key: node-{{ checksum "package-lock.json" }}
21
paths:
22
- ~/.npm
23
​
24
# Store project files for the next job
25
- persist_to_workspace:
26
root: ~/
27
paths:
28
- project
29
​
30
test-setup:
31
docker:
32
- image: cimg/node:current
33
steps:
34
# Start with the initial build
35
- attach_workspace:
36
at: ~/
37
​
38
# Optionally add a couple of fields such as the git hash and link to the build
39
- run:
40
name: Set up resources
41
command: |
42
npx testmo automation:resources:add-field --name git --type string \
43
--value ${CIRCLE_SHA1:0:7} --resources resources.json
44
npx testmo automation:resources:add-link --name build \
45
--url $CIRCLE_BUILD_URL --resources resources.json
46
​
47
# Create test run in Testmo and store ID in file
48
- run:
49
name: Create test run in Testmo
50
command: |
51
npx testmo automation:run:create \
52
--instance $TESTMO_URL \
53
--project-id 1 \
54
--name "Parallel mocha test run" \
55
--resources resources.json \
56
--source "unit-tests" > testmo-run-id.txt
57
​
58
# Update workspace to persist run ID file
59
- persist_to_workspace:
60
root: ~/
61
paths:
62
- project
63
​
64
test:
65
docker:
66
- image: cimg/node:current
67
parallelism: 4
68
steps:
69
# Start with the initial build
70
- attach_workspace:
71
at: ~/
72
​
73
# Split & run tests and report results to Testmo
74
- run:
75
name: Split and run tests
76
command: |
77
npx testmo automation:run:submit-thread \
78
--instance $TESTMO_URL \
79
--run-id $(cat testmo-run-id.txt) \
80
--results results/*.xml \
81
-- npm run mocha-junit-parallel # Note space after --
82
​
83
# Store test artifacts and results
84
- store_artifacts:
85
path: results
86
destination: test_results
87
- store_test_results:
88
path: results
89
​
90
test-complete:
91
docker:
92
- image: cimg/node:current
93
steps:
94
# Start with the initial build
95
- attach_workspace:
96
at: ~/
97
​
98
# Mark run as completed
99
- run:
100
name: Mark run as completed
101
command: |
102
npx testmo automation:run:complete \
103
--instance $TESTMO_URL \
104
--run-id $(cat testmo-run-id.txt)
105
​
106
deploy:
107
docker:
108
- image: cimg/node:current
109
steps:
110
# Start with the initial build
111
- attach_workspace:
112
at: ~/
113
​
114
# You would deploy your code here
115
- run: echo "Deploying code here"
116
​
117
workflows:
118
version: 2
119
full:
120
jobs:
121
- build:
122
context:
123
- testmo
124
- test-setup:
125
requires:
126
- build
127
context:
128
- testmo
129
- test:
130
requires:
131
- test-setup
132
context:
133
- testmo
134
- test-complete:
135
requires:
136
- test
137
context:
138
- testmo
139
- deploy:
140
requires:
141
- test-complete
142
context:
143
- testmo
Copied!
Important security considerations
When working with the Testmo API and CLI to submit test automation results to Testmo, you need to specify an API key for the authentication. You can learn more about generating API keys in Testmo here.
When integrating Testmo with your CircleCI pipeline, it is important not to store the API key in your code repository. If you store the API key in your code repository (e.g. in Git), everyone with access to the repository would be able to access the key. You should never add and store your Testmo API key as part of your code repository.
Instead, use CircleCI's secrets feature to securely store keys and passwords. You can then make these secrets available as environment variables in your pipeline. You can see this in the example configuration above, as well as in our detailed CircleCI guide. You can also learn more about CircleCI secret management in the official documentation.
Last modified 5mo ago