Bitbucket

Testmo supports full integration with Bitbucket Pipelines to submit test automation results from your CI pipelines and deployment workflows to implement Bitbucket test management. This only requires a few lines of code and is easy to set up. Any test automation tool, framework and platform can be used.

Also works with Atlassian Jira

You can also integrate Testmo with Atlassian Jira to implement a full testing & DevOps workflow with Atlassian tools.

You can integrate simple Bitbucket Pipelines workflows (such as running just a single test automation suite) as well as complex workflows with multiple test suites (e.g. frontend, backend, API etc.) and even parallel test steps with multiple threads.

Regardless of your specific configuration and workflow, the basic concept of integrating your Bitbucket Pipelines workflows with Testmo is always the same:

  1. You execute your automated tests as part of your Bitbucket CI workflow. It doesn't matter if you run a single test suite, multiple test suites or even parallel test steps.

  2. You configure your automated tests to generate a result file using the standard JUnit XML file format. This file format has become the de facto standard format to exchange test results between testing tools. So practically any test automation tool and framework either supports this format directly or can convert to this format.

  3. You then use the Testmo CLI tool to easily submit the results to Testmo, enabling you to track your test suites over time, identify problematic test cases and improve testing performance.

Basic Bitbucket & Testmo workflow

The following Bitbucket Pipelines workflow shows how to run a simple test automation run as part of your CI workflow and submit the test results to Testmo. In this example we are using a test suite with the Mocha/Chai (Node.js/Javascript) testing framework. Remember that you can use any test automation tool and framework with Testmo and we are just using this framework for our example.

We are using the Testmo CLI tool here as it makes it very easy to run our tests and submit all results with a single command line call (using the automation:run:submit command).

We are also sending a few additional fields to Testmo here, such as a link back to the Bitbucket run as well as the Git commit hash. These details are then displayed in Testmo along with the test results, console output, exit code etc.

You can find the full code for this project on Bitbucket in the example project repository. You can also find a detailed explanation of this project along with more code samples and screenshots in our reference guide:

Guide: Bitbucket CI Pipelines Test Automation & Reporting ➞

Important: For this code to work, it is required to define the variables TESTMO_TOKEN and TESTMO_URL in Bitbucket with your Testmo address and API token as explained below.

# bitbucket-pipelines.yml
image: node:19

pipelines:
  default:
    - step:
        name: Build
        script:
          - echo "Building .."

    - step:
        name: Test
        caches:
          - npm
        script:
          # Install your project packages & dependencies here
          # [..]

          # Install Testmo CLI tool locally (then use npx testmo .. to run it)
          - npm install --no-save @testmo/testmo-cli

          # Optionally add a couple of fields such as the git hash and link to the build
          - npx testmo automation:resources:add-field --name git --type string
              --value ${BITBUCKET_COMMIT:0:7} --resources resources.json
          - BUILD_URL="$BITBUCKET_GIT_HTTP_ORIGIN/addon/pipelines/home#!/results/$BITBUCKET_BUILD_NUMBER"
          - npx testmo automation:resources:add-link --name build
              --url $BUILD_URL --resources resources.json

          # Run automated tests and report results to Testmo
          - npx testmo automation:run:submit
              --instance "$TESTMO_URL"
              --project-id 1
              --name "Test run" \
              --source "frontend" \
              --resources resources.json
              --results results/*.xml
              -- <your-test-command> # Run your automation tool; note space after --

    - step:
        name: Deploy
        script:
          - echo "Deploying .."

definitions:
  caches:
    npm: ~/.npm

Full example repository on Bitbucket

Advanced Bitbucket workflows

We also support advanced Bitbucket Pipelines workflows. For example, you can submit test results for many test suites from a Bitbucket CI run and Testmo also fully supports parallel test runs.

  • Running multiple test suites in a build If you have multiple test suites for your project (such as frontend, backend, API, mobile tests etc.) you can simply submit these as separate test runs from the same Bitbucket CI pipeline. You would just use multiple calls to automation:run:submit to submit each test suite separately. Important: if you have different test suites, you need to use different source names for the test runs in Testmo. This way Testmo can differentiate where the test results are coming from. Learn more about test automation sources.

  • Using parallel test steps with Bitbucket Pipelines Testmo also fully supports tracking test runs with multiple parallel test steps. So if you have configured your Bitbucket workflow to run multiple parallel steps to speed up your test execution, you can also submit each step as a separate thread of the same run to Testmo. You can learn more about implementing this in the following example.

Bitbucket parallel testing workflow

Testmo fully supports Bitbucket Pipelines workflows with parallel test steps, so you can submit the results as separate threads of the same run to Testmo. You can then see the results (and console output, exit codes & testing times) of each thread separately, as well as the overall results and statistics for the entire test run in Testmo. The following illustration shows how to implement the workflow with Bitbucket Pipelines.

In this workflow we use separate steps to set up our test run (test-setup), then submit a thread for each testing step of the run (test), and finally mark the test run as completed (test-complete).

We have a detailed guide with example code, screenshots and explanations on implementing this for Bitbucket. You can also find a complete example project repository on Bitbucket. You can find the full guide here:

Guide: Bitbucket CI Parallel Test Automation Pipelines ➞

# bitbucket-pipelines.yml
image: node:19

definitions:
  caches:
    npm: ~/.npm
  steps:
    - step: &test
        name: Test
        caches:
          - npm
        script:
          - npm ci

          # Run automated tests and report results to Testmo
          - npx testmo automation:run:submit-thread
              --instance "$TESTMO_URL"
              --run-id "$(cat testmo-run-id.txt)"
              --results results/*.xml
              -- npm run mocha-junit-parallel # Note space after --

pipelines:
  default:
    - step:
        name: Build
        script:
          - echo "Building .."

    - step:
        name: Test setup
        caches:
          - npm
        script:
          - npm ci

          # Optionally add a couple of fields such as the git hash
          # and link to the build
          - npx testmo automation:resources:add-field --name git --type string
              --value ${BITBUCKET_COMMIT:0:7} --resources resources.json
          - BUILD_URL="$BITBUCKET_GIT_HTTP_ORIGIN/addon/pipelines/home#!/results/$BITBUCKET_BUILD_NUMBER"
          - npx testmo automation:resources:add-link --name build
              --url $BUILD_URL --resources resources.json

          # Create test run
          - npx testmo automation:run:create
              --instance "$TESTMO_URL"
              --project-id 1
              --name "Parallel mocha test run"
              --resources resources.json
              --source "unit-tests" > testmo-run-id.txt
        artifacts:
          - testmo-run-id.txt

    - parallel:
      - step: *test
      - step: *test
      - step: *test
      - step: *test

    - step:
        name: Test complete
        caches:
          - npm
        script:
          - npm ci

          # Mark test run completed
          - npx testmo automation:run:complete
              --instance "$TESTMO_URL"
              --run-id "$(cat testmo-run-id.txt)"

    - step:
        name: Deploy
        script:
          - echo "Deploying .."

Full example repository on Bitbucket

Important security considerations

When working with the Testmo API and CLI to submit test automation results to Testmo, you need to specify an API key for the authentication. You can learn more about generating API keys in Testmo here.

When integrating Testmo with your Bitbucket projects, it is important not to store the API key in your code repository. If you store the API key in your code repository, everyone with access to the repository would be able to access the key. You should never add and store your Testmo API key as part of your code repository.

Instead, use Bitbucket's repository variables feature to securely store keys and passwords. You can then make these secrets available as environment variables in your pipeline. You can see this in the example configuration above, as well as in our detailed Bitbucket guide. You can also learn more about Bitbucket repository variables in the official documentation.

Last updated