Contributing

Tips and guidelines to optimizing and developing test automation tools to work well with Testmo.

Are you working on an automation tool, are an open source developer/maintainer or a Testmo enthusiast looking into improving a tool to work better with Testmo? Great! In this guide we are sharing some tips on how to improve & optimize tools for Testmo. Also feel free to contact us at any time if you have questions or need additional tips.

The ideal test automation integration workflow with Testmo is as follows:

• Use a standard test automation tool/framework to write tests
• Configure the tool to generate JUnit XML files with the test results
• Optionally extend the tests with additional fields, steps & attachments
• Then use our CLI tool to submit the results from the XML result file(s) to Testmo

This workflow has many advantages. It works well with any test automation tool & platform, as well as in any environment (any operating system, CI services, containers, developer machines etc.), as we've designed our CLI tool to be easily deployed & used everywhere.

This workflow also doesn't require any custom programming, API code, complicated configurations or similar. Testers and developers also automatically benefit from our CLI tool's many optimizations & extra features, such as automatic fast bulk uploads, field mapping & parallel test job support. Therefore, here are some do's and don'ts when it comes to building tools optimized for Testmo:

✅ Do: Optimize the automation tool to write JUnit-style XML files

✅ Do: Extend the JUnit XML support with test case properties (to enable all Testmo features)

✅ Do: Alternatively develop a new Testmo-optimized logger/reporter for JUnit XML files

✅ Do: Encourage testers to generate JUnit XML files & to use our CLI tool

❌ Don't: Extend a tool by calling Testmo's API directly (this is less flexible for users)

Optimizing JUnit XML files for Testmo

It is easy to optimize JUnit XML files to better support Testmo (and other test management tools). Here are a few tips for optimizations & useful features.

Add test case property support

The best way to optimize a test automation tool/framework for Testmo and other test management or reporting tools is to extend the existing JUnit XML file format to support test case properties. This is a popular extension to the JUnit XML format that is used by many tools. If a tool supports test case properties for XML files, then all Testmo features can be used:

<testcase name="testCase3" classname="Tests.Registration" time="3.441">
    <properties>
        <property name="priority" value="high" />
        <property name="language" value="english" />
        <property name="author" value="Adrian" />
        <property name="attachment" value="https://ci/files/22-dashboard.png" />
        <property name="attachment" value="https://ci/files/23-users.png" />

        <!-- Optional support for properties with text values -->
        <property name="description"><![CDATA[
            This text describes the purpose of this test case and provides
            an overview of what the test does and how it works.
        ]]></property>
    </properties>

    <!-- Optional output or error/failure/skipped -->
</testcase>

When adding support for test case properties, ideally choose an API that allows multiple properties with the same name. This can be useful to easily include multiple attachments, steps etc. Otherwise testers need to add a unique index after the name, such as attachment1, attachment2 etc. Here is a pseudo-code API to illustrate this:

// Pseudo code example for test automation tools/frameworks
function testAuthenticationLogin() {
    // Offer a basic API to add test case properties
    this.test.properties.push({ name: 'author', value: 'Adrian' });
    this.test.properties.push({ name: 'priority', value: 'high' });

    // Ideally also support multiple properties with the same name
    this.test.properties.push({ name: 'attachment', value: 'dashboard.png' });
    this.test.properties.push({ name: 'attachment', value: 'users.png' });
    this.test.properties.push({ name: 'attachment', value: 'error.png' });
    
    // Property names should allow special characters, e.g.:
    this.test.properties.push({ name: 'url:github', value: 'https://github.com' });
    this.test.properties.push({ name: 'step[passed]', value: 'Step 1..' });
    this.test.properties.push({ name: 'step[passed]', value: 'Step 2..' });
    this.test.properties.push({ name: 'step[failed]', value: 'Step 3..' });

    // Multi-line text properties can be useful for HTML or longer content
    this.test.properties.push({ 
        name: 'html:description', 
        value: '<h2>Content</h2> ..',
        text: true 
    });
}

If you are contributing to an existing open source automation tool/framework, ideally contribute to and extend the existing JUnit XML logger/reporter to implement these features and work with their maintainers. This way other testers and teams can also benefit from these changes.

Optimize class and test names

Testmo and other tools automatically use the class names and test names to nicely group test results into folders. It is encouraged to use the class name to indicate the hierarchy of the tests, and only output the name of the test (without the class/folder name) for the tests' name attributes. Here's a full example:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
    <testsuite name="Tests.Registration">
        <!-- Test case names should NOT include the class name again -->
        <testcase name="testCase1" classname="Tests.Registration" />
        <testcase name="testCase2" classname="Tests.Registration" />
        <testcase name="testCase3" classname="Tests.Registration" />
    </testsuite>
    <testsuite name="Tests.Authentication">
        <testsuite name="Tests.Authentication.Login">
            <!-- The full hierarchy should be included in class names -->
            <testcase name="testCase4" classname="Tests.Authentication.Login" />
            <testcase name="testCase5" classname="Tests.Authentication.Login" />
            <testcase name="testCase6" classname="Tests.Authentication.Login" />
        </testsuite>
        <testcase name="testCase7" classname="Tests.Authentication" />
        <testcase name="testCase8" classname="Tests.Authentication" />
        <testcase name="testCase9" classname="Tests.Authentication">
            <failure message="Assertion error message" type="AssertionError">
                <!-- Call stack printed here -->
            </failure>            
        </testcase>
    </testsuite>
</testsuites>

Or develop new XML logger/reporter

If a test automation tool/framework doesn't have good JUnit-style XML file support, or the existing maintainers aren't interested in supporting test case properties, then you can fork/build a simple new third-party JUnit XML logger. Generating JUnit XML files is pretty simple and we also have full documentation for this format:

• Testmo JUnit XML support

• JUnit XML file format & examples

Usually you can start by forking an existing reporter/logger library or module for the testing tool/framework you are targeting. You can then either write a new JUnit XML logger, or extend the existing JUnit XML logger to extend it with additional features (such as test case properties as shown above).

Publish, promote & let us know

Once you have optimized and added better Testmo support to a testing tool/framework, we encourage you to publish and promote the integration, and let us know so we can help spread the word. Here are some ideas to promote and make it easier to use the integration:

• Publish the fork, library or code on GitHub

• Publish your package to your platform's registry (such as NPM, Maven, NuGet etc.)

• Share the integration on Twitter and LinkedIn (so we can re-share it)

• Update the tool's documentation to mention the Testmo support/optimization

• Write a blog posting explaining how to use the tool with Testmo