Automation Jobs FAQ's
In Beta & Early Access
Testmo now allows you to launch test automation jobs in your CI/CD pipelines directly from Testmo. Read more below, or sign up for beta access here:
When launching automation jobs from Testmo, your CI provider may return various response codes depending on the outcome of your job. We aim to answer frequently asked questions relating to those responses below.
Testmo will usually display an error message in the log to make it easy to understand what the cause is, however there might be times when this isn’t made perfectly clear from there. Remember that you can also always review the error responses straight from the GitHub or GitLab documentation to complement the information provided by Testmo.
GitHub Actions
Testmo’s existing GitHub app requires enhanced permissions to function correctly with the new automation launching capabilities. The following steps should be completed by Testmo & GitHub admins for a seamless experience:
Approve (or ask your GitHub project administrator to approve) the new settings via the Testmo App:
Navigate to the Testmo GitHub app via https://github.com/apps/testmo-app and hit the Configure button to see which GitHub repositories the app is installed for.
Approve the permission change for each repository you want to use the app in.
Re-authorize the existing Testmo-GitHub connection:
Navigate to the Admin > Inegrations view and click the "Edit" button on your existing connection(s)
Click the "Authorize with GitHub" button to re-authorize the app with the new permissions.
If you haven't already installed and connected GitHub with your Testmo instance, no actions are required. Installation of the GitHub app in your repositories will require you to confirm the new permissions during the initial setup stage.
GitHub returned error code 403
If you get this error code with the message
“This installation has been suspended”, please ask your GitHub admin to Unsuspend the Testmo App on GitHub.If the message is
"Resource not accessible by integration", you need to approve the new permissions for GitHub Integration App and reauthorize the connection in Testmo to allow Automation Launching.Navigate to the Testmo GitHub app via https://github.com/apps/testmo-app and hit the Configure button to see which GitHub repositories the app is installed for.
Approve the permission change for each repository you want to use the app in.
You may also need to re-authorize the GitHub connection in Testmo:
Navigate to the Admin > Inegrations view and click the "Edit" button on your existing connection(s)
Click the "Authorize with GitHub" button to re-authorize the app with the new permissions.
GitHub returned error code 404
If you make a request to access a repository or workflow you don’t have access to or are not mapped in Testmo, you will receive a 404 response.
Make sure that the repository you are trying to reach is available and is selected in the integration mapping settings in Admin > Integrations > GitHub.
This error code might also mean that the Testmo GitHub App has been uninstalled. If none of the steps above work, please contact your GitHub admin to verify that the Testmo App is correctly installed on GitHub.
GitHub returned error code 422
If you omit a required parameter, mistype the parameter name, or input a wrong parameter, reference, or value, you may receive a 422 response. The message will have the details of the specific case.
GitLab CI/CD
Testmo will usually display an error message in the log to make it easy to understand what the cause is, however there might be times when this isn’t made perfectly clear from there. Remember that you can also always review the error responses straight from the GitLab documentation to complement the information.
GitLab returned error code 401
This can be caused by an issue with your connection, rather than the trigger. You can review the status of your connection to GitLab under Admin > Integrations > GitLab, by opening the connection being used by the target and clicking “Test Connection”.
GitLab returned error code 404
The
{"message":"404 Project Not Found"}response can be caused by pointing to a non-existent or mispelled group or project name/ID. Double-check your project group and/or project name/ID in the corresponding Automation target you configured in the Admin > Automation view.
GitLab returned error code 400
Triggering a pipeline using an incorrect reference (wrong branch name), which will have GitLab return a 400. In Testmo this will be displayed as
{"message":{"base”:[ “Reference not found"]}}You may have accidentally used main for the branch name in a project that uses a different branch name for its default branch, such as master.
Further FAQ's
Does the pre-existing connection remove the need to create a token?
Yes in terms of tokens specific to triggering pipelines, workflows, etc in your tools, but remember that the standard connection requires either a token or an integration to establish the connection, which in turn is used for your target.
How is an automation target added?
The target is added under Admin -> Automation. In the case of GitLab/Github for example, assuming you’ve set up a connection to the instance in our Integrations tab, then you can proceed to adding a target. It’s at this stage that you’ll need information specific to your workflow or pipeline to properly create the target (in the case of GitLab this would be your group/project name and reference).
In the GitLab target creation panel, what are the “project path” and the “reference”?
The project path is the GitLab group and name of the project/ID (Ex: group/myproject), while the reference is either a branch or tag name (Ex: main or master).
Is it possible to relaunch past jobs?
Yes, we have a Quick Launch function at the top right of the Jobs view in your project, where the latest targets used will be presented.
Why can I only delete one job at a time?
It’s definitely possible to delete more than one job. The way to accomplish this is by selecting multiple jobs, and then using the dropdown options for any of the jobs. You’ll be presented with the “Delete jobs” function!
I think something went wrong, I’m not seeing the run show up in Testmo.
If an issue occurs with one of your jobs, whenever possible Testmo will display the log for you to review the potential cause. Simply click the job itself or on “Details” to see what the log presents.
My job ran correctly, however I’m not seeing my run. What could be causing this?
Though the log can reflect issues with the connection, from the actuation of the job to the triggering, issues with the test or submission of the results to Testmo will not be displayed. To understand the potential causes you can review the tests themselves, and if the configuration of your result file submission process.
For any other questions or issues, please feel free to contact our support team at contact@testmo.com.
Last updated
