Updates & Stability
Last updated
Last updated
Many teams are migrating and switching from other tools to Testmo. A question we sometimes receive from new customers is about Testmo product and platform updates and our processes & practices to ensure the stability of Testmo. We understand that our customers are using Testmo every day and that it is an important tool in their workflow.
On this page we want to share our processes related to product and platform updates so you have all the details needed to make an informed decision about Testmo.
Interested in Testmo's security & infrastructure details?
We share detailed information about our security program and cloud infrastructure on our website. If you are also interested in these topics, please see the related sections:
We have an extensive test strategy for Testmo and all major features we build have extensive automated testing in place. This not only ensures that we release well-tested code for new features. It also avoids breaking any existing code or features when making changes to Testmo.
Backend & API tests: We have many thousands of automated tests verifying our backend code and API logic. This ensures that our data model is consistent and well-tested and it also verifies the interface between our client and server code. Our backend code is developed with the relevant tests at the same time so our tests are an integral part of our development workflow and are always up-to-date.
Frontend end-to-end tests: Additionally, we have a completely separate additional fully-featured test suite for frontend end-to-end tests with Selenium. This tests all aspects of the application using Selenium browser automation. This not only verifies many aspects of the application logic, data model and network communication again. It also tests the user interface and ensures that we find regressions in the interface, user interaction and frontend rendering.
CI pipeline integration: For every release and update we build and deploy, our entire test suites (backend and frontend) are run as part of our CI pipeline. This ensures that no update can be deployed without passing all tests and it also ensures that our complete tests are run regularly and are well-maintained.
Our Testmo cloud infrastructure features tenant isolation by using a separate database for each Testmo instance. This means that each customer has its own database, so changes, migrations and updates only affect specific databases and tenants. This not only has various scalability, performance, optimization and security benefits. It also allows us to selectively update individual instances and apply new product versions to a subset of instances. We use this capability to implement staged deployments (see below).
Because we can selectively update individual instances, we do not need to use error-prone feature flags to gradually introduce new features and product updates. Instead, we can migrate individual instances and databases as needed. Many SaaS providers implement multi-tenant environments with shared databases and tenant IDs & feature flags. With our improved approach, we can more easily & safely apply new versions, backup individual customer databases and carefully rollout new versions.
We use a staging and rolling deployment model to release new features and product updates of Testmo. This means that we do not release new versions to all our systems and for all our customers at once. Instead, we have the ability to use multiple environments and various stages to carefully deploy product updates so we can detect issues as early as possible and either stop deployments or replace updates with improved versions before completing a rollout.
This model also allows us to adjust the rollout for each update. We assess the risk of individual updates to decide how slowly and through which stages we deploy a new version. For example, a simple UI fix (which still automatically goes through our rigorous test suites) can be deployed faster compared to a feature update that involves database migrations. We can therefore deploy smaller fixes safely & continuously, while also use more stages and failsafes for updates that affect larger areas of our code.
We use channels to deploy new versions to different Testmo instances in production. Each Testmo instance is subscribed to a channel, which allows us to easily select groups of instances for each stage.
With our staged deployment process there's rarely a need to rollback a version. We either notice issues during rollout before they reach production customer instances. Or we can easily replace an update with a fix before they affect customers. Should there ever be the need to rollback a certain update to give us more time to investigate a problem, we have the ability to re-deploy an older version (and if necessary develop a migration to rollback database changes).
Regardless of the number of tests an application has and how good the test suites are, it can always be possible that issues make it to production (e.g. due to subtle but important differences in environments or because of unexpected user configurations such as browser plugins). That's why an error reporting & monitoring strategy is important to catch issues as quickly as possible (this is especially useful if paired with a staged deployment strategy, see above).
In addition to extensive logging, we have also implemented rich error reporting & monitoring for all our important applications. We use the popular Sentry service, so we are notified immediately if there are any backend, frontend or infrastructure problems with Testmo. Not only that, we've extended the default implementation to include more details in error reports so we can more easily reproduce and resolve problems. With our error reporting strategy we often notice and resolve issues even before users experience any problems.
Decisions made in product management can have deep and lasting effects on the quality of software applications. We recognize this and use various strategies to ensure that product management is contributing to stable, robust & scalable code.
Manageable updates: Our goal is to release smaller well-tested features compared to huge multi-months development efforts. Deploying new code in smaller updates reduces the risk of migrations, UI changes and feature releases significantly. We understand that our users depend on Testmo every day. So we are careful about what changes we make and how we introduce new features over time.
Technical debt: We specifically strive to avoid technical debt when we build new code for Testmo. A lot of quality issues, scalability problems and performance issues come from features and code that are rushed out too fast. Therefore when we release new features and code, we ensure that they are well-built, well-tested and complete.
Feature selection: This is a critical but often ignored aspect of product management. It's easy to come up with feature ideas that are great on paper and sound useful, but are at the same time disproportionately risky to implement from a technical and development standpoint. Sometimes it's just better to pass on an idea. That's why our development and ops team have a say in product management as well. This way we also keep the maintainability, stability & scalability of Testmo in mind when we make feature decisions.
Our development and ops team has many years of experience running and maintaining large SaaS implementations without issues.
Testmo is completely built by an experienced in-house development team. This means that every line of code specifically written for Testmo is directly coming from us. This ensures that all product and development knowledge is kept with our development team, which reduces quality issues and ensures that new features can be designed and built in robust and scalable ways.
We also try to keep dependencies on third-party code to a minimum. When it makes sense to choose third-party code, we prefer well-maintained, well-supported and best-in-class libraries such as CKEditor or HighCharts.
Our application infrastructure is also maintained and monitored by our own in-house team (our infrastructure is built with Amazon AWS). This allows us to quickly react to and resolve issues without delays.
