Can software/features be released every day? Every hour? Nowadays, one of the biggest challenges most companies face is getting their software to market quickly without renouncing quality. So how can we ship high-quality software even more rapidly?
We should be able to run automated tests to triage and fix defects. Testers should perform exploratory testing constantly against the latest builds to come out of CI. No one should say they are “done” with any work until all business value automated tests have been written and passed.
It could sound easy to tell, but in practice, Continuous Testing & Continuous Delivery can be a challenging goal for many organizations; in terms of software testing, this brings us to an inflection point. QAs were already racing to keep pace when delivery cycles were longer and application complexity was lower.
Fast Software Release Continous Testing/Continous Delivery
Continuous delivery is about quickly and sustainably delivering changes like new features, bug fixes, and configuration updates quickly and sustainably. The main goal is to have the ability to perform any deployment on-demand, from apps to large-scale distributed systems, but in a routine and predictable way. Delivering software and features more frequently doesn’t mean users should get a lower-quality product.
Continuous testing boils down to providing feedback to stakeholders at the right time. Before the age of Agile or even DevOps, testing was traditionally deferred until the end of the cycle. At that point, testers would provide all sorts of meaningful feedback, but only wanted to hear it once it was more costly to fix. Continuous testing focuses on providing actionable feedback to people who care about it at the point when they are genuinely prepared to act on it and avoid delays in our lead time to change metrics.
To release software fast and efficiently as possible. Continuous testing helps us achieve that; it also assists agile teams in identifying and fixing issues as efficiently as possible (accelerating innovation). It is achieved by mastering and going further with test automation. It needs aligning testing with business risks, ensuring that testing effectively assesses the end-user experience, providing the instant quality feedback required at different stages of the delivery pipeline, and accelerating our software releases.
We HAVE a problem, perhaps some challenges.
We are not quite there yet to constantly deliver changes/features to production. In most organizations, testing delays continuous delivery while providing limited insight into whether the applications under test are meeting stakeholder expectations. It's not fast enough to help teams find and fix defects when it's optimal. It's reporting on low-level test failures (e.g., 72% of our tests passed) rather than providing the business-focused perspective needed to make fast-release decisions (e.g., Only 28% of our business risk was tested, and 18% of that didn't work correctly).
Let's discuss why testing is a bottleneck and what we see in multiple companies.
- Most testing is “functional testing,” even more at large enterprise organizations
- Test Automation scripts built, maintained, and executed are redundant and add no business value.
- Testers spend their time dealing with false positives and additional test maintenance tasks in organizations with significant test automation
- Testers are routinely delayed by limited test environment access or other environment issues.
- The average regression test suite could take several days to execute, but the Agile sprint is two weeks from start to finish, including planning, implementation, and testing.
- The average application under test now interacts with multiple dependent systems. A single end-to-end transaction could cross everything from microservices and APIs to various mobile and browser interfaces, packaged apps, custom/legacy applications, and mainframes.
The software testing process wasn't working flawlessly even before the Agile Testing and DevOps. So, now we're asking testing teams to "speed it up" while modern application architectures are making testing even more complex.
Right Test Automation for Fast Feedback Challenge
Almost all organizations already understand that test automation is essential for modern application delivery processes. They're just unsure how to make it a reality on an enterprise scale. If we want to enable Continuous Testing, automation rates need to exceed 80%. The only remaining tests should be exploratory, and the type of automation should also shift. Automation should focus predominantly on API testing.
It sounds achievable, but in reality, test automation can be hard to implement for some organizations related to the complexity of their current process or perhaps time and resource availability. Sometimes the teams start excellently, but after trying to scale up, those teams can quickly lose the real goal, which is speeding up our testing.
What we can do better, of course, we should create better test scripts; in today's automation world, UI testing accounts for the vast majority of functional test automation, with only a tiny fraction of testing being conducted at the API level. Why API? Since APIs are considered the most stable interface to the system under test, API tests are less brittle and easier to maintain than UI tests.
Risk-based Testing Challenge
Many organizations think that this risk-based approach to testing sounds excellent in theory but doubt that they can achieve it, particularly given the limited downtime available with today's unrelenting, fast-paced Agile iterations. At its most basic level, risk quantifies the potential of losing something of value, and "business risk" quantifies the potential of negatively impacting the business.
When we start a new sprint, we have a list of user stories for that sprint already at risk, as we need to test them separately and then include them as part of our regression (Automated Regression). Why? Because new functionality is more likely to fail than functionality, you have already checked and verified. Therefore, all new functionality introduced in a given sprint will be riskier than your existing functionality (In-sprint testing vs. regression testing).
What we can do better is allocate limited in-sprint testing time among the various new user stories. Of course, all the latest user stories will likely have issues, but you want to focus on testing the ones with the most significant business risk potential.
The Continuous Testing Challenge
Providing the team with fast, quality feedback is one of the top goals of test automation. Most believe it's not the tester's job to provide all the data and insight. However, as CD evolves, it's essential to implement continuous testing and give your team feedback loops to fix software issues quickly.
A reliable automation testing team keeps a product's engine running but requires firm discipline throughout the Agile strategy. In addition, it requires getting everyone in agreement. Therefore, DevOps leaders need to ensure all groups agree on the process they choose; most importantly, leave shadow CI processes in the past to ensure teams have access to a single view of your product's code.
If you've ever struggled to reach a destination without a good plan, route, map, or trail, you know how frustrating a "trial-and-error" approach can be. Fortunately, you don't need to take that approach on your Continuous Testing journey. Instead, you can benefit from the lessons learned by others who have already taken that journey and adapt to those practices beneficial to your current process or organization.
Wrapping Up
QA activities are not something we should only start once a feature or a release is “dev complete.” Testing is essential; we should always do it as part of the Agile process. Automated unit and acceptance tests should be run against every commit to version control to provide developers with fast feedback on their changes.
"A QA Team can focus on only delivering high-quality features, but the only way to know if this is true for our customers is with the right metrics." -- Enrique A Decoss.
The best way to expand an initiative is to demonstrate the quantifiable gains achieved at each step and set realistic targets. For example, suppose we want to adopt Continuous Testing. In that case, we must ensure some KPIs we want to use to quantify and show progress in accelerated innovation, reduced business risks, and improved cost efficiency.
Enrique DeCoss
Senior Quality Assurance Manager | FICO
Enrique is an industry leader in quality strategy with 17+ of experience implementing automation tools. Enrique has a strong background in web Tools, API testing strategies, performance testing, Linux OS and testing techniques. Enrique loves to share his in-depth knowledge in competencies including, Selenium, JavaScript, Python, ML testing tools, cloud computing, agile methodologies, and people Management.
Related Posts
Top 10 Testing Communities to Join in 2023-2024
Top 10 Testing Communities to Join in 2023-2024
API Testing – What, Why, and How?
