DevOps and Software Testing

What is DevOps?

DevOps – a combination of Development & Operations – is a software development methodology which looks to integrate all the software development functions from development to operations within the same cycle.

This calls for higher level of coordination within the various stakeholders in the software development process (namely Development, QA & Operations).

Why DevOps?

Although there are subtle differences between Agile & DevOps, those working with Agile will find DevOps a little more familiar to work with (and eventually adopt). While Agile principles are applied successfully on the development & QA iterations, it is a different story altogether (and often a bone of contention) on the operations side. DevOps proposes to rectify this gap.

Now, instead of Continuous Integration, DevOps involves “Continuous Development”, whereby, code written and committed to Version Control, will be built, deployed, tested and installed on the Production environment ready to be consumed by the end-user.

This process helps everyone in the entire chain since environments and processes are standardized. Every action in the chain is automated. It also gives freedom to all stakeholders to concentrate their efforts on designing and coding a high quality deliverable rather than worrying about the various building, operations and QA processes.

It brings down the time-to-live drastically to about 3-4 hours, from the time code is written and committed, to deployment on production for end-user consumption.

In a nutshell, DevOps is an extension of Agile, or, as I like to call it, “Agile on Steroids”.

Changed Role of QA in DevOps

Traditionally, QA would get a build which is deployed on their designated environment and QA would then commence their functional & regression testing. The build would ideally sit with the QA for a couple of days before the QA sign-off on the build. All this changes in DevOps.

QA changes for DevOps:

QA are required to align their efforts in the DevOps cycle.
They have to make sure that all their test cases are automated and achieve near 100% code coverage.
They need to make sure that their environments are standardized and deployment on their QA boxes is automated.
All their pre-testing tasks, cleanups, post-testing tasks, etc. are automated and aligned with the Continuous Integration cycle.
As already mentioned, DevOps requires high level of coordination between various functions of the deliverable chain. What this also means is that boundaries between the various roles of contributors in the chain become porous.

DevOps encourages everyone to contribute across the chain. So, amongst other things, a dev can configure deployments. Deployment engineers can add test cases to the QA repository. QA Engineers can configure their automation test cases into the DevOps chain.

Collectively, everyone in the chain is responsible for the quality and timeliness of the deliverables.

DevOps and Test Automation

To achieve such speed and agility, it is important to automate all the testing processes and configure them to run automatically when the deployment is completed on the QA environment. Specialized automation testing tools and continuous integration tools are used to achieve this integration. This also necessitates building of a mature automation testing framework through which one can quickly script new test cases.

Tips for DevOps Success:

The test cases that are required to be executed for a particular build need to be identified.
The test execution should essentially be lean.
The QA and dev need to sit together and identify the areas affected due to a particular build and execute those related test cases plus a sanity test pass.
You also need to configure specialized code analysis and coverage tools to make sure you achieve near 100% code coverage.
The concept of executing all regression test cases for a test pass is soon becoming obsolete.
Strategy around testing new features need to be formalized and interim builds can be supplied to QA who would in turn create test scripts and run these automation tests on these interim builds till the code becomes stable enough to be deployed on to Production environment.
All the environments required for testing need to be standardized and deployments have to be automated.
Using various automation techniques, QA should be able to fire automation testing runs across various cross-platform (and cross-browser in case of web applications) environments.
Parallel execution of tests helps in reducing time-to-live which is the crux of a successful DevOps implementation.
Exit criteria need to be set for each run so that when the results of the tests are fed back to the chain, a go/no-go decision to Production is taken.
Blocker or Critical bugs found need to be reported and fixed and passed through the same chain of events before the code is deployed on to the Production environment.
Application Monitoring

QA should also be able to detect problems early and report them proactively. To achieve this, they need to set up monitoring on the Production environment to be able to expose bugs before they cause a failure.

Setting up specialized counters like response times, memory & CPU utilization, etc. can provide a lot of insight into the end-user experience.

For example, if the average response time for login is gradually increasing over the various builds, QA should proactively report this issue for optimizing the login code, else future builds might cause end-user frustration due to high response times.

QA can also use a small subset of existing high priority test cases to be executed periodically on production to actively monitor the environment. Bugs like, “This bug appears sometimes” or “Cannot Reproduce” can be caught through this strategy which, in the end makes the application more stable and also gets more satisfied end-users.

Again, these monitors need to be configured to run automatically with rich reporting (like logs & screenshots of failures, etc.).

Conclusion

Waterfall gave way to V-Model which in turn was replaced by Agile as the preferred choice for software development. DevOps is the future. It’s a continuous improvement cycle that software development models undergo from time-to-time. You need to embrace, understand and inculcate it.

You need to master the various automation and continuous integration tools, so that your automation efforts add value to the chain and are lean enough to quickly adapt to changes. You may be working on projects that may involve an alpha, beta and UAT environments before being deployed on the production environment.