DevOps and Software Testing

What is DevOps?

DevOps – a combination of Development & Operations – is a software development methodology which looks to integrate all the software development functions from development to operations within the same cycle.

This calls for higher level of coordination within the various stakeholders in the software development process (namely Development, QA & Operations).

Why DevOps?

Although there are subtle differences between Agile & DevOps, those working with Agile will find DevOps a little more familiar to work with (and eventually adopt). While Agile principles are applied successfully on the development & QA iterations, it is a different story altogether (and often a bone of contention) on the operations side. DevOps proposes to rectify this gap.

Now, instead of Continuous Integration, DevOps involves “Continuous Development”, whereby, code written and committed to Version Control, will be built, deployed, tested and installed on the Production environment ready to be consumed by the end-user.

This process helps everyone in the entire chain since environments and processes are standardized. Every action in the chain is automated. It also gives freedom to all stakeholders to concentrate their efforts on designing and coding a high quality deliverable rather than worrying about the various building, operations and QA processes.

It brings down the time-to-live drastically to about 3-4 hours, from the time code is written and committed, to deployment on production for end-user consumption.

In a nutshell, DevOps is an extension of Agile, or, as I like to call it, “Agile on Steroids”.

Changed Role of QA in DevOps

Traditionally, QA would get a build which is deployed on their designated environment and QA would then commence their functional & regression testing. The build would ideally sit with the QA for a couple of days before the QA sign-off on the build. All this changes in DevOps.

QA changes for DevOps:

QA are required to align their efforts in the DevOps cycle.
They have to make sure that all their test cases are automated and achieve near 100% code coverage.
They need to make sure that their environments are standardized and deployment on their QA boxes is automated.
All their pre-testing tasks, cleanups, post-testing tasks, etc. are automated and aligned with the Continuous Integration cycle.
As already mentioned, DevOps requires high level of coordination between various functions of the deliverable chain. What this also means is that boundaries between the various roles of contributors in the chain become porous.

DevOps encourages everyone to contribute across the chain. So, amongst other things, a dev can configure deployments. Deployment engineers can add test cases to the QA repository. QA Engineers can configure their automation test cases into the DevOps chain.

Collectively, everyone in the chain is responsible for the quality and timeliness of the deliverables.

DevOps and Test Automation

To achieve such speed and agility, it is important to automate all the testing processes and configure them to run automatically when the deployment is completed on the QA environment. Specialized automation testing tools and continuous integration tools are used to achieve this integration. This also necessitates building of a mature automation testing framework through which one can quickly script new test cases.

Tips for DevOps Success:

The test cases that are required to be executed for a particular build need to be identified.
The test execution should essentially be lean.
The QA and dev need to sit together and identify the areas affected due to a particular build and execute those related test cases plus a sanity test pass.
You also need to configure specialized code analysis and coverage tools to make sure you achieve near 100% code coverage.
The concept of executing all regression test cases for a test pass is soon becoming obsolete.
Strategy around testing new features need to be formalized and interim builds can be supplied to QA who would in turn create test scripts and run these automation tests on these interim builds till the code becomes stable enough to be deployed on to Production environment.
All the environments required for testing need to be standardized and deployments have to be automated.
Using various automation techniques, QA should be able to fire automation testing runs across various cross-platform (and cross-browser in case of web applications) environments.
Parallel execution of tests helps in reducing time-to-live which is the crux of a successful DevOps implementation.
Exit criteria need to be set for each run so that when the results of the tests are fed back to the chain, a go/no-go decision to Production is taken.
Blocker or Critical bugs found need to be reported and fixed and passed through the same chain of events before the code is deployed on to the Production environment.
Application Monitoring

QA should also be able to detect problems early and report them proactively. To achieve this, they need to set up monitoring on the Production environment to be able to expose bugs before they cause a failure.

Setting up specialized counters like response times, memory & CPU utilization, etc. can provide a lot of insight into the end-user experience.

For example, if the average response time for login is gradually increasing over the various builds, QA should proactively report this issue for optimizing the login code, else future builds might cause end-user frustration due to high response times.

QA can also use a small subset of existing high priority test cases to be executed periodically on production to actively monitor the environment. Bugs like, “This bug appears sometimes” or “Cannot Reproduce” can be caught through this strategy which, in the end makes the application more stable and also gets more satisfied end-users.

Again, these monitors need to be configured to run automatically with rich reporting (like logs & screenshots of failures, etc.).

Conclusion

Waterfall gave way to V-Model which in turn was replaced by Agile as the preferred choice for software development. DevOps is the future. It’s a continuous improvement cycle that software development models undergo from time-to-time. You need to embrace, understand and inculcate it.

You need to master the various automation and continuous integration tools, so that your automation efforts add value to the chain and are lean enough to quickly adapt to changes. You may be working on projects that may involve an alpha, beta and UAT environments before being deployed on the production environment.

New Technique of Covering Test cases Confidentially.

New Technique of Covering Test cases Confidentially.

Let’s take an Decision Table example of below business requirement for a login screen

The first step we do is to name all the branches and leaves with numbers or alphabets as below.

1, 2, 3 are the leaves and a, b, c are the branches.

Test Auto Mation

What makes automated testing so important to these successful companies?

Automated Software Testing Saves Time and Money

Software tests have to be repeated often during development cycles to ensure quality. Every time source code is modified software tests should be repeated. For each release of the software it may be tested on all supported operating systems and hardware configurations. Manually repeating these tests is costly and time consuming. Once created, automated tests can be run over and over again at no additional cost and they are much faster than manual tests. Automated software testing can reduce the time to run repetitive tests from days to hours. A time savings that translates directly into cost savings.

Testing Improves Accuracy

Even the most conscientious tester will make mistakes during monotonous manual testing. Automated tests perform the same steps precisely every time they are executed and never forget to record detailed results.

Increase Test Coverage

Automated software testing can increase the depth and scope of tests to help improve software quality. Lengthy tests that are often avoided during manual testing can be run unattended. They can even be run on multiple computers with different configurations. Automated software testing can look inside an application and see memory contents, data tables, file contents, and internal program states to determine if the product is behaving as expected. Automated software tests can easily execute thousands of different complex test cases during every test run providing coverage that is impossible with manual tests. Testers freed from repetitive manual tests have more time to create new automated software tests and deal with complex features.

Automation Does What Manual Testing Cannot

Even the largest software departments cannot perform a controlled web application test with thousands of users. Automated testing can simulate tens, hundreds or thousands of virtual users interacting with network or web software and applications.

Automated QA Testing Helps Developers and Testers

Shared automated tests can be used by developers to catch problems quickly before sending to QA. Tests can run automatically whenever source code changes are checked in and notify the team or the developer if they fail. Features like these save developers time and increase their confidence.

Team Morale Improves

This is hard to measure but we’ve experienced it first hand, automated software testing can improve team morale. Automating repetitive tasks with automated software testing gives your team time to spend on more challenging and rewarding projects. Team members improve their skill sets and confidence and, in turn, pass those gains on to their organization.

TestComplete is a Powerful and Affordable Automated Software Testing Tool

TestComplete addresses a full range of software testing challenges facing corporate IT departments, product developers, QA engineers, and consultants. TestComplete enhances the software testing process by increasing efficiency, removing complexity and lowering costs

- See more at: http://support.smartbear.com/articles/testcomplete/manager-overview/#sthash.V0w8mfc2.dpuf

QA TIPS

1. Focus on Risk/Change Based Testing – In an Agile environment, the goal is always to get the highest quality product to market in the shortest amount of time. Therefore an Agile tester must be able to assess which areas of the application are changing, how they all fit together and prioritize tests based on areas that pose the greatest risk of failure.

2. Understanding Product Architecture – Put simply, this means understanding exactly how data flows through an application. This allows us to test the impact of sub-system component failures, as well as address potential security vulnerabilities. As a result, when defects are found an Agile tester will be able to help developers fix the issues quickly and thoroughly early on.

3. Understanding Business Objects and Context – Since testing should always align with customer context, an Agile tester must understand how the end user will interact with the product. If you focus on the applications and scenarios that provide value to the end user you will be able to dimensionalize your testing strategy. Meaning, if you can divide a product up based on the product architecture and customer context you will be better equipped to test for potential issues that would negatively impact end users.

The added benefit of dimensionalizing the product from the customer view point is that the tester will be able to tag defects with the affected business objects. That way when you begin to review your test sets, you will be able to recognize areas with higher instances of defects. This will allow you to address the issues in a more focused and effective way.

4. Application Logs - While reporting ‘when feature X breaks we do Y’ provides value, it is not enough in an Agile environment. Agile testers should be leveraging the great amount of information contained in application logs for a number of reasons. First, it provides insight into the underlying architecture that is occurring on the system level, which gives you a better understanding of what the defect is. Second, application logs also allow you to address ‘silent errors’ – i.e. errors that occur without the end user ever even knowing. Lastly, it allows testers to gain credibility and work more closely with the development team, which is an integral component of being Agile.

5. Based Tools – In addition to application logs, browser based tools are incredibly important for Agile testers to be able to quickly troubleshoot defects. Two examples of easy-to-use browser tools are Developer Tools in Google Chrome and Firebug in Firefox. Tools such as these provide value in the form of metrics, errors, analysis, java script consoles and debuggers.

6. Test Document Requirements – Many Agile organizations utilize some variant of SCRUM or Kanban, in which there is an element of work tackled by a team to ensure that all aspects function as intended. In order to do this, the team will need to create test scenarios. The expectance criterion of these test scenarios often becomes a requirement repository over the long term. While some people view the written test scenario as a waste, Brian Rock contends that these test scenarios become very valuable to the Agile tester for future automation, regression testing and product analysis.

7. Automation – Automation is an excellent tool but it goes without saying that it is not the ‘silver bullet’ to fix the testing problem. Frameworks can be expensive to create and even more expensive to maintain. However, Agile testers need to be able to leverage automation for quick, easily repeatable tasks. If the delta of change is low, chances are you should automate it.

8. Exploratory Testing – Exploratory testing (ET) is common sense testing. ET focuses on instantaneous testing and learning and thus should be a primary tool of all Agile testing. Agile testers can leverage knowledge gained through ET on every future product iteration. Additionally, it allows the Agile tester to make variations in testing to quickly ferret out bugs.

9. Testing From the Customer Perspective – In many ways, this goes back to understanding the business objects and how testing should focus the end user – despite the fact that the end user is going to use the product in ways that you never thought feasible, wise or even sane. Testing from the customer perspective goes a step further to state that the fitness of the product for use by the end user is the primary standard of quality. After all, without customers there is no product. Agile testers are in a unique position to face QA challenges from this context.

10. Know That Change is Constant – Perhaps this is a bit cliché, but a static product is a dead product. As an Agile tester you should be able to give a quality assessment of the product at any stage in its life cycle. If you are able to do this, you will be better equipped to handle the ever changing landscape that is Agile.