Software testing best practices

Often under-estimated and not much considered by developers, testing is however one of the key elements of software engineering, and a fundamental pillar without which no software can be put on the market with appropriate quality. The goal of this article is to evangelize the major dimensions of software testing and share our experience of best practices as we perceive them at SoftFluent.

An often undersized investment in quality

Software testing is a broad area that allows the control of software quality. This quality is made of several dimensions that can be grouped within the three following categories: features, engineering and adjustability.

Still investment in testing is often undersized from software vendors or IT teams. Similar to what happens with security, the feeling of spending non-productive time is stronger than the power of best practices that recommend forecasting at least one tester for two developers. Field reality can easily be two times less.

In France, this phenomenon is probably aggravated by the important weight of service companies and custom developments. Their relative importance is stronger than anywhere else, partly for cultural reasons, and consequently, the culture of packaged software is weaker. Service companies have a lower sensitivity than software vendors to quality, because the impact of a correction is often limited. On the contrary, a software vendor must invest upfront on product quality; otherwise the deployment of its product can quickly become very costly.

At companies such as Microsoft, whose software products are deployed by tenths of millions, some development teams can have up to two testers for one developer!

The different test levels

First, let us note that software testing is accomplished at various levels. Potentially, one can distinguish four levels:

• Unit tests are accomplished by the developer to check the good execution of functions that he is responsible for. These functions are often tested independently, with limited set of data. It is a good practice to early define representative test samples, as the developer will be able to develop unit tests based on these samples which will improve quality.
• Integration tests are usually accomplished by a dedicated tester, within the development team or inside a quality team which works closely with the development team. The particularity of these tests is that they target the testing of a full set of features that often relies on leveraging several components a full process beyond unitary functions. One usually talks about “white box” testing, as it is possible to have a look at relationship between components to check their efficient cooperation. Having test data early in the cycle is also an important element of efficiency.
• System tests or qualification consist in playing complete scenarios, representing software use cases, without considering implementation or underneath components. These tests follow a “black box” logic and It is highly recommended that the team realizing these tests be distinct from development team. This team can even be placed “off-shore” with the need of requiring detailed formalism to fully describe usage scenarios. During system evolution, these scenarios will be replayed to validate non-regression.
• Acceptance tests are lead with pilot users to validate software adequacy to the business and easy adoption by those who will use it regularly. Once again, involving users early in the cycle to use part of the software will be a good manner to get aware of user acceptance criteria, and, if necessary, adjust some elements such as ergonomics.

Our belief in unit, integration and system testing is that it is always a good idea to automate these tests. A manual test is an expense. A test which is consolidated in a replayable script is an investment. Automating unit testing and some integration tests allow extraction of some of these tests to be replayed directly in the automated build of the system.

Additionally, test artifacts such as unit test code, configuration files or data set are part of the software documentation necessary for its maintenance and serve as examples for using software public interfaces (APIs). This is particularly important for systems that will expose features as services, or get integrated with other systems.

Different test natures

It is also worth noting the different natures of testing:

• Functional tests validate the right behavior of the software as per the service delivered. These tests are often described in detail in scenarios in which most of the investment is made, consistently with the effort made in specifications formalizing the feature. Our recommendation on this topic is consistent with points mentioned above. Writing scenarios early is a very good practice as it validates the functional analysis, while giving material to the developer for its own unit tests. But putting the exclusivity of the effort on these tests is a common mistake.
• Platform compatibility
  tests include as a minimum setup tests on targeted environments (operating system, database…) or even more advanced tests for certifying a specific platform, especially for ISVs. These tests allow checking the right application of best practices such as the use of virtual directories, execution without administrator rights, uninstall procedures, resource localization, etc. Developer desktops should not be confused with the target execution environment.
• Robustness tests are targeted to testing the software behavior in suboptimal conditions. These tests are scarce systematical, although they are not so complex to implement: unplugging a cable, stopping a machine… It is important to know how the software will behave within these circumstances and define a recovery scenario. Not testing these cases saves a few hours in the short term but preserves a risk whose financial consequences are often impossible to evaluate. It is quite obvious that the user will sooner or later be exposed to a network breakdown, so it is better to anticipate the software behavior and document it.
• Performance tests are often delivered at the end of the project. Factors influencing performance are numerous and it is risky to conclude early on partial tests without having representative volume of data and exact execution conditions. However, developers can usually test behavior of some unitary services quite early in the cycle when these elements bear a performance risk. Moreover, experience demonstrates that the result of performance tests must be precisely documented to formalize measuring conditions. Conclusions must be made based on relevant performance counters such as CPU consumption, memory usage, requests per second, Input/Output flow…
• Scalability tests should be distinguished from performance tests. An application can perform well but be limited in its ability to sustain an increasing number of users (or any other parameter). A software system, especially if it is supposed to be used through Internet, should be designed to scale up almost without limit by sampling adding machines. Tests must be made for this, with tools allowing simulating numerous simultaneous users.
• Usability tests are quite never lead as they require involvement of final users. However, they are a significant factor for real and perceived quality, improving comfort of software usage. These tests include both “Look” (visual appearance) and “Feel” (ease of use).

Favor early detection of bugs

The reader will have understood, through the different comments made in previous paragraphs, that at SoftFluent we are closer to agile approaches than pure waterfall methodology, and we favor everything that allows short cycles, for testing as well as for functional specifications by the way. The reason is above all financial.

The cost of a bug is very different depending on the stage at which it is introduced and at which it is detected. As an example, if a design error, detected in the specification stage cost 1 to correct, the same error detected at qualification stage will cost 10, and potentially up to 100 if the software is already deployed. The following table estimates ratios for different kind of bugs depending on these stages.

This is why we try to validate most elements upfront, before having a significant development effort, costly by nature. Furthermore, bugs are easier to understand and correct on a limited perimeter than on a complete application which obviously bear some complexity.

Evolution towards agile methodology

The positioning of tests across time is quite different using a traditional V Cycle methodology or using an agile approach:

From a human prospective, agile approaches require a permanent collaboration between developers and functional experts, collaboration which is very often the secret of successful software. In traditional methods, an advanced formalism is often a way to be theoretically more confident, but it is also a way for developers to protect themselves behind written documents to avoid any modifications, even if these are necessary to fulfill business needs. The phenomenon also happens with testers, although the tester should be the developer’s best friend, as it helps him find his bugs!

As a conclusion, let us remember that software testing is an important job, that comprises multiple aspects, but is not enough recognized and valued, although it plays a critical role in software quality. Everyone being able to evaluate its daily experience with computing should easily admit that progress is still to be made in this area! So let the developers and IT people give to software testing the place it deserves.

10 pieces of advice

1. Test is a real job, having test roles distinct from developers is an absolute necessity
2. Test should not be underestimated, about 25 to 30% of a development project
3. Test data should be produced early in the cycle to detect design errors and facilitate the work of developers
4. An application should always be designed to be testable, and a good split in components is a very important factor
5. One should invest in all dimensions of testing, including acceptance and usability tests
6. It is necessary to define operating conditions for the system to test configurations, scalability or robustness in suboptimal conditions
7. It is always a good idea to invest in automating the tests, especially for non-regression
8. Tests are developments that should be versioned
9. As for development, a partial work which is well targeted will cover most of the important elements following an 80/20 rule. It is better to start this way and improve through an iterative model than being utopist and believe in total completeness of tests

A model-driven strongly typed static approach facilitates testing, whereas a fully dynamic approach based on metadata, although very flexible for the developer, will postpone the discovery of key issues to very late moment in the cycle.

Daniel COHEN-ZARDI