The following short paper was presented in 2021 at the 1^st International Workshop on the Body of Knowledge for Software Sustainability (BoKSS) that was co-hosted with the IEEE ICSE conference.

Abstract— Testing is a core software development activity that has huge potential to make software development more sustainable. In this paper, we discuss how environmental, social, economic, and technical sustainability map onto the activities of test planning, design, execution, and evaluation.

Keywords—test process, sustainability, software testing

Introduction

Software development involves complex and dynamic systems. Changing technologies and markets in industry require improvements in software testing. To ensure a sustainable positive impact of improvement measures in a software project, multiple perspectives, including cultural, economic, and social issues, have to be taken into account [1][2][3]. The sustainability definition of the Brundtland Commission [1], i.e., “meeting the needs of the present without compromising the ability of future generations to meet their own needs,” has environmental, social,  economic, and technical consequences that also impact how software testing should be conducted.

Experience, for instance in government projects, shows that software testing consumes resources and time comparable to the development tasks of the project [4]. An increase in complexity of requirements and architecture due to frequent adaptions in a project with a duration of 3-5 years is observed. Time pressure and cost restrictions prevent many projects from performing regenerative testing. The goal to perform software testing with “minimized environmental impact, a sufficient economic balance, and good working conditions” [2] is not taken into account.

Currently, there is little guidance on how to integrate “green” strategies in the test process and how to align them to testing activities. Penzenstadler and Femmer published a generic model for sustainability [3]. However, they do not explicitly address the sustainability aspects of software testing. Testing activities need time and resources. Testing tasks consume energy and resources because large-scale projects have a long development and maintenance period. Server-client systems, e.g., for regression testing and performance testing, consume a lot of energy.

Experiences from large-scale projects [4] indicate the following problems:

• Release and iteration planning neglects waste of resources and energy consumption.

• Lack of testability leads to increased complexity from version to version [9].
• Knowledge networking and technology management are not taken into account as a long-term investment in future business.

As a consequence, the objective of setting up a sustainable test process is not taken into account [5].

Dimensions of Sustainability in the Test Process

The sustainability dimensions consist of the following viewpoints: environmental, social, economic, technical, and individual sustainability (Fig. 1, generic model of Penzenstadler and Femmer, adapted).

The environmental dimension is a central “green” objective, for instance, energy efficiency in software engineering.

The social dimension aims at social capital as well as social communities and poses the question of how software impacts society in terms of social constructs and communication [3]. Further values of social sustainability include the health of employees, employment opportunities, the application of human rights at the workplace, and social networks within and outside an organization [1]. In the context of software testing, these principles apply to employees directly involved in the testing process, such as testers, test managers, and test engineers, as well as structures within an organization and society as a whole. Testing is also well-suited for onboarding new employees, and is therefore an activity that promotes  inclusiveness.

Economic sustainability aims at reducing the costs of test execution and the maintenance of test artifacts. Automated regression tests are a typical measure for improving the economic sustainability of a software project. The additional effort for automating the tests will pay off after a certain number of test cycles and result in considerable savings in terms of manual effort and resources [8].

Moreover, every change to the system under test requires an adaptation of the corresponding test artifacts. Thus, the effort required to maintain test cases, test data, and scripts also constitutes a major economic factor in a typical software project. It can be significantly reduced by applying a model-based test (MBT) [10] [11] approach. MBT relies on explicit models that encode information about the system under test.

Test cases, test data, and scripts can be automatically generated from the model by an MBT tool.

Technical sustainability is related, for instance, to long-living test environments for performance tests and test automation and their adequate evolution [9]. In this context, today’s fast-paced changes of the available technologies, tools, and interfaces can be a major challenge. For instance, a switch to a new user interface technology of the system under test can invalidate the existing test automation approach if the new technology is not supported by the test automation tool. Other examples include new network protocols, devices, operating systems, and so on.

A possible mitigation strategy is to focus on maintainability and expandability aspects during the design of the test framework. Also, testability aspects should be taken into account when deciding about changes to the system under test. In addition, technology management and a collaboration model leverage knowledge networking in an organization if experts are willing to help each other.

The green level of sustainability aims at test efficiency and effectiveness of testing in general, thereby reducing resource and energy consumption. The following sustainability capabilities support these goals:

• Plan testing: Professional requirements management, reduction of complexity, and good testability are preconditions to save resources and reduce waste when performing test activities. Release decisions should focus on quality.
• Design tests: Test cases based on functional and non-functional requirements are designed by testers with domain know-how and technical skills. The degree of experience of testers influences the quality of the test cases as well as the cost of maintenance and execution of test cases.
• Execute tests: Test case execution is the most expensive task. Every corrected release has to be retested and regression-tested automatically or manually. The degree of testability has a strong impact on resource and time consumption.

• Evaluate tests: Long-term maintenance and use of systems require a measurement system to assess the trade-off, e.g., between technical sustainability criteria and economic criteria. Decisions of a test manager should be supported by a dashboard.

To introduce it into a software project, two distinct phases are recommended: 1) analysis and 2) application and assessment. The analysis phase consists of instantiating the generic sustainability model, and the application & assessment phase consists of specifying responsibilities and monitoring sustainability using previously defined indicators and metrics.

Conclusion

Changing requirements, complexity, and a rigid time scale of projects are reasons why the long-term benefits of a sustainable test process are often not taken into account. We introduced dimensions and possible enhancements into the test process. However, concrete guidelines and measures are needed to promote environmental sustainability in software testing.

References

• [1] Al Hinai M., Chitchyan R.; (2016, August). Engineering requirements for social sustainability. In ICT for Sustainability 2016. Atlantis Press.
• [2] Penzenstadler B.; “Towards a definition of sustainability in and for software engineering”, SAC ’13, Coimbra, Portugal, 2011.
• [3] Penzenstadler B., Femmer H.; A generic model for sustainability with process- and product-specific instances; GIBSE ’13, Fukuoka, Japan, 2013.
• [4] Mohacsi S., Ramler R.; “Why Software Testing Fails: Common Pitfalls Observed in a Critical Smart Metering Project”, Software Quality Days, Vienna, 2019
• [5] Dao V., et al.; From green to sustainability an integrated sustainability framework, JSS 20 (2011), pp. 63-79., 2011.
• [6] Lago P., Kocak S., Cernkovic I., Penzenstadler B.; Framing sustainability as a property of software quality; Comm. ACM, Vol.58, NO.10, 2015.
• [7] Calero C., Piattini M.; Green in Software Engineering, Springer, ISBN 978-3-319-08580-7, 2015.
• [8] Felderer M., Ho J., Beer A., Ruhe G.; Industrial evaluation of the impact of quality driven release planning. In: 2014 ACM/IEEE Int. Symposium on Empirical Software Engineering and Measurement (ACM), 2014
• [9] Felderer M., Beer A.; Estimating the return of investment of defect taxonomy supported system testing in industrial projects. In: 38th EUROMICRO Conference on Software Engineering an advanced Applications (SEAA), IEEE(2012) 426-430.
• [10] Beer A., Mohacsi S.; Efficient Test Data Generation for Variables with Complex Dependencies. In: IEEE ICST in Lillehamer (2008).