Will My Tests Tell Me if I Break This Code?
Title | Will My Tests Tell Me if I Break This Code? |
Publication Type | Conference Paper |
Year of Publication | 2016 |
Authors | Niedermayr, Rainer, Juergens, Elmar, Wagner, Stefan |
Conference Name | Proceedings of the International Workshop on Continuous Software Evolution and Delivery |
Date Published | May 2016 |
Publisher | ACM |
Conference Location | New York, NY, USA |
ISBN Number | 978-1-4503-4157-8 |
Keywords | code coverage, Measurement, Metrics, metrics testing, Mutation testing, pubcrawl, regression testing, test suite effectiveness |
Abstract | Automated tests play an important role in software evolution because they can rapidly detect faults introduced during changes. In practice, code-coverage metrics are often used as criteria to evaluate the effectiveness of test suites with focus on regression faults. However, code coverage only expresses which portion of a system has been executed by tests, but not how effective the tests actually are in detecting regression faults. Our goal was to evaluate the validity of code coverage as a measure for test effectiveness. To do so, we conducted an empirical study in which we applied an extreme mutation testing approach to analyze the tests of open-source projects written in Java. We assessed the ratio of pseudo-tested methods (those tested in a way such that faults would not be detected) to all covered methods and judged their impact on the software project. The results show that the ratio of pseudo-tested methods is acceptable for unit tests but not for system tests (that execute large portions of the whole system). Therefore, we conclude that the coverage metric is only a valid effectiveness indicator for unit tests. |
URL | https://dl.acm.org/doi/10.1145/2896941.2896944 |
DOI | 10.1145/2896941.2896944 |
Citation Key | niedermayr_will_2016 |