Biblio

In many domains, engineers build simulation models (e.g., Simulink) before developing code to simulate the behavior of complex systems (e.g., Cyber-Physical Systems). Those models are commonly heavy to simulate which makes it difficult to execute the entire test suite. Furthermore, it is often difficult to measure white-box coverage of test cases when employing such models. In addition, the historical data related to failures might not be available. This paper proposes a cost-effective approach for test case selection that relies on black-box data related to inputs and outputs of the system. The approach defines in total five effectiveness measures and one cost measure followed by deriving in total 15 objective combinations and integrating them within Non-Dominated Sorting Genetic Algorithm-II (NSGA-II). We empirically evaluated our approach with all these 15 combinations using four case studies by employing mutation testing to assess the fault revealing capability. The results demonstrated that our approach managed to improve Random Search by 26% on average in terms of the Hypervolume quality indicator.

Cyber-Physical Systems (CPSs) are often tested at different test levels following "X-in-the-Loop" configurations: Model-, Software- and Hardware-in-the-loop (MiL, SiL and HiL). While MiL and SiL test levels aim at testing functional requirements at the system level, the HiL test level tests functional as well as non-functional requirements by performing a real-time simulation. As testing CPS product line configurations is costly due to the fact that there are many variants to test, test cases are long, the physical layer has to be simulated and co-simulation is often necessary. It is therefore extremely important to select the appropriate test cases that cover the objectives of each level in an allowable amount of time. We propose an efficient test case selection approach adapted to the "X-in-the-Loop" test levels. Search algorithms are employed to reduce the amount of time required to test configurations of CPS product lines while achieving the test objectives of each level. We empirically evaluate three commonly-used search algorithms, i.e., Genetic Algorithm (GA), Alternating Variable Method (AVM) and Greedy (Random Search (RS) is used as a baseline) by employing two case studies with the aim of integrating the best algorithm into our approach. Results suggest that as compared with RS, our approach can reduce the costs of testing CPS product line configurations by approximately 80% while improving the overall test quality.