Automated Specification Extraction and Testing of Autonomous Systems

This exhibition presents two projects. The first project is a framework and associated tooling for the extraction of system specifications from test data that has been automatically generated from automated executions of the system. The test data in our framework consists of sequences of inputs to and the corresponding outputs from the system. From an initial collection of test data, data-mining techniques are used to infer the invariants; then the system is instrumented with the invariants and new test data is generated to try to invalidate the mined invariants. This process can be iterated, yielding a procedure which converges to a set of invariants that the testing technology is unable to invalidate. The process is currently being applied to pacemaker models from the UPenn CyberCardia team.

The second project presents an approach for model-based metamorphic-testing of autonomous systems in a simulated environment. We present a method for automatically generating a large number of test cases for an autonomous drone, which is a cyber physical system. Metamorphic testing is used to analyze the output from test cases and flag unstable or unsafe behavior. This method can help identify potential problem in a simulated test without the need to specify the correct behavior in every scenario or manually investigating the output from every test.