Comparative Evaluation of Static Analyses that Find Security Vulnerabilities

To find security vulnerabilities, many research approaches and commercial tools use a static analysis and check constraints. Previous work compared using a benchmark several approaches where the static analysis and constraints are combined, and the evaluation focused on corner cases in the Java language. We extend the comparative evaluation of these approaches to include one approach that separates the constraints from the static analysis. We also extend the benchmark to cover more classes of security vulnerabilities. Approaches that combine the static analysis and constraints work well for vulnerabilities that are sensitive to the order in which statements are executed. The additional effort required to write separate constraints is rewarded by better recall in dealing with dataflow communication and better precision for callback methods that are common in applications built on frameworks such as Android.