Vendor Truth Serum

Abstract:

Software engineering tools are an important technology for building secure software and cyber-physical systems. There is a class of tools targeting software security, stability, and correctness. Experience is that any single tool covers a relatively small percentage of potential errors. A mix of tools should increase coverage. One needs to know what the tools cover in order to make a rational decision of which tools a software engineering project needs. Likewise, one needs to know where the vulnerabilities are that no tools can check. As such, industry and government software developers need to test their code against multiple test tools. As such, a real issue is deciding which tools a given developer needs to use to ensure satisfactory test coverage over their software artifact. What is needed is a way for developers to know which tools provide what coverage, so they can make informed choices and accomplish satisfactory testing in minimal time at minimal expense. Unfortunately, there are common industry contractual practices which inhibits making such knowledge generally available.

This report, a result of the Vendor Truth Serum project in the S2ERC, examines approaches for enabling the sharing of software engineering test tool performance and coverage benchmark results.

--

Dr. Eric Burger is Research Professor of Computer Science and Director of the Georgetown site NSF-sponsored Security and Software Engineering Research Center (S 2 ERC). Prior to Georgetown he was an executive at both public and private companies in the network equipment, enterprise software, and network services market, including Neustar (CTO) and BEA Systems (VP Engineering and Deputy CTO). He has a PhD, MBA, and SBEE from MIT, KU Leuven, and IIT.