Software, including common examples such as commercial applications or embedded device firmware, is often delivered as closed-source binaries. While prior academic work has examined how to automatically discover vulnerabilities in binary software, and even how to automatically craft exploits for these vulnerabilities, the ability to answer basic security-relevant questions about closed-source software remains elusive. This project aims to provide algorithms and tools for answering these questions. Leveraging prior work on emulator-based dynamic analyses, we propose techniques for scaling this high-fidelity analysis to capture and extract whole-system behavior in the context of embedded device firmware and closed-source applications. Using a combination of dynamic execution traces collected from this analysis platform and binary code analysis techniques, we propose techniques for automated structural analysis of binary program artifacts, decomposing system and user-level programs into logical modules through inference of high-level semantic behavior. This decomposition provides as output an automatically learned description of the interfaces and information flows between each module at a sub-program granularity. Specific activities include: (a) developing software-guided whole-system emulator for supporting sophisticated dynamic analyses for real embedded systems; (b) developing advanced, automated techniques for structurally decomposing closed-source software into its constituent modules; (c) developing automated techniques for producing high-level summaries of whole system executions and software components; and (d) developing techniques for automating the reverse engineering and fuzz testing of encrypted network protocols. The research proposed herein will have a significant impact outside of the security research community. We will incorporate the research findings of our program into our undergraduate and graduate teaching curricula, as well as in extracurricular educational efforts such as Capture-the-Flag that have broad outreach in the greater Boston and Atlanta metropolitan areas. The close ties to industry that the collective PIs possess will facilitate transitioning the research into practical defensive tools that can be deployed into real-world systems and networks.
TWC: Medium: Collaborative: Automated Reverse Engineering of Commodity Software