The integrity of commodity operating system kernels is threatened by rootkits that modify key kernel data structures to achieve a variety of malicious goals. While rootkits have historically been known to affect control data in the kernel, recent work demonstrates rootkits that affect system security by modifying non-control data, such as linked lists used to manage bookkeeping information and metadata used for memory management. Existing techniques fail to detect such rootkits effectively. This project is developing techniques to provide real-time protection against rootkits by detecting anomalies in both control and non-control kernel data behavior using automatically-generated integrity specifications. This goal is being achieved in two steps. First, a technique to mine specifications of kernel data structure integrity is being developed. These specifications are be mined automatically as data structure invariants. Second, these techniques are being extended using operating system support to provide real-time detection. Impacts and Results: The techniques developed in this project will defend against the next generation of rootkits, and will enable real-time detection of such rootkits. In addition, techniques to infer kernel invariants may also find applications in operating system reliability, fault tolerance and software engineering. The PIs will disseminate the results by releasing the tools developed. The results of this project will equip the workforce with an inter-disciplinary toolkit, that combines operating systems, computer security, and software engineering, to address the challenges posed by the next generation of stealth malware.
CT-ISG: Advanced Techniques to Detect Kernel-Level Rootkits