Biblio

Code reuse techniques can circumvent existing security measures. For example, attacks such as Return Oriented Programming (ROP) use fragments of the existing code base to create an attack. Since this code is already in the system, the Data Execution Prevention methods cannot prevent the execution of this reorganised code. Existing software-based Control Flow Integrity can prevent this attack, but the overhead is enormous. Most of the improved methods utilise reduced granularity in exchange for a small performance overhead. Hardware-based detection also faces the same performance overhead and accuracy issues. Benefit from HPC's large-area loading on modern CPU chips, we propose a detection method based on the monitoring of hardware performance counters, which is a lightweight system-level detection for malicious code execution to solve the restrictions of other software and hardware security measures, and is not as complicated as Control Flow Integrity.

Today the technology advancement in communication technology permits a malware author to introduce code obfuscation technique, for example, Application Programming Interface (API) hook, to make detecting the footprints of their code more difficult. A signature-based model such as Antivirus software is not effective against such attacks. In this paper, an API graph-based model is proposed with the objective of detecting hook attacks during malicious code execution. The proposed model incorporates techniques such as graph-generation, graph partition and graph comparison to distinguish a legitimate system call from malicious system call. The simulation results confirm that the proposed model outperforms than existing approaches.