Antibypassing Four-Stage Dynamic Behavior Modeling for Time-Efficient Evasive Malware Detection

With the widespread adoption of virtualization technology, it is imperative to strengthen its security, and dynamically modeling and instantly trapping malicious behaviors are challenging problems. Extant detection methods will be invalidated after the evasive malware manipulates the behavior trace. Currently, there is no approach to model the complex dynamic behavior of evasive malware, leading to missed opportunities for optimal detection. This work first presents antibypassing four-stage dynamic behavior modeling for time-efficient evasive malware detection (AFDBM-TEMD). AFDBM-TEMD models the interaction between evasive malware and its execution environment, identifying the optimal detection phases for various evasive malware. Moreover, it traps the crucial instructions and system calls invoked by the evasive malware into the virtual machine monitor layer to obtain the dynamic behavior information (including transmitted parameters, execution time, process information, return values, etc.) to identify the malicious software. Experimental results show that AFDBM-TEMD achieves new state-of-the-art results, and the proposed dynamic behavior modeling method has wide applicability, while the average detection time reaches milliseconds. Specifically, the detection rate is improved from 0-56.52% to 100% in contrast with the comparative methods, and the detection speed is increased by more than six times.