Buffer overflow protection based on adjusting code segment limit

Stack smashing is a common mode of buffer overflow attack for hijacking system control. A segment-based non-executable stack approach is proposed and evaluated to defend against stack-based buffer overflow attacks under Windows operating system and Intel 32-bit CPUs. A kernel device driver is designed to relocate the application's user-mode stack to the higher address and to modify the effective limit in the code segment descriptor, in order to exclude the relocated stack from the code segment. Once any code that attempts to execute the malicious code residing in the stack, a general-protection exception of exceeding the segment limit is triggered so the malicious code will be terminated. It is highly effective in preventing both known and yet unknown stack smashing attacks, and its performance overhead is lower than the page-based non-executable stack approach.