Detection of Self-modifying and Indirect Jump Obfuscation Code

The new techniques we propose to detect self-modifying and indirect jump exploit code are called static taint analysis and initialization analysis. We observe that self-modifying and indirect jump exploit code first need acquire the absolute address of payload. Accordingly, we first try to find the piece of code which acquires the absolute address of payload at runtime from an instruction sequence. The variable which holds the absolute address will be marked tainted.

Then, we use the static taint analysis approach to track the tainted values and detect whether tainted data are used in the ways that could indicate the presence of self-modifying and indirect jump exploit code. A tainted variable is propagated to a new tainted variable by data transfer instructions that move data (e.g., push, pop, move) and data operation instructions that perform arithmetic or bit-logic operations on data (e.g., add, sub, xor). For data transfer instructions, the destination operand will be tainted if and only if the source operand is tainted. For data operation instructions, the destination operand will be tainted if and only if either source or destination operand is tainted.

Finally, we use initialization analysis to reduce false positives. We observed that the operands of self-modifying and indirect jump code must be initialized. Specifically, the jump target of indirect jump should be initialized; the operands of memory updating or writing instructions in self-modifying code should be initialized. If they are uninitialized, we will not consider them as attacks.