I found this nice page about Anti-Debugging tricks. It covers so many of them and if you know the techniques it’s really fun to read it quickly one by one. You can take a look yourself here: Window Anti-Debug Reference. One of the tricks really attracted my focus and it was soemthing like this:
push ss
pop ss
pushf
What really happens is that you write to SS and the processor has a protection mechanism, so you can safely update rSP immediately as well. Because it could have led to catastrophic results if an interrupt would occur precisely after only SS is updated but rSP wasn’t yet. Therefore the processor locks all interrupts until the end of the next instruction, whatever it is. However, it locks interrupts only once during the next instruction no matter what, and it won’t work if you pop ss and then do it again… This issue means that if you are under a debugger or a tracer, the above code will push onto the stack the real flags of the processor’s current execution context.
Thus doing this:
pop eax
and eax, 0x100
jnz under_debugging
Anding the flags we just popped with 0x100 actually examines the trap flag which if you simply try to pushf and then pop eax, will show that the trap flag is clear and you’re not being debugged, which is a potential lie. So even the trap flag is getting pended or just stalled ’till next instruction and then the debugger engine can’t get to recognize a pushf instruction and fix it. How lovely.
I really agree with some other posts I saw that claim that an anti-debugging trick is just like a zero-day, if you’re the first to use it – you will win and use it well, until it is taken care of and gets known. Although, to be honest, a zero-day is way cooler and another different story, but oh well… Besides anti-debugging can’t really harm, just waste some time for the reverser.
Since I wrote diStorm and read the specs of both Intel and AMD regarding most instructions upside down, I immediately knew about “mov ss” too. Even the docs state about this special behavior. But it never occurred to me to use this trick. Anyway, another way to do the same is:
mov eax, ss
mov ss, eax
pushf
A weird issue was that the mov ss, eax, must really be mov ss, ax. Although all disassemblers will show them all as mov ss, ax (as if it were in 16 bits). In truth you will need a db 0x66 to make this mov to work… You can do also lots of fooling around with this instruction, like mov ss, ax; jmp $-2; and if you single step that, without seeing the next instruction you might get crazy before you realize what’s going on. :)
I even went further and tried to use a priviliged instruction like CLI after the writing to SS in the hope that the processor is executing in a special mode and there might be a weird bug. And guess what? It didn’t work and an exception was raised, of course. Probably otherwise I won’t have written about it here :). It seems the processors’ logic have a kind of an internal flag to pend interrupts till end of next instruction and that’s all. To find bugs you need to be creative…never harm to try even if it sounds stupid. Maybe with another privileged instruction in different rings and modes (like pmode/realmode/etc) it can lead to something weird, but I doubt it, and I’m too lazy to check it out myself. But imagine you can run a privileged instruction from ring3…now stop.
Nice reference. You may already know about the page: http://www.openrce.org/reference_library/anti_reversing. My contribution with OutputDebugString came before reading “Windows Anti-Debug Reference.”
Funny enough that I took a look at it before posting this one. And I remember your trick with the anything_but_two thingy that was weird. I would code it differently – doing if (gle != 2) … :) cool one nevertheless.
Another interesting resource:
https://www.blackhat.com/presentations/bh-usa-07/Yason/Whitepaper/bh-usa-07-yason-WP.pdf
Thanks to Kasperle.
I’m working on a paper that talks about anti-unpacking tricks. It will include some anti-debugger tricks, too. I found some really cool ones, but no hints yet.
The pop ss/pushf thing is from the DOS days to detect debug.exe.
Looking forward to it, gimme a word when it’s out, I will publish it right here.
Ain’t new trick. Was widely used in early-mid 90’s in complex viruses and anti-viruses.
Most viruses tried to get real DOS int 21h handler by tracing to the DOS kernel. Anti-viruses tried to fight tracing with few tricks including this one. And vice a versa – if virus loaded first, it tried to not allow AV software to work with DOS kernel directly.
Nice explanation.
But I wonder why the trap flag isn’t set when single-stepping over the pushf/pushfd instruction in a debugger without a preceding pop ss. Does the debugger correct the pushed flags on the stack?
I will appreciate if you provide more details on this. Thanks.