Peter Ferrie, a former employee of Symantec, who now works for MS wrote a paper about Anti Unpacker tricks. I was really fascinated reading that paper. There were so many examples in there for tricks that still work nowadays. Some I knew already some were new to me, he covers so many tricks. The useful thing is that every trick has a thorough description and a code snippet (mostly Assembly). So now it becomes one of the most valueable papers in the subject and you should really read it to get up to date. The paper can be found here.
One idea I that I really like from the paper, is something that Peter himself found, that you can use ReadFile (or WriteProcessMemory) to override a memory block so no software breakpoints will be raised when you execute it. But on a second thought, why a simple memcpy won’t do the same trick?
If you guys remember the Tiny PE challenge I posted 2 years ago in Securiteam.com, then Peter was the only one who kicked my ass with a version of 232 byts, where I came with 274 bytes. But no worries, after a long while I came back with a version of 213(!) bytes (over here) and used some new tricks. Today I still wait for Peter’s last word…
Have fun
Pretty interesting paper. I think that the ReadFile / WriteProcessMemory work because memory are copied for the kernel and this operation cannot be catched by the debugger. It will work with any function that write predictable data in a submited userland buffer. (I didn’t test that so that just an idea).
Ok, it depends what kind of breakpoint we talk about. Peter talked about execution breakpoints, while I was talking about memory access bp’s.
The thing is that if you put a hw memory access bp, then when you use the kernel to do the writings, the event won’t be raised in usermode. Which is pretty much cool. Now, if we talk about execution bp’s, then if the debugger won’t trace the memory writes it doesn’t have any chance to fix an overriden CC instruction in memory. Or maybe this is where I am wrong at, anyone?
Software breakpoints (int 3s) will be removed by ReadFile, and yes – memcpy(), too. The memcpy() technique was used in DOS days (it was just rep movsb, at that time). It’s just the ReadFile is less obvious.
Hardware break-on-write doesn’t trigger for ReadFile, but hardware break-on-execute always works.
The second paper contains only new tricks. Hopefully, that will be even more interesting. Coming as soon as I can find a place to present it. :-)