I have recently come up with a new anti-debug trick, which can be useful only if the "Break on new thread" option is set. The trick has been tried on OllyDbg v1.10 and Immunity Debugger v1.83 in WOW64 running on Windows 7. Actually, i am not sure if someone else has already found it.

In any affected debugger, if CREATE_THREAD_DEBUG_EVENT is received and the "Break on new thread" option is set, the debugger places an int3 software breakpoint on the lpStartAddress. There is a narrow time window between setting the int3 software breakpoint and recovering the original byte and this is what we are going to exploit.

N.B. The next few lines are only for demonstration. More complicated methods may evolve out of them.

Having two threads in an application, the first thread does almost nothing and the second one checks the first byte of the first thread's entrypoint, we can simply detect the debugger. See the image below.

http://2.bp.blogspot.com/-PKiZ37_05-Q/Txldw3Fm_GI/AAAAAAAAATI/QztIPGPiwnA/s1600/1.jpg

Here are the demo and its source code.
http://ollytlscatch.googlecode.com/files/demo.exe
https://docs.google.com/document/d/1kd-Fw110lbK9h-i6Jc2fs57LUjdU2sYji97XCLTTawE/edit

An XP-compatible demo and its source code.
http://ollytlscatch.googlecode.com/files/demo_xp.exe
https://docs.google.com/document/d/1G-6VSCrqM9KI_t82kPTGdo05cmaqyoVZG23o304Pk_o/edit


The original topic.
http://waleedassar.blogspot.com/2012/01/yet-another-anti-debug-trick.html