Hi

I'm debugging my way through the (in my view) fairly complex unpacking of a windows malware binary. After some interesting use of SEH, creation of new segments and a jump into such a new section, i come across the following statements:

int 3
push ebp
hlt

No exception handler is registrered except the standard in kernel32.dll, and in my view the program should be allowed to continue while passing exceptions to the application instead of the debugger. And hlt should throw an exception since the cpu is not in protected mode, but be allowed to continue afterwards.
But when i set a hardware breakpoint on the following statement, and allow the process to run, it shuts down.

Could anyone please tell me whether i am wrong in my assumptions on this particular part, or if it looks like i have made some error earlier in the unpacking?

Best regards,
Lasse

If there is an instruction that will cause an exception and no exception handler is registered, the program will naturally crash. There are a number of possibilities here:

1) The program is using the SetUnhandledExceptionFilter antidebug trick. This function registers a top-level exception handler, separate from the SEH stack, that gets run when no SEH handler is able to handle the exception. The trick here is that this handler is *not* called if the program is being debugged. This way when the program is running normally the top level handler will catch the exception and allow execution to continue, while running it in a debugger will cause the exception not to be handled resulting in a crash.

There are OllyDbg plugins to counter this trick, just do a quick search.

2) The program has already detected that it's being debugged, and now sends you into a bunch of garbage code that will crash it. You want to simply avoid being detected here. Again most (user mode) debugger detection tricks have been circumvented in the form of OllyDbg modifications and plugins. See the Shadow's Olly modification and the AdvancedOlly plugin; both are available from the CRCETL on this site. Or if you are using IDA for debugging, there is IDAStealth.

Good luck.

Thanks, i guess i'll start over again, and see if i missed anything... I prefer not to use such plugins til i can understand what happens.