Log in

View Full Version : Tracking threads in Olly

live_dont_exist
July 28th, 2011, 12:04
Hi,
I've been trying to reverse a piece of malware. Everything is fine until I came to a point where I can see traffic going out on to the Internet in Wireshark..but the code I am looking at in Olly cannot possibly be doing that [But it is].. I've set breakpoints and traffic appears only between those breakpoints.

So I'm thinking it is some other thread which is doing it somewhere else. I can use the Debug option to break when a Thread starts...or even switch to that thread after its created but this doesn't tell me where the traffic is getting generated from..it just goes into System DLL space.

For e.g.. I can see an SSL connection getting established but I don't know from where..

I have attached a snapshot of the relevant assembly code from Olly.

There are breakpoints(F2) at 0040D118 and 0040D15C. Its a loop from which the process exits only when a lot of data is sent.

Any ideas?? Please let me know if you need any further information.

Thnx
Arvind
disavowed
July 31st, 2011, 09:19
Set a breakpoint on WS2_32!send and look at the callstack once that breakpoint is hit.
live_dont_exist
September 18th, 2011, 01:04
Old thread I know and I'm bumping it but I thought this would be relevant for any other newbies reading this.

Whenever a new thread is started by a program it calls the CreateThread function (in whatever I have seen anyway). One of the arguments of the CreateThread function is the address (in user space) of the new thread. So if you look at that address, go to that address and set a breakpoint there; when the thread starts you will be able to track it.

Another interesting thing(for me anyway) is that the thread do not need to start immediately, thats defined by another argument to the CreateThread function, which determines whether the thread should run immediately or be in a Suspended state and start after the 'ResumeThread' function is called sometime later.

Hope this helps!

Arvind
Darkelf
September 18th, 2011, 14:15
Hi Arvind,

it's unrelated in a way but I'd like to let you know that you've shown an attitude I like a lot.
You once had a problem, stuck to it, found a solution and now you are sharing your gained knowledge with others.
That's great!

Keep doing so.

Regards
darkelf
live_dont_exist
September 19th, 2011, 00:30
Thanks darkelf! I hope to do so more often as I gain skill.

However why do you feel it is unrelated? The reason I ask is that it seemed to fit exactly what I needed. The scenario being:


Program starts - runs runs runs...
Comes to a Create Thread function with a '0' for the dWCreationFlags which means the thread starts immediately after CreateThread completes.
Now the original thread just keeps rotating in a Kernel32.sleep(3000) or whatever number loop..indefinitely.. and I have no clue what is happening
However I keep seeing traffic in Wireshark even during the sleep function of the first thread..so it means something else is doing it.
At this point if I look at the address in lpStartAddress (3rd argument to create thread) and set a breakpoint there... the Thread which is created eventually does reach there and I can follow its progress in user mode


So I feel it did help. Is there anything I missed?

Thnx
Arvind


Edit: I got what you meant later
live_dont_exist
September 22nd, 2011, 08:55
I forgot to mention 1 thing here which I stumbled upon today.

When you break in the new thread and follow its execution, all is well for as long as the thread exists. However if the thread terminates, you have a problem as it would seem that Olly is hanging and not moving forward. At least I thought it was .

Well..it isn't.. you just have to go back to the place where the thread was initially called. I did this by pausing Olly(F12) and then doing an (Alt+F9) to go back to user mode. This took me right where the CreateThread was initially called.

Probably its obvious to many..but still thought I'd put that bit in here as well to complete my solution
blabberer
September 23rd, 2011, 00:44
olly appears to hang because ExitThread Is A Function That Is Declared As NORETURN

meaning this function is going to return nowhere

so if you want ollydbg to return to somewhere you either need to setup breaks in advance or you need to pause and continue from another point


Code:


WINBASEAPI

DECLSPEC_NORETURN

VOID

WINAPI

ExitThread(

    __in DWORD dwExitCode

    );


Code:


C:\WinDDK\7600.16385.1\inc>grep -r -i NORETURN  *.* | grep -i "#define DECLSPEC_

NORETURN"

api/ntdef.h:#define DECLSPEC_NORETURN   __declspec(noreturn)

C:\WinDDK\7600.16385.1\inc>



http://msdn.microsoft.com/en-us/library/k6ktzx3s%28v=vs.80%29.aspx

so When zwTerminateThread is Called System simply transfers execution to the open ended WinProc Loop That keeps on looping for GetMessage() Translatemessage() DespatchMessage() Loop for the next message

open up iczelions Win32Asm Tut 015 threads.exe in ollydbg and watch , break , and f9 and f12 on it to see how things behave

if you f8 on a create thread call

your ThreadFunc may never be hit and olly may appear to be hung because
Threads Are SCHEDULED and it may Happen that CPU never Got a Chance To Schedule Your Thread

Debugging MultiThreaded App needs a good Defense Rather Than offense
You Need To setup Traps and mines all along the border and double fence it with barbed wire and setup command posts with alert sentries on duty

and still there will be trespassers that will tunnel below your mines and make free contraband routes
live_dont_exist
September 23rd, 2011, 01:07
So what you're saying is, if the thread function (which CreateThread called) had a RETN it would automatically jump back to the main code, to the place just after CreateThread was called which in this case, wasn't there, so it "hung".

So whats the Defense in Depth strategy for this? I can think of a few:
--- Break on Thread Start
--- Break on Thread Close
--- Break on Create Thread address
--- Break on code just after CreateThread
--- Study code before hand to find what system calls used by thread, and break there (maybe conditionally?)

What else? I cant think of anything else as of now; so please add to this

Thanks
Arvind
p.s.... Downloaded all of iczelions tutorials just now as well.. thanks.
blabberer
September 23rd, 2011, 03:00
if the thread function didnt have a return it wouldnt compile

if it had a return it would return back to the callee

and the callee is not any code in main but system code

the flow of CreateThread Is like this

your code calls CreateThread which is in kernel32.dll
it is a wrapper to
7c8106f0 e8d7fdffff call kernel32!CreateRemoteThread (7c8104cc)

Code:




0:000> kbn1

 # ChildEBP RetAddr  Args to Child              

00 0013fe30 00401172 00000000 00000000 004011bd kernel32!CreateThread



0:000> ub 00401172 l1

image00400000+0x116d:

0040116d e8d4000000      call    image00400000+0x1246 (00401246)



0:000> u 00401246 l1

image00400000+0x1246:

00401246 ff2508204000    jmp     dword ptr [image00400000+0x2008 (00402008)]



0:000> dps 00402008 l1

00402008  7c8106d7 kernel32!CreateThread



0:000> tc

kernel32!CreateThread+0x19:

7c8106f0 e8d7fdffff      call    kernel32!CreateRemoteThread (7c8104cc)


which then calls into kernel (Ring 0) via sysenter using NtcreateThread in ntdll.dll

Code:




0:000> tc 10000

kernel32!CreateThread+0x19:

7c8106f0 e8d7fdffff      call    kernel32!CreateRemoteThread (7c8104cc)

kernel32!CreateRemoteThread+0xa:

7c8104d6 e8fb1fffff      call    kernel32!_SEH_prolog (7c8024d6)

kernel32!CreateRemoteThread+0x61:

7c81052b e88cfdffff      call    kernel32!BaseCreateStack (7c8102bc)

kernel32!BaseCreateStack+0x2f:

7c8102eb ff15c012807c    call    dword ptr [kernel32!_imp__RtlImageNtHeader (7c8012c0)]

ntdll!RtlpImageNtHeader+0x7:

7c9102e0 e8e6e5ffff      call    ntdll!_SEH_prolog (7c90e8cb)

ntdll!RtlpImageNtHeader+0x51:

7c91031e e8e3e5ffff      call    ntdll!_SEH_epilog (7c90e906)

kernel32!BaseCreateStack+0x105:

7c8103a5 ffd3            call    ebx

ntdll!NtAllocateVirtualMemory+0xa:

7c90cf78 ff12            call    dword ptr [edx]

ntdll!KiFastSystemCall+0x2:

7c90e512 0f34            sysenter

kernel32!BaseCreateStack+0x158:

7c8103f6 ffd3            call    ebx

ntdll!NtAllocateVirtualMemory+0xa:

7c90cf78 ff12            call    dword ptr [edx]

ntdll!KiFastSystemCall+0x2:

7c90e512 0f34            sysenter

kernel32!BaseCreateStack+0x1a1:

7c810425 ff15c412807c    call    dword ptr [kernel32!_imp__NtProtectVirtualMemory (7c8012c4)]

ntdll!NtProtectVirtualMemory+0xa:

7c90d6f8 ff12            call    dword ptr [edx]

ntdll!KiFastSystemCall+0x2:

7c90e512 0f34            sysenter

kernel32!CreateRemoteThread+0x85:

7c810550 e8eefeffff      call    kernel32!BaseInitializeContext (7c810443)

kernel32!CreateRemoteThread+0x95:

7c810560 e8338effff      call    kernel32!BaseFormatObjectAttributes (7c809398)

kernel32!CreateRemoteThread+0xc4:

7c81058f ff155014807c    call    dword ptr [kernel32!_imp__NtCreateThread (7c801450)]

Breakpoint 0 hit

ntdll!NtCreateThread:

7c90d1ae b835000000      mov     eax,35h

0:000> tc

ntdll!NtCreateThread+0xa:

7c90d1b8 ff12            call    dword ptr [edx]

0:000> ln edx 

(7ffe0300)   SharedUserData!SystemCallStub   

Exact matches:

    SharedUserData!SystemCallStub = <no type information>

0:000> tc

ntdll!KiFastSystemCall+0x2:

7c90e512 0f34            sysenter


there inside the system your ThreadFunc is Called Via Despatch Mechanism
usually depending on the function if it is gui it most probably would be DispatchClientmessage()

Code:


ChildEBP RetAddr  Args to Child              

0013fe18 7e418816 00401130 0010023e 00000500 user32!InternalCallWinProc+0x28

0013fe80 7e428ea0 00000000 00401130 0010023e user32!UserCallWinProcCheckWow+0x150

0013fed4 7e428eec 006687e8 00000500 00000000 user32!DispatchClientMessage+0xa3

0013fefc 7c90e473 0013ff0c 00000018 006687e8 user32!__fnDWORD+0x24

0013ff20 7e4191be 7e42776b 0013ff60 00000000 ntdll!KiUserCallbackDispatcher+0x13

0013ff48 00401111 0013ff60 00000000 00000000 user32!NtUserGetMessage+0xc

WARNING: Stack unwind information not available. Following frames may be wrong.

0013ffac 0040102b 00400000 00000000 00161ee0 image00400000+0x1111

0013fff0 00000000 00401000 00000000 78746341 image00400000+0x102b



and once your threadfunction ends with a return it will go back to system via some callbackreturns

Code:


0:000> tc

user32!UserCallWinProcCheckWow+0x109:

7e41881d e81f000000      call    user32!UserCallWinProcCheckWow+0x157 (7e418841)

0:000> 

user32!UserCallWinProcCheckWow+0x161:

7e41884b e878030000      call    user32!_EndUserApiHook (7e418bc8)

0:000> 

user32!_EndUserApiHook+0xb:

7e418bd3 ff152013417e    call    dword ptr [user32!_imp__InterlockedDecrement (7e411320)]

0:000> 

user32!UserCallWinProcCheckWow+0x16a:

7e418854 ff153414417e    call    dword ptr [user32!_imp__RtlDeactivateActivationContextUnsafeFast (7e411434)]

0:000> 

user32!UserCallWinProcCheckWow+0x111:

7e418825 e8d6fdffff      call    user32!_SEH_epilog (7e418600)

0:000> 

user32!DispatchClientMessage+0xa9:

7e428ea6 e855f7feff      call    user32!_SEH_epilog (7e418600)

0:000> 

user32!__fnDWORD+0x2f:

7e428ef7 e8a805ffff      call    user32!XyCallbackReturn (7e4194a4)

0:000> 

kernel32!WaitForSingleObjectEx+0xcb:

7c8025f3 e828000000      call    kernel32!WaitForSingleObjectEx+0xdb (7c802620)


if you want the thread to wait you use Wait Functions and act upon the event signals
if you dont want the the thread you call Exit thread like you call ExitProcess in your main Code
or simply return back

on simply returning back system will Act as a Garbage Cleaner will call your ThreadExit proc if you have set it up
and then will Terminate the thread and will send the execution into an infinite msgpump loop

and once a pertinent messgae that is meant for your app is recieved the system will pass it to your app

it may be another create thread

a user mode debugger has no way of intercepting the flow that happens inside system so it appears that the user mode debugger is hung
live_dont_exist
September 23rd, 2011, 03:30
Okay...so while I didn't completely understand the debugger output you posted, I now understand that CreateThread returns 'only' into System code and I can sit and wait for as long as I want in user mode; it will never come back. The only way to track it is to use a kernel mode debugger, like (I think) you have done.

So in that case, is the only way to return to user mode to F12 and Alt+F9 like I did?.. assuming I want to get back to the main thread? What do these do? As in, I know that F12 'pauses' the program [Paused in Yellow - bottom right corner] and Alt+F9 runs a lot of System code and comes back to a 0400xxxx address. Will read more about what exactly these do.. but is there any other way to return to the main code?
blabberer
September 23rd, 2011, 04:41
none of the above debugger output is from kernel mode it is all in user mode only

to return to main code you need to find the the winproc
and set up a breakpoint there

or look at the call stacks of each thread and set bps on the functions that exist in there

in ollydbg you can click the K (view callstack ) (alt+k) to view the call stack of the threads

edit
in the window that pops up right click again to view the various threads and upon selecting a threadid its call stack will be visible

ALt+K --> rightclick Thread -> ThreadId

you will find functions which are queued

set relevent breakpoints