Log in

View Full Version : Hooking __thiscall subroutines

fywdm
April 6th, 2012, 18:03
Hi,

I'm using the MS Detours library for hooking and I've read about several ways to hook C++ __thiscall subroutines (e.g. member functions of classes). For example, using __stdcall, __fastcall and __declspec(naked).

I've got pretty good results by using __declspec(naked). For example, the following detour subroutine works well.

Code:
__declspec( naked ) int foo(int pThis, int bar){

* * * * _asm push 1

* * * * _asm mov ecx, eax

* * * * _asm call originalSubroutine

* * * * _asm retn 4

}


Unfortunately, it has some limitations which have been described at http://msdn.microsoft.com/en-us/library/4d12973a(v=vs.80).aspx.

Whenever I use, __stdcall it causes ecx to become 0 (the this pointer is not being passed). So __stdcall seems to be out of the question in this case. Unless I'm missing something here...

I've tried using __fastcall. However, it eventually leads to crashes. For example, the following piece of of code does not work properly.

Code:
__fastcall int foo(int pThis, int bar){

* * * * _asm push 1

* * * * _asm call originalSubroutine

* * * * _asm retn 4

}


I've omitted "_asm ecx, eax" due to ecx being passed.

What's recommended method for hooking class member functions?

Thanks in advance,

FY
disavowed
April 7th, 2012, 20:39
You'd want to use __declspec(naked). Are there specific limitations described on MSDN that make it a non-starter for you? Many can easily be overcome (for example, even though you can't "return" a value, you can do an "__asm mov eax, <return value>" at the end of your function).
fywdm
April 8th, 2012, 06:43
Quote:
[Originally Posted by disavowed;92257]You'd want to use __declspec(naked). Are there specific limitations described on MSDN that make it a non-starter for you? Many can easily be overcome (for example, even though you can't "return" a value, you can do an "__asm mov eax, <return value>" at the end of your function).


The only thing that seems odd to me is not being able to declare code within function scope. I would need to use nested scopes for declaring variables. Which seems counterintuitive. For example,

Code:
__declspec(naked) int __fastcall  power(int i, int j) {

   // calculates i^j, assumes that j >= 0



   // prolog

   __asm {

      push ebp

      mov ebp, esp

      sub esp, __LOCAL_SIZE

     // store ECX and EDX into stack locations allocated for i and j

     mov i, ecx

     mov j, edx

   }



   {

      int k = 1;   // return value

      while (j-- > 0) 

         k *= i;

      __asm { 

         mov eax, k 

      };

   }



   // epilog

   __asm {

      mov esp, ebp

      pop ebp

      ret

   }

}


However, I'm fine with it if it is the best way to hook member functions. I'm pretty new to this so I'm exploring the available options
ptr0x
April 16th, 2012, 21:05
I can't understand.

You create a naked function (which don't have prolog nor epilog) and creates a prolog and epilog on the function.

If you need a prolog and epilog, why don't you just use the cdecl calling convention?

Naked function is designed to situations where you need to work without a prolog and epilog (e.g. when u have to access the stack frame of the function that u r hooking).

I did not understand at all what is your trouble.
fywdm
April 17th, 2012, 13:05
Quote:
[Originally Posted by ptr0x;92321]I can't understand.

You create a naked function (which don't have prolog nor epilog) and creates a prolog and epilog on the function.

If you need a prolog and epilog, why don't you just use the cdecl calling convention?

Naked function is designed to situations where you need to work without a prolog and epilog (e.g. when u have to access the stack frame of the function that u r hooking).

I did not understand at all what is your trouble.


My problem was that ecx (the this pointer) was not passed properly to the hook function. So if I didn't use __declspec(naked), ecx was 0 or somehow corrupt. Some of my hook functions work without the need for a custom prolog/epilog. Some others require a custom prolog/epilog.

Does cdecl work properly for class member functions which have (originally) been declared as __thiscall? How would you declare the hooks of __thiscall routines?

I was looking for the most "user-friendly" way to hook __thiscall routines. Anyway, I've been using __declspec(naked) for a while now, and I'm starting to like it.
ptr0x
April 17th, 2012, 17:45
Well, I usually do a middle-hook function in these cases.

Add me on Skype or msn and we can talk about it.

Skype: andre.scos
msn: ptr0x@live.com

bye