During reverse engineering it has been noticed that while performing analysis of hooking functions ,system gets in hang state.The problem of this kind inherits an element of Return Value Failure.Usually this kind of behavior is shown when an object does not return the unique value. This means the object is not returned properly. In multithreading enviornment , this type of problem occurs when number of threads are activated and one out of them perform malfunctioning. Other cause is failure in returning the thread. This is due to cross referential interdependencies of various functional threads in the code.

In some cases , it has been observed that the hook function does not return properly and entangled some where after performing the task. Another possible cause is Argument Specification. The passing arguments should be scrutinized properly.Try to check the Hook functions.One can set practical Debug Checks while Traversing Hooks in a code.

Lets see:

static int my_something_doer(request_rec *r, int n)
{
...
return OK;
}

static void my_register_hooks()
{
ap_hook_do_something(my_something_doer, NULL, NULL, HOOK_MIDDLE);
}

mode MODULE_VAR_EXPORT my_module =
{
...
my_register_hooks /* register hooks */
};

static void register_hooks()
{
static const char * const aszPre[] = { "mod_xyz.c", "mod_abc.c", NULL };

ap_hook_do_something(my_something_doer, aszPre, NULL, HOOK_MIDDLE);
}

So mostly when python scripts are designed the calling functions easily hooks the registered object but unble to return properly.The designed code is not able to handle the response , as a result functions are hanged. So the scripts usually fail during hooking scenarios.In windows enviornment :

HHOOK SetWindowsHookEx( int idHook,
HOOKPROC lpfn,
HINSTANCE hMod,
DWORD dwThreadId
);

BOOL UnhookWindowsHookEx( HHOOK hhk
);

HOOKPROC hkprcSysMsg;
static HINSTANCE hookDLL;
static HHOOK hhookSysMsg;

hookDLL = LoadLibrary((LPCTSTR) "c:\\windows\\hooking.dll";
hkprcSysMsg = (HOOKPROC)GetProcAddress(hookDLL, "SysMessageProc";
hhookSysMsg = SetWindowsHookEx(WH_SYSMSGFILTER,hkprcSysMsg,hookDLL,0);

For simple hook function prototypes in current process:

LPVOID HookFunctionInCurrentProcess(LPCSTR,LPCSTR,LPVOID);
BOOL UnHookFunctionInCurrentProcess(LPCSTR,LPCSTR,LPVOID);

So the system specific hooking procedures follow the same paradigm as defined above. The scripting code for automating the hooking procedures should be implemented with designated checks of error handling to have good reverse engineering analysis.

----
0kn0ck






https://www.openrce.org/blog/view/998/Hang_problem_due_to_Hooking_Curb_in_Codes.