Log in

View Full Version : Confusing code snippet

VectoR.360
September 17th, 2006, 21:09
While in the process of reverse engineering a C++ windows application,
and ran across a block of code that has me stumped.

I am not certain, but the code appears to be jumping to a
C++ method based on a VTABLE offset. Has anyone seen
similar code? Can anyone shed some light on this code snippet?

Code:


 SUB_L00A27C60:

  		push	ecx

  		cmp	eax,00001000h

  		lea	ecx,[esp+08h]

  		jc 	L00A27C80

 L00A27C6C:

  		sub	ecx,00001000h

  		sub	eax,00001000h

  		test	[ecx],eax

  		cmp	eax,00001000h

  		jnc	L00A27C6C

 L00A27C80:

  		sub	ecx,eax

  		mov	eax,esp

  		test	[ecx],eax

  		mov	esp,ecx

  		mov	ecx,[eax]

  		mov	eax,[eax+04h]

  		push	eax

  		retn
VectoR.360
September 17th, 2006, 21:38
Never mind the request; I figured it out. Commented code follows...

It appears that the function is used to adjust the stack pointer, with a memory probe every 1000h bytes to check for stack overflow.

Code:


;------------------------------------------------------------------------------

; ExpandStack

; eax = number of bytes to be subtracted the stack pointer

;------------------------------------------------------------------------------

 SUB_L00A27C60:

    push ecx                    ; save ecx

    cmp eax,00001000h

    lea  ecx,[esp+08h]       ; ecx = original stack pointer

    jc   L00A27C80

 L00A27C6C:

    sub  ecx,00001000h       ; subtract 1000 from stack pointer

    sub  eax,00001000h       ; decrement the argument

    test [ecx],eax              ; probe the stack - exception if overflow

    cmp  eax,00001000h

    jnc  L00A27C6C           ; loop

 L00A27C80:

    sub  ecx,eax             ; calculate the absolute address

    mov  eax,esp             ; get the original stack

    test  [ecx],eax          ; probe the stack - exception if overflow

    mov  esp,ecx             ; set the stack pointer

    mov  ecx,[eax]           ; restore ecx

    mov  eax,[eax+04h]       ; get the real return address

    push eax                   ; push it on the adjusted stack

    retn                       ; return

;------------------------------------------------------------------------------
naides
September 17th, 2006, 22:28
Glad we helped you help yourself!
Silver
September 18th, 2006, 05:43
Oh, that looks suspiciously like a Boundschecker/Devstudio instrumented build. I never really paid attention to the disasm of a BC instrumentation but I do remember seeing a very similar pattern.
reverser
September 18th, 2006, 12:08
This code is used when local variables for a function occupy more than 1000h bytes, or in the implementation of _alloca function. Since the stack section has only one quard page, you cannot just sub from esp such a size, because in that case you might write beyond the guard page with the first push. So the compiler inserts the code which probes stack in 1000h increments to make it grow if needed.

See "\Program Files\Microsoft Visual Studio\VC98\CRT\SRC\Intel\CHKSTK.ASM" for the commented code.
dELTA
September 18th, 2006, 15:39
Nice info reverser, thanks.
LLXX
September 19th, 2006, 02:42
I remember when older compilers would just set a fixed stack size and allocate with a simple sub esp xxxxxxxx... any obvious advantages of this new method, as I only see the main disadvantage (increased superfluous code) and the older method worked fine...
reverser
September 19th, 2006, 03:55
LLXX
Let's say you need to allocate a big structure in stack, e.g. 5000h bytes big. Try compiling this code:
Code:


.386

.model flat, stdcall

ExitProcess proto uExitCodeWORD

MessageBoxA PROTO hwndWORD, lpTextWORD, lpCaptionWORD, uTypeWORD

.data

MsgBoxCaption  db "Iczelion Tutorial No.666",0

MsgBoxText       db "I use a lot of stack space!",0

.code

start:

  	push ebp

	mov ebp, esp

	sub esp, 5000h

	invoke MessageBoxA, 0, addr MsgBoxText, addr MsgBoxCaption, 0

	invoke ExitProcess,0

end start

Then run it. WTF, where's my message box
Silver
September 19th, 2006, 06:11
This one caught my interest, thanks reverser. I found some related articles which may teach many here to suck eggs, but useful nevertheless:

http://www.codeproject.com/cpp/StackShrink.asp
http://groups.google.com/group/comp.lang.forth/msg/6d16ff18755c33ff
http://www.bluebytesoftware.com/blog/PermaLink,guid,5df78179-d995-487e-a6a8-3b040f59051e.aspx
LLXX
September 20th, 2006, 03:00
Quote:
[Originally Posted by reverser]LLXX
Let's say you need to allocate a big structure in stack, e.g. 5000h bytes big. Try compiling this code:
Code:


.386

.model flat, stdcall

ExitProcess proto uExitCodeWORD

MessageBoxA PROTO hwndWORD, lpTextWORD, lpCaptionWORD, uTypeWORD

.data

MsgBoxCaption  db "Iczelion Tutorial No.666",0

MsgBoxText       db "I use a lot of stack space!",0

.code

start:

  	push ebp

	mov ebp, esp

	sub esp, 5000h

	invoke MessageBoxA, 0, addr MsgBoxText, addr MsgBoxCaption, 0

	invoke ExitProcess,0

end start

Then run it. WTF, where's my message box
Linked it with /stack:4096,20480 (or larger) - no problem. If you remember the days of 16-bit DOS, we had stack size options since then. It's just that the default size provided by the linker is too small, so we have to increase it.