Log in

View Full Version : Help!Frozen while debugging


kangdangda
June 23rd, 2006, 07:30
I am debugging a 3D game application using olly.The app behaves strangly when it encounters break points.If set a break point whereever in olly,run the application,the application will get into full screen mode.As soon as the app run at the break point, it stops and becomes frozon ,the app's UI becomes black and we can't switch to olly by Alt+Tab any more。If we press Alt+Tab,the olly's UI will not appear,what we can do is moving cursor to the left top corner of the screen and left clicking,the olly 's system menu will appear then and clicking the Close item to totally close olly and the app.This makes debugging extremely difficult.
I guess when the app encounter break point,it stops and don't refresh its 3D UI.The frozen UI makes any other app hidden behind even if we press Alt+Tab.
Any ideas to overcome it?

naides
June 23rd, 2006, 10:02
My guess. . .

The application, being video intensive, is bypassing windows video drivers and/or using its own video drivers, non standard access to the video bus. When the game closes or pauses gracefully, it returns the control to standard windows drivers, where olly displays and works seamlessly.
When you breakpoint in the middle of the game app, it has full control of the video bus and does not give it back to windows (and olly).

Solutions??

Two monitor debugging? (Worth a try. . .)

Run and debug the game inside a virtual machine ( I bet you it will not like the standard video driver installed in VMware)

Two computer debugging using visual SoftIce (I would put my bet on this one)

blabberer
June 23rd, 2006, 11:48
try ctrl+esc+shift -> select ollydbg and click bring to front

try setting ollydbg always on top

use conditional log break points without pausing application

if it is using ddraw see if you can force it run in windowed mode
by manipulating DDSEL_something flag while initializing in ddsetcooperativelevel

LLXX
June 23rd, 2006, 17:52
Quote:
[Originally Posted by naides]Solutions??

Two monitor debugging? (Worth a try. . .)

Run and debug the game inside a virtual machine ( I bet you it will not like the standard video driver installed in VMware)

Two computer debugging using visual SoftIce (I would put my bet on this one)
I've run into this problem before, most games and other full-screen graphics apps don't expect their graphics state to be changed halfway through execution and expect to "own" the video system for the duration of their execution.

SoftIce can be configured to use a second video card, it doesn't have to be a particularly powerful one - standard VGA will do just fine. Of course, you don't need two monitors these days, even cheap LCDs have dual inputs now.

kangdangda
June 24th, 2006, 07:34
Thanks every body!At last I've made the game UI windowed.Now break points works and no freezing any more.
The game uses d3d9,and I modified CreateDevice() paramter to do it.
ooops!

naides
June 24th, 2006, 18:34
Quote:
[Originally Posted by kangdangda]At last I've made the game UI windowed.Now break points works and no freezing any more.



could you give more detail on how you managed to make the app run in a window instead of whole screen?

I am just curious

kangdangda
June 24th, 2006, 20:46
I've looked up the manual of directx9,It says after Direct3DCreate9(), app should use CreateDevice() to Create a device instance at first. CreateDevice() has 6 parameter ,and the fifth parameter is a pointer to D3DPRESENT_PARAMETERS structure. The ninth member of D3DPRESENT_PARAMETERS is a bool to indicate whether directx should treate the app windows as fullscreen. Just changed it and everything go smoothly.