I'm having problem using directx acceleration on my applications. When i run some application which uses DirectDraw/Direct3D, my Windows simply craches! Then, i ran DXDIAG.EXE to check what is wrong.
Result tests:
Using DirectDraw acceleration= Windows crashes
No acceleration = no crash! Same result for the Direct3D tests (with acceleration = crash, no = works fine)

The problem, apparently, is an infinite loop on an important piece of code of the Window's core, because Windows stops completely! I can't do anything on it (keyboard/mouse = stopped). Oh, i'm using win xp pro!

So, i openned dxdiag.exe in OllyDbg and started debugging it. Tracing a lot of shi*** after the first breakpoint on the OK button that starts the DirectDraw test, i found this code fragment:


The piece of code above is executed a lot of times and when it executes 12 times it crashes the Windows. So, i putted a BP on the SYSENTER instruction and waitted for the eleventh breakpoint. Then used F7 to trace INTO this instruction, but i see the Windows crashes too on debugging time!

I need information about SYSENTER instruction. I know it is inside the ntdll.dll but what is the puporse of it?

Hi,
From Intel doc: SYSENTER - Fast System Call
"Executes a fast call to a level 0 system procedure or routine, it is a companion of SYSEXIT..."

You need a r0 debugger to correctly follow the rabbit in its hole, under a VM, imho =)
I can't help more, I'm not skilled but I hope it helps (or simplifies the task) mates that will help you.

Regards.

PS: You can find Intel manuals on internet, good reference.

well for some kind of expalnation about sysenter you can refer to this thread
http://www.woodmann.com/forum/showthread.php?t=9730

you cannot single step into syscalls using a ring 3 debugger you would need a r0 debugger (either windbg or erstwhile softice) for stepping into them

with softice you can simply step through the transition phase
with windbg you can skip the transistion phase (which is always almost the same code executed innumerable times that transfers control to selected kernel apis) and set break point directly on win32.sys

which contains lot of hackme babies api ill post a sample


ddentry11 is primarily called from DD_SURFACE_BLT in a loop till all the surfaces are shown and



i dont know why it should crash
and your offsets are different from mine
whats your os ? what sp ?

if you dont have a kernel debugger you could try making a dump (my computer->properties->advanced->startup and some option--> select the option )
after crashing system will write a dump which you can analyze with local
windbg or probably even with inbuilt ntsd (ntsd ships by default in os greater than w2k) and possibly find some clues

Sounds to me like you've got a conflict between your video driver and some other application.

In the past, I've solved problems like that by uninstalling every driver in Device Manager, rebooting, and letting windows redetect them.

What happens sometimes is that drivers that need to share an IRQ get into a conflict. That can cause Windows to crash, or not even start. By redetecting them all, Windows can decide which one goes where for the IRQ sharing. It's worked for me several times.

When I say uninstall all the drivers, I mean that all I had left in device manager was the name of my computer and nothing else: no drives, no mouse, cdroms, system drivers or anything. I know it sounds kinda harsh, but it works.

I've even gone into HKLM\System\CurrentControlSet\Enum and wiped out the entire Enum key. Although Microsoft calls it's install system a Wizard, I prefer to call it what it is, a Dummy. There is no Wizardry in what it does, it just reads INF files and looks at what was there before. If you clean house in Device manager, it will look up your INF files for each installed piece of hardware and reinstall it. If you've lost an INF file, it will ask for the installation disk or app.

Of course, I can be a bit reckless, so don't take my word as gospel. There can be problems if you have loaded apps with their own special installers, like Intel drivers for their motherboard. Or, in a situation like I have with my Creative Audigy I that I'm running off Audigy IV drivers. You'll have to reinstall those by hand.

I hear so many people talking about reinstalling Windows. I've never found that to be necessary. If push comes to shove, just do a repair install, but be careful with it. One wrong move and you lose eveything. If it's your DirectX install that's screwed, a repair install should fix it.

I would advise you to research it first, however, by Googling.

I'd reinstall DirectX itself (get the ~30Mb redistributable from M$, I've found the "web installer" sometimes gets confused) - latest version I believe is 9.0c.

Hey...no need to kick a guy when he's down.

I don't want to hijack this guys thread, but could you say a few words about windbg, which might be of assistance to him too? You didn't point out in your reply that it's free, in the Microsoft Debugging Tools for Windows package.

I haven't tried it yet because I'm under the impression it requires two monitors. Is that correct, or am I off on a tangent?

BTW...I pass SYSENTER regularly these days, in softice, on my way in to visit Hal and the Krnl.

hehe waxford
no point posting stuff which wont register in readers mind it was just a generic comment i dont think he would be getting into kernel debugging so fast

there is a bias against windbg even though it has been freely available
right from its inception the user base for it in no source debugging world is very very less or lets say nil

the contributing factors being it requires two computers (no not two monitors but two real machines (one that runs target can be a dumb garaged machine if you really want to test the iron of real hardware ) or a virtual machine running inside a real machine connected via \\.pipe\debugPipe if you are not debugging real hardware

2) its almost horrible gui (goddamn unusable idiotic interface)

3) its queer commandline syntax (albeit damn powerful capabilities exist in those horrible bp foo!SomeCrapFunction ":q:ab;c:d:e:bs:crap; g "
to run a script you need to type ><$$ "file.scr" now wtf is that >< cant it be really made to some understandable way (please understand im just ranting about it once you get the feel of it it is absolutely amazing to peek into every corner of the operating system )

4) almost no mouse interaction (it sucks too much in front of ollydbg if you are only using it for user level debugging without source)

5) its hardcore dependancy on ddk for creating any of extensions
not every one out there will download or order a free dvd (you have to pay shipping costs ) and then learn again another horrible commandline build
process

6) almost cryptic (encrypted with rsa 9192 or higher version )documentation to start with

but inspite of these putoffs it underneath is a very good and capable debugger

if you want to take a dive and dont mind mashing your brain for few initial weeks and have a determination like a bloodhound or lone wolf following a trail of scent i would suggest you get a virtual machine (again there is a marked preferance for vmware over microsofts virtual pc (free and slim just 18.2 mb nice and easy to use trade mark microsoft point click and go software that hasnt crashed even once for me inspite of me doing all kind of terrible crashes )


as to softice i cant comment on it much becuase i have hardly used it

with hindsight i can say softice could be really usefull in a few situations where it performs by hook or crook other than those situations a legit no hooks method employed by windbg that in no way alters the system to me sound a better method

also it is a known fact that getting it to run on newer os is a real pain
and compuware has discontinued it
possibly in a few years time if no other single machine kernel debugger evolves windbg is going to rule the world with its warts in full force
so better get used to it rather than clinging to some legacy unsupported product

I'm back. This kind of problem where dxdiag crashes is 99.9% a driver issue. Don't necessarily update your drivers to the latest then assume that will fix the problem. DXDiag is an incredibly robust app - it has to be.

Back about 4 years ago the 29.xx revisions of NVidia dets were full of bugs. They repeatedly caused random crashes. The solution wasn't to update, it was to roll back to 28.xx. The moral of the story is, keep an eye on the developer forums and the gamer forums.