Thus, a highly useful tool right from the MS DirectX SDK for e.g. finding the cause of a rendering problem: for any captured frame, you can click through the executed DX API functions and see how the frame is being built up, eventually finding out what part is to blame.

http://files.uploadffs.com/c/5636cc39/pix.png

But what about reversing a closed source application's renderer? PIX does not store a call stack; it merely logs *what* DX functions are called, but not from *where*. Therefore it is not very useful for reversing by default.

I didn't want to let such a great tool go to waste. After some reversing work I ended up patching PIX to log and show (part of) the call stack for each DirectX call that the target program makes. Each call stack entry has both the virtual address and the module name.

http://files.uploadffs.com/5/5e3e56be/callstack.png

Example usage of the resulting modified tool is finding out about and messing with a game's renderer, or more simply locating the HUD rendering code and quickly finding the data that it represents (e.g. health, money) rather than having to resort to memory scanning.

Fullsize screenshot: http://files.uploadffs.com/0/54880c80/callstack.png
Download: http://www.mediafire.com/file/nhtzmjdi0z2/pix-callstack.rar

I'm not sure what you're saying. Are you looking for tools to reverse DX apps? Softice works if you move the DX app into a box from fullscreen, so it behaves. Silver is the expert on that. Look up some of his stuff on DX.

You probably know that a DX app is like any other app up to the point where the DirectX function is called which initializes the directx process. From there on the mouse cursor is controlled by the DX process, so you can't use any normal windows functions or messaging. Besides that, DX apps take over full screen and don't like relinquishing control (BSODs). I got around that, with judicious amounts of help from Silver, by coming in the back door through the mouse driver in ring 0, using softice. Once you get into the DX code it's like any other reversing, with access to all your DX processes.

Silver has at least one DX crackme. Although DX takes over the mouse cursor, it can't take over the mouse per se. The actual hardware is connected through Windows and that is controlled by the mouse driver. If you can tap into that, which requires some ring 0 work, you can get access to any DX stuff.

First of all, I'm not looking for anything (anymore) - if you had read the rest of my post, you would have seen that I answered my own question by creating this patch. And now I'm offering it to the public. This is not a question thread, but a tool offering thread

The point is that having a list of DX calls that the program does to render a frame, and being able to see what each one changes on screen, is easier than taking guesses and placing breakpoints that might be hit hundreds of times in a frame. Take the healthbar example: this could be a polygon rendered using DrawIndexedPrimitive, but that function is called for pretty much every single object in the scene - so with a regular debugger and just a breakpoint on this DX function, you'll spend quite some time to find the game's code that draws the bar (and subsequently, find the memory address/class offset where the health value is stored).

Concerning full-screen and mouse cursor stuff:
- PIX works by injecting a .dll into the target process and capturing a frame into a log file when a hotkey is pressed. Later, you exit the target and view the log.
- Generally it's possible to make DirectX programs run in window mode anyway


Looking at it again my original post was a bit confusing though. I edited it and added inline screenshots.

Great tool. Better then adding a bunch of DirectX hooks manually for suspect interfaces, etc.

Had to comment on the IMHO non-sequitur argument by "WaxfordSqueers" detracting your thread.

When thinking of understanding how a process is using DirectX, honestly who thinks of SoftIce?
If you use SoftIce as your main debugger, or for some reason wanted to do stuff in kernel mode
(to hide from anti-hack systems?) then understandable, but otherwise..
The processes interface to the DX API is userland.
IMHO it's much easier to look at this stuff from there. Having the return addresses like he's added really
helps with RE'ing stuff. You can pinpoint where things are happening in the code.

And for that matter, why couldn't you use a userland debugger like OllyDBG?

What you, or Sifu "Silver", needs to learn is to force DX to start in windowed mode.
Use a loader w/DLL and hook the interfaces "->CreateDevice()" (or what ever depending on the DX version).
In theory you can't do this with any app (depending on how it restores lost devices/surfaces, etc.) but I've been able to force
every application I've ever tried (to go windowed mode).
Probably because a lot of developers cut and paste DX samples, and, or, have dev code in that still supports windowed mode et al.
Some times your stuck with a bare window in the corner at "0,0", but I've hooked CreateWindow/Ex() some times to make the applications main window look better, add a minimize button, etc.

This applies to DirectInput too (since you mention mouse problem).
You can hook the "Create", polling, and or event (to inject inputs), etc., to do what you want there.
I recently wanted an application to respond to fake/spoofed keyboard inputs while it was minimized or in background.
I hooked the interface "SetCooperativeLevel()" to it set "DISCL_BACKGROUND | DISCL_NONEXCLUSIVE" for the keyboard device.
(Incidentally using a tool like this you could find the return address to the code and patch it there instead of hooking).

If you can't or just don't want to do this, you can probably use a dual monitor setup instead.
I've used this in a dev setup (I work/ed in game dev), should work in a hack/RE/debug setup.
Game window on the primary, your tools (this, debuggers, etc.) on a second monitor.
This way you should be able to work on a full screen DirectX window in real time.

Consider such a tool has usage beyond (gasp) games. Other applications Poker Clients, Educational tools, monitor/control programs, etc.

And don't be confused by anti-hack systems and reversing.
Meaning, think of the ideal of reversing then think of what you might have to do to either circumvent, bypass, or otherwise hide behind anti-hack protections and detection systems. The lengths of which that you might have to do, that don't necessarily apply to normal reversing, etc.

Like I said, I did not understand your question. I did not know if you were asking about other apps like softice with relationship to DX. If your tool does the job, all the better.

From what I recall about DX, which isn't a lot, the code is written like a script. The code also flows regularly from one frame mechanism to another and there isn't a lot the programmer can do to affect that. The thing to remember is that DX came from Microsoft and it's a special engine on its own. As far as I know, the programmer can only affect it in limited ways.

If you stepped into the code with softice, you could watch one frame being formed. I found ways to step through repetative code by letting certain sections run themselves to completion. I have never been into just setting breakpoints and hoping.

The way the post was worded it sounded like the poster might have been asking a question rather than offering advice on a tool. He admitted in a later post that his wording was ambiguous. I took the post as a question asking if other tools worked on DX, so I offered my experience with softice.

Maybe someone who's into reverse engineering?

I use only softice and it's all I have ever used. I have not turned to Olly because it can't go into ring 0. I have nothing against anyone using Olly and I can't think of any reason why someone would be opposed to using an excellent app like softice. It's becoming an exclusive club, most people can't even get it going in XP.

Real mode is DOS and DX does not work in DOS mode. Softice will not only get you directly to the code in question, you can follow the DX engine through kernel mode. In fact, with the DX nms files loaded, I can get the same information presented on the softice screen as the PIX app gives. I'm all for using whatever tools are available. If PIX helps in making RE more intuitive, great. My reply had nothing to do with the validity of the app.

When I did Silver's DX crackme, I traced the app from the code entry point using softice. It went through the typical windows registration up to UpdateWindow, but that only brought up a windowed frame. Then the code diverted into the DX code, where the engine was initialized. To get through the first part of that code, you had to changed the code to put the DX app in a window, but then the DX switched cursors to DX mode and none of the windows messages or functions could do anything. You could set BPs on anything you wanted, but none of them would take.

That's why Olly is useless for DX. The only way in was to BP on a mouse driver function and follow it back through ring 0 code to the DX code. Once in the DX code, you could set BP's on the DX functions, and with softice and the nms files, they were all there in plain English.

no...what you need to do is learn some respect. Silver is a good member on this board and an expert in DX. He's a decent person who is very helpful with people. Before you go insulting people, maybe you should find out who you're insulting.

Correction (and edit) I meant "user space" aka "userland", or R3..

I intend not to start a flame war, but rather enlighten.
Anything else PM please, we already noised up the guy's poor thread enough.

Very nice tool.

CRCETL:
http://www.woodmann.com/collaborative/tools/PIX_with_callstack_patch