The topic of debugging full-screen Direct3D applications came up a little while ago. If you’ve ever tried it on a single-monitor setup (or even multi-monitor if the app wasn’t designed to handle it) then you’ll know how much of a pain it is. Windows just can’t handle focus being stolen from a suspended exclusive-mode program. The solution’s exactly what you’d expect - to intercept the relevant window- and device-creation calls and coax the debuggee into running in a window. This works, but fiddling with the calls manually each time you restart the process quickly gets boring. So here’s my first attempt at a generic solution.

D3DLookingGlass is a DLL which, if injected into a Direct3D process early enough, will make sure that all video devices are created in windowed mode, allowing the hosting process to coexist with a debugger without any bother. If you can inject this DLL into the target process before the first call to CreateWindow, then everything should go smoothly. I think. Any later than this and your mileage may vary.

I’ve also written a ‘loader’ program that installs the DLL as a system-wide CBT hook, so that you don’t need to inject it manually. This kind of worked for my limited set of test-cases, but there seems to be no Windows-hooks method of injecting a DLL globally and beating the call to CreateWindow. Windows installs the DLL containing the hook at the latest possible moment for its function, and I can find no type of hook that needs to be around before a window is created. I’d love for somebody to prove me wrong (or suggest another way to install the DLL system-wide), but by the looks of things, my loader is of limited use.

In particular, I recall a situation where the game (Call Of Duty 4 Demo, I think) creates a non-overlapped window, which works fine for full-screen mode, but causes problems when you try force the device to bind as windowed. This will still be a problem unless the call to CreateWindow can be intercepted (and a well-formed window induced), which means that D3DLookingGlassLoader will struggle. Confirmation would be nice.

Here’s the source: D3DLookingGlass_Source.zip ("http://www.ring3circus.com/source/D3DLookingGlass_Source.zip")

Here’s the binary: D3DLookingGlass_Binary.zip ("http://www.ring3circus.com/source/D3DLookingGlass_Binary.zip")

Here’s the small-print:

The DLL hooks CreateWindowExW and ShowWindow in its DLLMain. I think this is kosher in terms of loader-lock, but it’s obviously not too cool with regard to system stability. Especially if it’s being installed in every running process. If d3d9.dll isn’t found in the address-space then the hooks fall straight through, so that shouldn’t be too much of a problem. But if it is found then all attempts to create or show (or hide) a window will be overridden - possibly to the demise of the process if it’s doing anything but the basic behaviour. So in all cases, watch out, and make sure you aren’t running anything important in the background (in particular, I’ve noticed that it doesn’t play nice with Firefox).
The loader uses a system-wide hook, and you hate system-wide hooks. I trust that anybody who needs this tool has some degree of technical expertise and is aware of the stability concerns inherent in installing somebody else’s barely-tested system-wide hook.
This was harder to put together than I anticipated, and that’s probably evident from the slightly shabby code. Again, I intend for this only to be used for debugging purposes, so you’ll have to forgive me for the sub-production-quality code.
Despite the focus on Direct3D of this blog, I’m not really a gamer and I don’t actually have any commercial games installed on this machine. So I only got a chance to test this against my own programs. Obviously, there are several ways to skin the metaphorical Direct3D-initialisation cat, so please leave a comment when you find a game that this chokes on.

http://www.ring3circus.com/gameprogramming/d3dlookingglass-v01/

Nice!

CRCETL:
http://www.woodmann.com/collaborative/tools/D3DLookingGlass

Very interesting as your other articles about D3D, keep up the good work.

handy tool, thanks

I have also been searching such a tool, i found one called DXWnd, but i have 0% success
I tried your tool also on an app but it had no effect which is a shame since ive been trying
to get this particular one into a window, will try it on some other things when i have time :-)

The source code I am posting is for a game specific and not a general way to archive window mode to a game, so it might help or not
It has two functions:
a) make the game from full screen to window (the timer it kills I think it is because that game implements a way to terminate itself if it lose focus)
b) hide the window

[yAtEs], is there any chance that the target you're using has a freely available demo online? If so, could you PM me the details. I anticipated the tool not working 100% at first, and in retrospect I should have phrased things more as a request for testing

Actually, since forcing a D3D app into a window should not break any intellectual property rights, you are welcome to discuss the names of the test targets publicly too (as long as they are not cracked versions of commercial products to begin with).

The game is FEAR, there is a demo available online

This will be interesting, please let us know of your results with this app Admiral. And if you can't get it to work, I'm sure some of the other DirectX/reversing pros around here would be happy to join in.

Did anyone get any interesting results or findings with this?

I'm, err, still working on it
It seems that CreateWindowExW & ShowWindow are not necessary to create a window with a device-context that Direct3D can bind to. Either that, or FEAR is getting around INT3 breakpoints. If anyone has any idea what I'm missing, please speak up. Otherwise, I'll give it some real attention when I get a free hour.

Admiral

Hey Admiral, interesting project. Regarding your question, DX simply needs an available HWND and nothing more, so perhaps you need to widen your search (for example, by backtracking from the HWND the demo is using rather than going the other way).

The classic example of this is the old cube in a button trick - it's trivial to write a D3D app that puts a rotating cube on the Start button by simply grabbing the existing HWND and using that...

I've been very busy recently so haven't had time to look at your source, but one thing occurs to me:



Although it's a bit silly we can easily write an app to create a fullscreen D3DDevice without including a single CreateWindow in the same .exe that has the D3D initialisation (as per cube in the start button). D3D itself doesn't care about the HWND apart from when initialisation occurs.

So, using that theory, why not do it slightly differently and create your own window, then alter the D3DPRESENT_PARAMETERS.hDeviceWindow in the target app and set it to your own window? That way you won't have to worry about well-formed windows and you have a generic solution that will work for all apps (regardless of how those apps create their windows)...

Hey Silver. Thanks for taking an interest in the project.

I assume you're right about the HWND situation - I was working under a probably fallacious assumption that the window needs to be associated with a device-context. I'm not sure exactly when I came to this conclusion, but I vaguely recall failing, back in the day, to bind a Direct3D 7 (immediate mode) device to a hastily created window. I shall look into this a little further before trying to fix the code.

And as for the method of creating a dedicated window, I considered and discarded the option. It's simple enough to create a perfect overlapped, resizeable window, but getting it to play nice with the message pump is another story. Not only would the WndProc need to be hooked, but we'd have to go through hell to make sure that no user-code makes any references to the original window's handle and so on. At least, that's the case without playing about with some Windows internals, but we obviously want to stick to ring3.

Keep the ideas coming though and, of course, correct me if I've misunderstood something

You're right from a down-the-line logical point of view, but bear with me on this one. You're trying to bounce fullscreen D3D apps into windowed mode so that they're debugger friendly, right? But let's think about that D3D app in its original form.

If it's full screen, then we know that the GDI won't be used (best case GDI is used on an off-screen surface and blitted, which won't touch the actual "visual" window). If it's full screen we also know that standard Windows features are going to be ignored - no minimize, menu bar, messageboxes etc etc that post messages back to the parent. So I would make an educated guess and say that the only real use for the msgproc in a full screen D3D app is timer events and possibly mouse/keyboard input, and even those may well be RTDSC and DInput respectively.

So we're taking a bit of a risk, but it should be perfectly fine to create our own window, bind it to the D3DDevice and simply bounce all the timer, user, mouse and keyboard messages to the app's original hwnd. You wouldn't need to hook the Wndproc because you don't actually care about it as such, you just want to change the HWND in the presentparams.

Perhaps I'm suffering from brain-lag (been travelling a lot recently!) but I can't think of many instances where a fullscreen D3D app would need to use the *original* HWND in code. And, if I've understood your intention, you're going for a debug/reversing tool (ie: to be used whilst reversing an app) rather than a generic end-user solution to force windowed mode on D3D apps (ie: to be used by anyone at all times). In that context I think any issues would be minor.

There's something nagging at me about lost surfaces with this, though. I'm sure it'll come to mind shortly...