I am posting this thread after some private conversation with dELTA and Zairon about the issue.

Goes like this: The new version of VMware includes as a feature a virtual debugger, in which you can, from the host side of the machine, debug modules running inside the virtual side of the machine.

Needless to say such combination of tools could be exceedingly powerful. dELTA called a Ring -1 debugger: Total access to the state of the virtual CPU, indefinite number of hardware-type breakpoints in read-write and execution, The ability to go back in time if the code gets out of control, an unexpected exception happens, wrong decision is taken. Also, the Debugger modules are outside the VM in a quasi undetectable/god like status with full control of the VM virtual CPU. almost like a Hardware Level Debugger!

The bad news is that the feature is implemented with two rather specific, code IDE level debuggers: Visual Studio 5 and Eclipse.


it would be interested to see if the interface with the debugger is that of a general purpose debugger API calls which would allow a more RCE oriented debugger such as Olly to control the VM virtual debugger.

My RFC the board: Is any one here familiar with the VM debugger interface?

why such a limited choice of debuggers?
Any VMware employee here with some inside information?

I was talking with one member at arteam forum about extending boch's to perform similar stuf(it has plugin support if I remember ocrrect). Was just idea which obviously never was coded, nor I'm sure if it could be done.

Based on http://www.vmware.com/pdf/ws6_manual.pdf, it sounds like the "virtual debugger" is just a stub that sits between the host and guest that uses the normal Windows debugging APIs and sends and receive notifications between guest and host via a virtual network.

Example:
1. Host sets "virtual breakpoint" in guest.
2. Under the covers, "virtual debugger" host communicates over virtual network to "virtual debugger" guest the address of breakpoint.
3. When breakpoint is hit in guest, "virtual debugger" guest communicates over virtual network to "virtual debugger" host to say, "breakpoint hit."

I don't see anything here that implies anything like a "Ring -1 debugger" or unlimited HW breakpoints, etc.

Some big AV companies prolly have their finger in this, already with general purpose debuggers. I expect in a year we (we being high school students with too much time on our hands), will have similar tools in our hands.

You'd be surprised at what kinds of technologies the AV companies don't have...

Just to make things clear, I didn't really say that the new VMware virtual debugger was such a ring -1 debugger, I just said that if anyone, they would probably be the most likely to be able to create such a powerful beast, which would be the wet dream of every reverser...

I really hope they do though, it could be really popular, appreciated and get a lot of attention I think, not to mention being the natural heir of Softice...

I apologize if I put words in dELTA's mouth. Neither he or I had dug deeply into the VMware virtual debugger implementation, so it was more a hope than a statement.

Ah well, since we're all apologizing.. it looks like I misread the original post, so sorry for putting words in your mouth or making it seem like you were claiming something that you weren't.

It probably wouldn't take more than a few days of fiddling with Bochs to get it to do what you want. The real question is, how easy would it be to use such functionality while still keeping it hidden from the guest?

Well, if properly implemented (i.e. purely from the "outside" of the virtual hardware, never touching or changing its internal state in a way that can be sampled from the inside), practically only timing tests in the target would work, and even that could be easily circumvented by the virtual debugger, since it could then just really "stop the time" each time a breakpoint is hit etc, and simulate real instruction time increases when single stepping (because it has total control of the clock in the virtual hardware too).

It could truly be made completely undetectable. All efforts of protectionists would most likely go into detecting if its executing in a virtual machine at all to begin with (like current VMware detection methods etc today), but once those checks are defeated (which is mostly quite easy), the target would be a sitting duck.

As mentioned above, there is practically no end to the debugger stealth possibilities when you have total control of all the hardware, it would be even better than a real hardware debugger, due to the tricks that can easily be done with virtual hardware that cannot as easily be done with its real counterparts!

Not to mention if it were in something that was open source, you could write plug-ins to do specific things, and have a truly programmable environment. That's a SERIOUS advantage.

Imagine being able to break on XOR EBX, EBX anywhere in the program without having to search for it.

I worked on a Z-80 target once using a virtual machine running on the PC, and did all kinds of crazy things with my homebrew debugger.

You can't do that in a virtual machine (aside from single-stepping each instruction, which is HORRIBLY slow and could be detected by the guest OS); you need an emulator for that (like Bochs).

first, the most interesting, IMHO:

qemu -s -p 12345 ...

This starts qemu listening for debugging connections (on port 12345). The protocol is gdb's, so:

gdb
(gdb) target remote localhost:12345
Remote debugging using localhost:12345

now you are inside gdb, debugging the guest operating system running inside qemu.

I used this to debug the boot process of GRUB for example.

Now, remember that you'll be seeing the hole computer/memory, so you'll single step through kernel unless you do something more specific.

qemu (without the acceleration module) is an emulator, so I bet it's quite possible to add extra capabilities like unlimited hardware breakpoints, breaking on a specific opcode, etc... Quite likely the gdb interface will not support this, but an extra tcp port for extended debugging iface wouldn't harm qemu.

Of course, you can use ddd or gvd or others for graphical interface. And of course, you can use a windows gdb version and debug a non ia32 guest (quemu is more than ia32)

I'm very interested in this debugging techniques, so I'll really appreciate what you can share!

On a second note. As vmware (and non emulators) truly run code, I'm not sure how possibly it is to have infinite hardware breakpoints (more than unmapping memory pages, what can also be done in a real system).

gera

I found it:

http://stackframe.blogspot.com/2007/04/debugging-linux-kernels-with.html

apparently VMWare workstation 6 has the same capabilities as the ones I just described for qemu. In this post they talk about debugging the linux kernel, but it's obviously a lot more than that.

Again, please report any advances in this line, I have never tried it, so I'll be happy with any info about it.

Hey all,

I'm not so sure this would be useful through VMWare, 'cause VM ware still lets most code execute on the real processor. I think the debug thing just gives you access to that process's state, registers, etc, probably thru normal API's the VM does on your behalf.

To truly pause the whole state of the system, etc, might only be possible with an emulator.

One thing that most definitely should be possible using VMWare is to dump a process from memory without it being able to do anything about it. I've tested modifying memory in a process in VMWare successfully. I make a test EXE with a string in it, and then I ran that EXE in VMWare, and then I opened VMWare process itself with TSearch ( a memory searcher/editor) and found my string, modded it, and it worked


However, I've always found one advantage to debugging with VMWare is that you can take snapshots at any time.

-nt20

Pure genius.