Here's a bit of info. I've been able to map out some of the main interface in VMware, which occurs through numerous DeviceIOControl calls between vmware-vmx.exe and vmx86.sys.
What I did first was to use the debug version of vmware-vmx provided. Replace the file
"C:\Program Files\VMware\VMware Workstation\bin\vmware-vmx.exe"
with the version in the \bin-debug\ directory.
Then delete the vmware.log file in your My Virtual Machines directory for the OS you plan on starting.
In the log will be lots of debug output (there is also a small bit of info through regular DbgPrint messages when VMware starts up).
Code:
: vcpu-0| VTHREAD thread 4 "vcpu-0" host id 484 started
: vmx| VTHREAD create thread 4 "vcpu-0"
: vcpu-0| MX: init lock: rank(offline)=89
: vcpu-0| MX: init lock: rank(monitorPoll)=65534
: vmx| DnD rpc already set to 1
: vcpu-0| APIC: Setting up interrupts
: vcpu-0| APIC: version = 0x14, max LVT = 5
: vcpu-0| APIC: LDR = 0x1000000, DFR = 0xffffffff
: vcpu-0| APIC: spurious interrupt reg = 0x11f
: vcpu-0| APIC: thermal interrupt reg = 0x10000
: vcpu-0| APIC: performance counter reg = 0xfe
: vcpu-0| APIC: timer interrupt reg = 0x300fd
: vcpu-0| APIC: local interrupt 0 reg = 0x1001f
: vcpu-0| APIC: local interrupt 1 reg = 0x184ff
: vcpu-0| APIC: local error reg = 0xe3
: vcpu-0| IOAPIC: Setting up interrupts
: vcpu-0| IOAPIC: IRQ 0x2 -> 0xff, timerReg=0x106ff
: vcpu-0| IOAPIC: IRQ 0x8 -> 0xd1, rtcReg=0x8d1
: vcpu-0| IOAPIC: NT/Win2000 using the CMOS clock for timer interrupts
I decided to explore the vcpu-0 thread closer. The Id 484 given is the ThreadID (1E4h). The THREAD command in Softice displays the stack backtrace going through DeviceIoControlFile. What's interesting is that the stack ends on KiUnexpectedInterrupt.
Code:
:thread -x 1e4
Extended Thread Info for thread 1E4
KTEB: 819EDDA0 TID: 1E4 Process: vmware-vmx(5EC)
Start EIP: KERNEL32!GetModuleFileNameA+0179 (77E88785)
Registers: ESI=819EDDA0 EDI=819EDE0C EBX=813587EC EBP=B79C7B54
Restart : EIP=8046A499 a.k.a. ntoskrnl!KiUnexpectedInterrupt+029F
FrameEBP RetEIP Syms Symbol
B79C7B54 EB5C24ED N ntoskrnl!KiUnexpectedInterrupt+029F
B79C7B78 EB5C4D47 N vmx86!.text+222D
B79C7B98 EB5C07E7 N vmx86!.text+4A87
B79C7BFC EB5C0649 N vmx86!.text+0527
B79C7C2C 804B14CC N vmx86!.text+0389
B79C7D00 804A96A0 N ntoskrnl!NtWriteFile+3F74
B79C7D34 80465D49 N ntoskrnl!NtDeviceIoControlFile+0028
0326FF3C 00562433 Y ntdll!ZwDeviceIoControlFile+000B
0326FF64 00445DB4 N vmware-vmx!.text+00161433
0326FF8C 0043A6A5 N vmware-vmx!.text+00044DB4
0326FF9C 0055DD9A N vmware-vmx!.text+000396A5
0326FFB4 77E887DD N vmware-vmx!.text+0015CD9A
0326FFEC 00000000 Y KERNEL32!GetModuleFileNameA+01D1
From the offsets given for vmware-vmx, I was able to determine the exact
DeviceIOControl call in question and the IOCTL code:
push 81013F67h ; dwIoControlCode
push eax ; hDevice
call ds

eviceIoControl
test eax, eax
...
push offset aIoctl_vmx86_ru ; "IOCTL_VMX86_RUN_VM failed: %s\n"
Now we need to find out where the corresponding dwIoControlCode is handled in the driver vmx86.sys. All roads lead to IRP_MJ_DEVICE_CONTROL of course, and in this case a 2-step jump table calculation:
Code:
:0001064E 55 push ebp
:0001064F 8B EC mov ebp, esp
:00010651 83 EC 48 sub esp, 48h
:00010654 8B 4D 24 mov ecx, [ebp+IOCTL_Code] ; 81013f67h
// dwIoControlCode 81013f67h
:00010657 53 push ebx
:00010658 8B 5D 28 mov ebx, [ebp+returnBuffer]
:0001065B 56 push esi
:0001065C 57 push edi
:0001065D 8B 7D 0C mov edi, [ebp+arg_4]
:00010660 33 D2 xor edx, edx
:00010662 8D 81 B8 C0 FE 7E lea eax, [ecx+7EFEC0B8h] ; 1F
// This instruction yields a calculation in eax:
// (81013f67h + 7EFEC0B8h] = 1F
:00010668 89 13 mov [ebx], edx
:0001066A 89 53 04 mov [ebx+4], edx
:0001066D 8B 77 0C mov esi, [edi+0Ch]
:00010670 3D AB 00 00 00 cmp eax, 0ABh
:00010675 89 75 28 mov [ebp+returnBuffer], esi
:00010678 0F 87 4D 03 00 00 ja IOCTL_not_supported
:0001067E 0F B6 80 8B 0D 01 00 movzx eax, ds:JumpIndex[eax] ; 07
// A hash table returns the jump index
// JumpIndex[1Fh] returns 07
// in this example corresponding to IOCTL_VMX86_RUN_VM
:00010685 FF 24 85 F3 0C 01 00 jmp ds:IOCTL_JumpTable[eax*4]
So from this can be identified all the IOCTL calls of vmx86.sys. One of them is IOCTL_VMX86_INIT, and it is in this one where the lone SIDT instruction in the driver is used. This may be where the host's IDT is first read in.
If you set a BPX on the IOCTL handling code you can monitor the various IOCTL calls coming in.
In tracing IOCTL_VMX86_RUN_VM I was able to reach a page fault handler of some sort, use of SGDT, SLDT, -> indirect call -> SGDT, SIDT, CR3 -> major crash probably from manually tracing too far...
I think fully naming the IOCTL calls through the JumpTable will be useful, time for an idc script I guess..
Kayaker