VMKD - Virtual Machine Kernel Debugging extensions.

Copyright (C) 2007 Ken Johnson (Skywing)

http://www.nynaeve.net
skywing_public@valhallalegends.com

---------------------------------------------------------------------


Overview

VMKD implements a set of high speed kernel debugging extensions for
virtual machines (currently only VMware is supported).  These
extensions enable a low latency, high bandwidth kernel debugger link
when used in conjunction with the DbgEng KD over named pipe support
with a local VM.

Operations such as complex conditional breakpoints, code stepping,
dump file writing, or complicated commands (such as "!process 0 0"
style process enumeration) are most likely to benefit from the extra
performance provided by VMKD.

Additionally, VMKD will eliminate the CPU spinning that occurs in a
VM that is waiting on input from the kernel debugger (if VMKD is in
use with that VM).  This is because VMKD allows the VM to sleep if
it is waiting for data from the kernel debugger, as opposed to
polling a virtual serial port in a loop.

---------------------------------------------------------------------


Requirements


- VMKD has only been tested with the following VMware releases:

  * VMware Server 1.0.3 (Windows x86 or x64 hosts).
  * VMware Server 1.0.4 (Windows x86 or x64 hosts).

  Although VMKD is designed in such a way as to maximize the chance
  of working on future versions, the author has no way of verifying
  functionality with future VMM versions.  If a breaking change is
  detected, then the vmxpatch.dll module should abort during loading
  in the vmware-vmx.exe process.  This will result in an error debug
  print being displayed if you debug the vmware-vmx.exe process during
  initialization.   The compatibility restriction is because VMKD must
  hook and call internal, non-exported functions in vmware-vmx.exe and
  depending on how vmware-vmx.exe is built, previous versions of VMKD
  may not be able to locate the necessary internal routines in the
  vmware-vmx.exe image.

- VMKD does not support Microsoft Virtual Server or other non-VMware
  virtualization solutions.  This is due to the fact that internal
  VMware interfaces are used in the program, and these interfaces do
  not exist on non-VMware platforms.  The kdvmware.sys driver should
  not be used in conjunction with a physical machine or other, third
  party virtualization solutions.  Attempting to use kdvmware.sys on
  a physical machine or a third party virtualization platform will
  likely result in the system crashing.

- VMKD has been tested with Windows XP SP2 (x86) and Windows Server
  2008 Beta 3 (x64) targets.  It should be compatible with all current
  Windows releases, however, other releases have not been verified as
  functional by the author.

- On x64 based guests, PatchGuard may bring the system down with a
  bug check 109 after several minutes if VMKD is used and the kernel 
  debugger is not attached at boot time.  This is because VMKD must
  rewrite the kernel debugger communication interface in KDCOM.DLL,
  and this module is protected by PatchGuard.  Attaching to the kernel
  at boot time with the virtual serial port, and then switching over
  to the VMKD named pipe will solve this problem.

- On Windows Server 2008 x64 and Windows Vista x64 based guests, it
  may be necessary to disable driver signature enforcement in order
  to load the kdvmware.sys driver.  This may be done at boot time,
  or by setting the "nt!g_CiEnabled" global variable (BOOLEAN) to
  zero in local kernel debug mode before attempting to start the
  kdvmware driver service.

---------------------------------------------------------------------


Manifest

vmxinject.exe     - Program to load vmxpatch.dll into vmware-vmx.exe.
vmxpatch.dll      - VMware VMM host-side hook module.
README.txt        - Program documentation.
x64\kdvmware.sys  - Kernel debugger transport guest-side hook module
                    for x64 targets running under VMware VMMs.
x86\kdvmware.sys  - Kernel debugger transport guest-side hook module
                    for x86 targets running under VMware VMMs.

---------------------------------------------------------------------


Installation

1) vmxinject.exe and vmxpatch.dll should be placed in the same directory on the VMware host.
2) The VMware guest should be configured for serial port debugging.  This is required to get
   the kernel to initialize kernel debugging support; the serial port will not be used once
   kdvmware.sys is loaded and operational.  The usual configuration options may be used for
   this step, just as if you were going to debug the guest with a virtual serial port.
3) kdvmware.sys (appropriate bitness) should be placed in the %SystemRoot%\system32\drivers
   directory on the VMware guest.
4) A driver service for kdvmware.sys should be created in the guest:
   sc create kdvmware start= demand type= kernel binPath= c:\windows\system32\drivers\kdvmware.sys DisplayName= kdvmware

---------------------------------------------------------------------


Usage

1) Power on the guest VM and note the PID of the vmware-vmx.exe process.
2) Change to the directory where vmxinject.exe and vmxpatch.dll are located in a command
   prompt, and use the command: "vmxinject vmware-vmx-pid".  If the operation is successful,
   "OK" will be printed to the console.  If an error occurs, an error message will be
   printed to the console.
3) Start the driver service in the guest VM with the command: "net start kdvmware".
4) Attach to the VMKD named pipe using the debugger.  A named pipe is created using the last
   path component of the file system location of the VM used.  For example, if a VM is present
   in "C:\VMs\myvm", a named pipe of the name "\\.\pipe\kdvmware_myvm" is created.  Note that
   the directory name of the VM may not match the name displayed for the VM in the VMware
   console.

---------------------------------------------------------------------


Troubleshooting

- VMKD replaces the internal VMware commands responsible for the "Operating system not
  found" dialog in order to communicate with the guest.  If vmxpatch.dll is not loaded
  in vmware-vmx.exe when kdvmware.sys is started in the guest, then several "Operating
  system not found" dialogs may appear in the VMware console.  These dialogs are not
  harmful and may be dismissed.  After vmxpatch.dll is properly loaded in vmware-vmx.exe,
  start the kdvmware driver service again in order to establish the connection.

- If the kdvmware driver service refuses to start (such as failing with an "invalid
  handle" or "port disconnected") error, then vmxpatch.dll is likely not correctly
  loaded in the vmware-vmx.exe instance corresponding to the target VM.  The kdvmware
  driver service will only succeed a load if it can successfully establish two-way
  communication with vmxpatch.dll on the host.

- If the debugger appears to hang while trying to connect to VMKD, try enabling KD
  protocol debugging and checking to see that you are receiving data.  This can be done
  in KD.EXE with the CTRL-D keypress, and in WINDBG.EXE with the CTRL-ALT-D keypress.
  Additionally, extensive debug prints inside vmxpatch.dll will be activated if a
  debugger is attached to vmware-vmx.exe; this may assist in troubleshooting problems.

- Rarely, the debugger has been observed to get into a state where it cannot properly
  synchronize with the target after several reboots of the guest operating system when
  using VMKD.  Restarting the debugger has typically resolved this issue in the past.
  It is not known at this time whether this is a problem with DbgEng.dll or VMKD itself.

- If nothing appears to happen when loading vmxpatch.dll into vmware-vmx.exe (that is,
  starting kdvmware in the guest appears to pop up the "OS Not Found" dialogs, but no
  vmxinject.exe is not returning any errors when run), try attaching the debugger to
  vmware-vmx.exe *before* injecting vmxpatch.dll for the first time.  There may be
  debug output that could shed light on the problem which can be captured by connecting
  a debugger to vmware-vmx.exe during the vmxpatch.dll loading process.

---------------------------------------------------------------------


Warranty, Support, and Disclaimers

This program is provided "AS-IS" and is not supported by the author.  It is experimental
software and the possibility exists that there are bugs present which may crash the
guest or even vmware-vmx.exe.  Usage of this program is not recommended in production
or mission critical environment.  The user assumes all risks from running the software.

This program was not created with the assistance or blessing of Microsoft Corporation or
VMware and relies on undocumented interfaces, and as such may not function on future
releases of Windows or VMware.

Although there is no guarantee of support offered for the program, the author would be
interested in hearing about problems (or suggestions) relating to VMKD.  Feedback may be
sent via the email address: "skywing_public@valhallalegends.com".  There is, however no
guarantee that the program author will be able to resolve difficulties or reply to all
mails.

---------------------------------------------------------------------


Acknowledgements

- vmxinject.exe injector based on remotehdl by Andreas Hansson (Adron).
- The author would like to thank (in no particular order) Andreas
  Hansson, Matt Miller, and Nate Lawson for their comments and advice
  in the development of VMKD.

---------------------------------------------------------------------


Changelog

- 1.1.1.7
  * Fixed a particularly dumb MDL misuse bug that was leading to asserts on
    checked kernels.  (There was a completely superfluous call to
    MmBuildMdlForNonPagedPool in place (whoops).  This has been rectified.)

- 1.1.1.6
  * Changed kdvmware.sys to support booting the system with a custom kernel, if
    the kernel binary is named "krnltest.exe".  This alternate name for the
    kernel is currently hardcoded and must be the name passed to /KERNEL= if you
    wish to use VMKD with a custom kernel build.

- 1.1.1.5
  * Changed vmxpatch.dll to reset KDCOM framing protocol state if the guest-side
    driver disconnects and reconnects to the host-side control code.  This is
    typically indicative of a guest-side reboot, and the change was made to more
    accurately reflect the normal behavior of KD during a reboot.

- 1.1.1.4
  * vmxinject.exe now prints "OK" to the console on success.
  * Recompiled vmxinject.exe and vmxpatch.dll to have subsystem version 5.00, which
    fixes the "not a valid Win32 application" / Access Denied errors that some users
    were receiving if the VMware host was running Windows XP or earlier.

- 1.1.1.1
  * Initial public release.

- 1.0.0.0
  * Test versions using redirected serial I/O.  Not publicly released.
