dELTA
April 21st, 2004, 11:02
I recently installed Softice (DriverStudio 3.1) inside a VMWare (4.0.0.4460) machine. While doing this, I experienced some problems, and I thought it would be a good idea to document these problems and solutions here, since I found that they had not earlier been discussed either on this board or the exetools board.
First of all, when installing Softice to a standard configured VMWare machine, it will hang the system as soon as you try to start Softice from the start menu. This problem is described in the following article in the VMWare support knowledge base:
http://www.vmware.com/support/kb/enduser/std_adp.php?p_faqid=965&p_created=1049834747&p_sid=l6hHm4Mg
and also in the following article in the Compuware support knowledge base (which in turn refers to the VMWare article):
http://frontline.compuware.com/nashua/kb/doc/1702.asp
For documentation reasons, I also quote it here:
The first time I tried this solution I had two problems:
First, I added those two lines to the end of the VMWare virtual machine config file (.vmx), but VMWare just kept deleting them! This happened also when VMWare was completely deactivated (and also did not have any active snapshots or suspension states) while I performed the edit of the file. As soon as I started the virtual machine (i.e. not directly when starting the VMWare program) those lines were deleted from the the config file and consequently ignored in the virtual machine.
For some reason, I found the following method to work to get those lines to stay:
1.
Install Softice in the virtual machine.
2.
Reboot the virtual machine.
3.
Pause the virtual machine after logging into it and being at the desktop of it.
4.
Perform a "save snapshot" in VMWare for the virtual machine.
5.
Add the two patch lines to the end of the virtual machine config file (.vmx):
----------------
vmmouse.present = FALSE
svga.maxFullscreenRefreshTick = 5
----------------
6.
Start the virtual machine again (with the resume/play button).
7.
The mouse should now act weird inside the virtual machine (you cannot click anything with it, although it moves, and it will not escape the virtual machine screen without pressing the ctrl+alt hotkey).
8.
Press the ctrl+alt hotkey to get the mouse out of the VMWare window, and click the "Reset" button for the virtual machine, it will now reboot.
9.
The lines should now still remain in the config file, and be used in the booted virtual machine.
10.
Start Softice from the start menu of the virtual machine. It should not hang the system anymore (nothing will display on the screen except a DOS-windows appearing and disappearing though).
Ok, now comes the second strange part. As described in the quoted VMWare support article text above, "Sometimes, even with these configuration options, pressing Ctrl-D does not display the SoftICE window and your mouse and screen appear unresponsive". At this point it is only the Softice window that is invisible for some reason (i.e. you can press ctrl-d again to get back to the system, or you can type "X" and then press enter too to get back, the Softice window is there and you can interact with it, but for some reason it's just invisible).
I have no idea why this happens to some people, and even more strange, it first happened to me too. I then tried the solution described above with the DOS window, and it worked, but it is not very desirable I guess, since when a breakpoint hits when you are not in DOS VGA mode you will be in trouble with the invisible window I would think (I have not confirmed this though).
Even funnier was that after almost giving up trying to get it working without the DOS window (I tried both the default universal video driver mode, and other video modes too, without any luck), I accidently reverted my virtual machine to a snapshot I made just before installing Softice (i.e. DriverStudio 3.1) into the virtual machine (this feature in VMWare kinda sucks btw, the revert button is just beside the tabs from which you choose your virtual machines, and there is not warning when you accidently press that button, it just reverts your virtual machine right away to the last snapshot, possibly causing you to lose tons of work, so be very careful with your aiming when clicking on those tabs!). Anyway, I then re-installed Softice and now all of a sudden it worked perfectly, the Softice window shows up nicely on the screen as soon as I press ctrl-d, and can be used successfully (using all default settings in Softice, both regarding video mode and otherwise, nothing was altered from the defaults). This means that this problem does not have anything directly to do with the virtual machine operating system itself, and neither with the VMWare version or something like that (since it was the exact same virtual machine used, just reverted to the point just before Softice was installed), but rather something else, seemingly random. I guess the VMWare/Compuware guys don't really now either, since they don't give any more info about it in their help either, except for that quite unsatisfactory workaround with the DOS VGA mode. Btw, I'm running my virtual machine in 800*600 resolution, and have VMWare Tools installed and running.
I hope this will be a help to at least someone, and it would be great if people could collect their own experiences, problems and solutions with VMWare + Softice in this thread too, to make it a really good resource for archival reasons.
dELTA
First of all, when installing Softice to a standard configured VMWare machine, it will hang the system as soon as you try to start Softice from the start menu. This problem is described in the following article in the VMWare support knowledge base:
http://www.vmware.com/support/kb/enduser/std_adp.php?p_faqid=965&p_created=1049834747&p_sid=l6hHm4Mg
and also in the following article in the Compuware support knowledge base (which in turn refers to the VMWare article):
http://frontline.compuware.com/nashua/kb/doc/1702.asp
For documentation reasons, I also quote it here:
Quote:
| Q: When I using the SoftICE debugger in a VMWare virtual machine, the guest operating system hangs. How can I work around this problem? A: If you plan to use SoftICE in a virtual machine, open the configuration file for the virtual machine in a text editor and add the following two lines: vmmouse.present = FALSE svga.maxFullscreenRefreshTick = 5 Otherwise, your mouse and screen may become unresponsive when breaking into the SoftICE debugger, making it appear as if your guest operating system has hung. svga.maxFullscreenRefreshTick can be set to any value greater than 0. Lowering the value of svga.maxFullscreenRefreshTick increases the responsiveness of SoftICE, but it also has a negative impact on the performance of the guest operating system when SoftICE is not running. We recommend removing or commenting out these configuration options when you are not using SoftICE. Sometimes, even with these configuration options, pressing Ctrl-D does not display the SoftICE window and your mouse and screen appear unresponsive. Exit SoftICE by pressing Ctrl-D again, then use SoftICE in VGA mode as explained below. Open a command prompt (for instance, choose Start > Run, type cmd and click OK). Press Alt-Enter to enter VGA mode. Press Ctrl-D to enter SoftICE. |
The first time I tried this solution I had two problems:
First, I added those two lines to the end of the VMWare virtual machine config file (.vmx), but VMWare just kept deleting them! This happened also when VMWare was completely deactivated (and also did not have any active snapshots or suspension states) while I performed the edit of the file. As soon as I started the virtual machine (i.e. not directly when starting the VMWare program) those lines were deleted from the the config file and consequently ignored in the virtual machine.
For some reason, I found the following method to work to get those lines to stay:
1.
Install Softice in the virtual machine.
2.
Reboot the virtual machine.
3.
Pause the virtual machine after logging into it and being at the desktop of it.
4.
Perform a "save snapshot" in VMWare for the virtual machine.
5.
Add the two patch lines to the end of the virtual machine config file (.vmx):
----------------
vmmouse.present = FALSE
svga.maxFullscreenRefreshTick = 5
----------------
6.
Start the virtual machine again (with the resume/play button).
7.
The mouse should now act weird inside the virtual machine (you cannot click anything with it, although it moves, and it will not escape the virtual machine screen without pressing the ctrl+alt hotkey).
8.
Press the ctrl+alt hotkey to get the mouse out of the VMWare window, and click the "Reset" button for the virtual machine, it will now reboot.
9.
The lines should now still remain in the config file, and be used in the booted virtual machine.
10.
Start Softice from the start menu of the virtual machine. It should not hang the system anymore (nothing will display on the screen except a DOS-windows appearing and disappearing though).
Ok, now comes the second strange part. As described in the quoted VMWare support article text above, "Sometimes, even with these configuration options, pressing Ctrl-D does not display the SoftICE window and your mouse and screen appear unresponsive". At this point it is only the Softice window that is invisible for some reason (i.e. you can press ctrl-d again to get back to the system, or you can type "X" and then press enter too to get back, the Softice window is there and you can interact with it, but for some reason it's just invisible).
I have no idea why this happens to some people, and even more strange, it first happened to me too. I then tried the solution described above with the DOS window, and it worked, but it is not very desirable I guess, since when a breakpoint hits when you are not in DOS VGA mode you will be in trouble with the invisible window I would think (I have not confirmed this though).
Even funnier was that after almost giving up trying to get it working without the DOS window (I tried both the default universal video driver mode, and other video modes too, without any luck), I accidently reverted my virtual machine to a snapshot I made just before installing Softice (i.e. DriverStudio 3.1) into the virtual machine (this feature in VMWare kinda sucks btw, the revert button is just beside the tabs from which you choose your virtual machines, and there is not warning when you accidently press that button, it just reverts your virtual machine right away to the last snapshot, possibly causing you to lose tons of work, so be very careful with your aiming when clicking on those tabs!). Anyway, I then re-installed Softice and now all of a sudden it worked perfectly, the Softice window shows up nicely on the screen as soon as I press ctrl-d, and can be used successfully (using all default settings in Softice, both regarding video mode and otherwise, nothing was altered from the defaults). This means that this problem does not have anything directly to do with the virtual machine operating system itself, and neither with the VMWare version or something like that (since it was the exact same virtual machine used, just reverted to the point just before Softice was installed), but rather something else, seemingly random. I guess the VMWare/Compuware guys don't really now either, since they don't give any more info about it in their help either, except for that quite unsatisfactory workaround with the DOS VGA mode. Btw, I'm running my virtual machine in 800*600 resolution, and have VMWare Tools installed and running.
I hope this will be a help to at least someone, and it would be great if people could collect their own experiences, problems and solutions with VMWare + Softice in this thread too, to make it a really good resource for archival reasons.
dELTA
But knowing vBull, a search is only as good as the last time the search index was updated. This thread was from February 24, 2004 and maybe Aaron hasn't updated the indexs since then. As I know from personal experience on this Forum, it does take a good long while on the version 3 series software to update them all. I can't think of anything else that might cause it. They are actually running the "final" 3.0.0 version. I have also noticed that vBull has released a 3.0.1 "bug fix" edition, which I'll download soon for installation here. 
