Hello,

Originally I've signed up here to ask a different question, but finally I've partially figured it out, just to stumble upon another problem.

I'm debugging a DOS DPMI application with SoftICE 4.05 under Windows 98, running in VirtualBox. SI works perfectly with only one glitch in the P command: stepping over functions (F10 over CALL; even empty functions are affected), or stepping over REP'ped instruction freezes SoftICE. For example, if a function has only:

push ebp
mov ebp, esp
xor eax, eax
leave
ret

when stepping over it, SI freezes. When tracing through it by F8, all is OK. After entering it, and P RET'ing with F12, it also freezes. Stepping over regular instructions works OK. Didn't check LOOP, but I suspect it might also be affected.

Breakpoints (BPX) also work without problems -- by now I've overwalked this issue with bpx'es after the CALL and resuming the application with G, so the issue isn't preventing me from my work, but it's annoying enough to ask someone for advice; especially that I remember, in the SoftICE glory days, there were no such issues at all, even with DPMI targets.

Could anyone shed some light on this?

Hi

I wonder if it's a VirtualBox problem? There is a known issue with Softice + VMWare, search this board or google for those terms if interested, but I think it boiled down to having these 2 lines in a cfg file

vmmouse.present = "FALSE"
svga.maxFullscreenRefreshTick = "5"

Perhaps there's something similar with VirtualBox, but this is the first time I've heard that problem mentioned here at least.

Thanks for the response.

I completely forgot to add the fact, that I've tried this also in QEMU, and after stepping over a function, it crashes with segmentation fault. Haven't tried VMware, it doesn't play nice with a Linux host.

I've googled the VMware/SI issue, it seems different, but it's good to know this.

I've also figured that stepping over works if I'm not debugging a DPMI target, but I enter SI f.e. by Ctrl+D. This could mean that the target may be doing something nasty with CPU, which is very probable, since it's DPMI .

OK, anyway, I thought someone had, or heard of, the issue I've described, but it not, then tough luck. Maybe some macros will ease the pain

maybe you already know this
but i thought ill still post

look for japheths debxxx a dos dpmi debugger i used it some time back for some ancient exe and it seemed to work well

Hi,

Thanks for info. I've been on his site previously, haven't tried his debxxf debuggers, but I've put it on queue if my previous approaches won't work .

First I've tried to use pure DOS and used 386SWAT to do debugging. By the way: I was pleasantly shocked when I found out that it saves 16 last debugger screen updates; so after tracing you can always go back 16 steps and see previous registry values and visible memory content, without doing expensive snapshots . I didn't manage to get it to debug, mainly because incompatibilities with the VMs I was using (Freezes, or internal error dialog boxes in VBox, qemu, Bochs under FreeDOS and DOS 6.22 - different permutations), so finally I decided to go with Win98 and SoftICE, because I already knew this setup

I will probably check debxxf in near future, to see how it performs - from curiousity - because I even somehow like commandline debuggers .

Why are you using Win98...it was prone to crashing? If I remember correctly, you need to do quite a bit of environment setup in W98, especially with relation to the 640k limit. You may need a memory manager (himem.sys and and emm extender).

I know you are debugging DPMI, but SI has to run in the 640k environment, and if your upper DOS area is not setup correctly it can give you all sorts of trouble.

I recall issues with Softice 2.8 that required a careful setup of the DOS environment.

Also, have you tried single-stepping a bit into the function then trying to step over it? How about breakpoints? Can you BP on the other side of a function and get it to break without crashing?

Is it possible there might be protection in involved? Some protections check for single-stepping.

Update...I know it may sound like I am confusing W98 with DOS, but here's some history. Back at the W98 level, Windows was called from DOS. That's right, when Windows booted, the process began in DOS, then called Windows with a 'win' instruction. You could actually boot into DOS with an F8, then type in 'win' (no quotes), at the DOS prompt, and Windows would start.

If I recall correctly, you need to have the DOS environment set up properly to run Windows correctly. You can set up the W98 environment using the win.ini and sys.ini files, which can be edited in a text editor. You may need to set up autoexec.bat and config.sys as well, where you can adjust the number of FILES and BUFFERS, as well as adding memory managers.

I need to call on Kayaker's softice expertise here, but softice loads W98 on top of it, so AFAIK, softice also depends on a clean DOS environment, since I think it installs something in UMB (DOS upper memory).

I would make sure the W98/DOS environment is set up correctly before going any further.

Thanks for your detailed input , however I have an impression you didn't entirely read the posts written by me at the beginning, when stating my original question .

I understand the difference between the DOS and Win98 , and in fact I remember the days of migration from Win95, when Win98 was bleeding edge. Countless reinstallations and determination to run old DOS real-mode games taught me how to setup both DOS and Windows. Learning assembly & reverse engineering on Win98 has shown me how SoftICE should work. Also I'm not sure why you suggest/suspect an invalid DOS environment, when my VirtualBox'ed Win98 works OK, and -- as I wrote earlier -- tracing and stepping over *works*, when not debugging my DPMI target?

As you wrote, it might be related to some software protection and while I agree, I don't think my target uses this kind of stuff, because if it would, I think it'd use more antidebug tricks than only a anti-step-over hack (but of course, you never know for sure until you check it). I remember in the old days I debugged DPMI software, but it used DOS4G/W dpmi host (WATCOM-compiled targets), and the one I'm debugging right now uses go32 extender, which is a part of djgpp compiler suite. Still, I don't have enough experience with DPMI as I would like to have, I never really took the time to learn the internals of protected-mode switching mechanisms and services offered by DPMI, so I'm sticking with my breakpoint-after-call method of evading this issue.

Update: also, back to your first question: I'm using Win98, because I didn't have success in configuring a DPMI debugger under pure DOS. My target requires pure DOS, because it's using raw I/O port access to hardware. Right now the target works under Win98 and I can analyze the initialization stuff, but I'm not sure if it'll work when more complex hardware interaction will occur. But I'll worry later

do your target also work on the dos command box in winxp ?

I though that I got your meaning clearly, but you seemed to have missed my point that W98 is quirky. It often crashed, or froze, for no apparent reason, and there's no reason to think the crashes and freezing you are experiences are not related to W98 itself, especially running in a virtual box.

If softice freezes during a function jump, it could be related to memory issues. I have experienced that directly while using it, having to page memory back in manually. I just feel uneasy running W98 in a virtual box while softice is involved. Running softice alone on a version of W98 installed as a standalone OS could cause sudden BSODs. W98 simply did not like anyone fiddling with its system level software.

I noticed a dramatic difference between the stability of softice on XP compared to 98. While tracing through Ring 0 code, W98 would often BSOD, but XP was very forgiving. 4.05 will run on XP, can you get a bare bones copy of XP to try it out, or wont it run on your virtual box?

This may seem dumb, bit is ice actually freezing, or is it just taking an awful long time to perform the F10 call? How long have you waited?

You have indicated that you are running Linux, using a virtual box with W98, running softice. I have seen older versions of softice take a long time to perform a function. With the set up you have, there may be something causing long delays.

Elenil, unfortunately target doesn't run on WinXP. If it would, I'd just probably use Olly

WaxfordSqueers, to be honest I've only waited for maybe maximum of 10 sec, so it might be a long delay. I'm going to check that . Update: I left SI running after F10 on a CALL for few hours, and it didn't unfreeze, so I assume it's freezing completely, not just delaying work.