Hello all. I have an idea for a new Windows project, one that I expect has been attempted before, although I could find no trace of it on the net. Ideally, this is what I would like to do:

I want to be able to record a program's path of execution, storing register values and memory read/writes. The prog would record only changed values to conserve space. Ideally, this might be facilitated through a callback function on each trace command. This recording would then be superimposed upon a dead listing for viewing and analysis later (or maybe into an IDA plug-in).
The "debugger" part of the program would not need any user-interface at all. A command line parameter could be used to designate the conditions for the initial breakpoint. You would then process exclusions (e.g. ignore calls to kernal, etc) and a conditional endpoint (or max bytes recorded) through a callback function (or maybe a script file).

I am not nearly as knowledgable at low-level opsys concepts as I would like to be, but I am studying pretty hard at it. I was hoping that any of you might be able to point me to some good references of study regarding low-level debugging, or maybe point out any problems that you might see with my idea. Maybe there is a much easier way to do this?
I think that I could have wriiten this without too much trouble using the Win32 Debug API, but alas, my first target is a 16-bit Windows program. Apparently, these 16-bit apps own the Win16Mutex, so the Debug API will not let you suspend their execution. I guess that leaves me with the debug interrupts.

Thanks for your time,

Bardiche

Hello Bardiche !

I don't know your coding skills, but you've choosen a very hard kind of program you would like to create. Can you tell us why you want to create such an application, because it would be easier to give you some hints after that. (i.e. unpacker, cheater, bug finder, patcher, ...) The only thing I can point you to at the moment is ICEDump. It comes with full source code and a build-in tracer which does not need user interaction.

Thanks for the reply DakienDX.

My target is a Win16 app, as I mentioned. My aim is to create a utility that would enable additional processing of the Win16 app's data files. The data is encrypted, but I have already obtained the decryption algo via winice. The problem lies in the complexity of the data's encoding. (The encoding has proven to be a far better detourent than the crypto!) You see, the data consists of many different object structures, packed using different algos for each different object type. As if this were not complex enough, when creating such a data file, a very specific sequence must be followed to write the objects, and again the sequence depends on the object (or set of objects) being written. Note: The encoding was not implemented as a protection scheme, but as the programer's idea of the best way to handle the data.

Of course it would be possible to obtain all of the info I need to master this decoding using winice and a good dead listing, but it seems like a horrable use of my time. I figured that writting the Trace app would take quite a bit of time also, but at least I would learn some nice things along the way!

Thanks for your recommendation of IceDump. I hadn't looked at that prog in a long time, and it turns out that it has a command "TRACE" that does just about exactly what I wish to do, except that it does not work on 16-bit code. Reading the doc on it, they give some hints as to what may need to be done to allow it to work with 16-bit code ("i.e. no V86 mode support, mainly due to some lazyness in emulating 16 bit instructions". Looking at the Trace source, I also noticed that it uses the SUSPEND functions, which also only work with 32-bit code, so that would also need to be handled.

IceDump may still be my best bet though, but I will certainly need to do some more research on V86 mode, emulating 16-bit instructions, and suspending 16-bit threads. I thought that I was a pretty decent searcher, but I am still having some trouble finding good references to writing trace code for Win16.

Thanks for your help,

Bardiche

Do not, at *ANY* point in time, underestimate the amount of code lines that you can get with this program. It will be TOO MUCH for you to handle.

Want a taste of things to come? Without loading softice and all??

Do this:

1. Download ollydbg
2. Run a trace with record (with <TRACE OVER SYSTEM DLLS> option ON!)
3. Run your app
4. See the lines of trace.
5. More than 40,000+ in a single APP!!

You want to analyze that??

But yes, your motives for these are noble indeed. Just remember, it is definitely possible, but of no practical use UNLESS you wait for the new version of OLLYDBG which will actually COMPARE traces !!

And NO, this post was *NOT* an ad for Ollydbg

...Have Phun

Hey Aimless, thanks for the info. You raise a very valid point in the potential for huge amounts of output data. I have considered this, and it is that very reason why I think that a callback function is needed in the trace. You could use it to execute algorithms for excluding data that is already known.
The code that you would put in your callback function could be complex indeed. Algorithms like those used in some processors to pre-process code would be very helpful, allowing you to recognize and record only the data needed to manipulate future code. A simple example might be a loop that XORs every element in an array. Only the orginal array need be known, not all of the data created by the loop procedure. Writing code to do these things would end up being extremely complex, I'm sure, but it *would* be an interesting excercize.
But as you've said, even taking steps to return the minimum amount of data could still leave you with a huge dump. I have taken my targ's IDA listing and converted it into an Access database. It has turned out to be very helpful in analyzing portions of code, using SQL and custom VBA algos. (Not that it doesn't have it's drawbacks...ever try to do an update query in Access on a table with 600,000+ records?) I think that a TRACE dump would work very well with this data.
I had already taken a cursory look at ollydbg, well, long enough to see that it does not work with my targ's 16-bit code. Pity though, because it does look like an interesting program.

Bardiche