This is sort of a general question.

For starters, packing and unpacking are not my favorite theme in RCE, so my question may seem trivial for more advanced board users.

If I am correct, the usual approach to packed apps is to allow the protection stub to do the unpacking for you, then recover the unpacked image of the file form memory. My question is, how feasible it is to reverse the packing/unpacking algorithm itself?
There is a small, finite number of packing methods. Would it be another way to defeat this sort of protection?

Exactly what do you mean with "reverse the unpacking algorithm"? You mean like rip it and make a separate unpacker tool that works on static executables? This is quite possible indeed. The problem is that the packer algorithms are often changed quite frequently to prevent such a thing (at least in smaller details, enought to make an old unpacker tool not work anymore).

Then we have of course all kinds of tricks that work more or less in "real-time", and hence are patched into the code of the unpacked executable image in one way or another (plus API emulation tricks of different sorts). These are often designed to be hard to defeat in the static executable, and are designed so that the program should not work without the packer wrapper present in memory. These are the biggest problems with fully static unpacking I think.

not to mention it involves rewriting the entire unpacking stub yourself / updating it on every version change of the protection. (_time_ consuming, particularly to maintain that code, but then this also gives you _total_ understanding of the protection)

Letting the unpacker do these initials steps is a much more generic method and is a lot faster.

I am confused as to what he means also, but I would suggest reading this, as it gives a general overview of the implementation, theory, and methods of packing and protecting.

Taken from anticrack.de before it shut down for the conversion.

Written by BigBoote from nakedcrew.net