Hi all,

I'm having fun learning/writing assembler on *BSD. It's been a while since I did so under Windows.

After reading some of the anti-hack guidelines on the site, I'm trying to come up with a plan for writing some spaghetti code to protect an app as an exercise.

Since hand-coding ML can be so tedious and error-prone, does anyone have any ideas on how to plan out a spaghetti code routine so it works, doesn't crash, and creates confusion for anyone trying to follow it?

I'm also interested in any advice for self-modifying code.

You would need to study the x86 Instruction Set carefully and have a good knowledge of which bytes are what instructions. As you have said "creates confusion for anyone trying to follow it", you would probably have a bit of difficulty planning it too. But draw a "flow diagram", showing at which addresses are placed what bytes (since I assume you'll want to overlap instructions, you cannot work on an instruction-by-instruction basis). This is the most easiest way to write such code, as doing it in an assembler would be even more difficult. The crucial part is the flow diagram.

As for self-modifying code, inspect virus source codes for some good examples. Writing a (seemingly) infinite loop and then "breaking out" of it by dissolving its jmp into some other instruction (not necessarily a NOP) comes to mind. A good self-modifying code should overlap instructions and re-use the operands of other instructions as opcodes themselves. Planning smc should be done with diagrams that indicate the "state" of the code (as it is changing continously), or you will soon be lost as to what the instructions are/were.

When you say overlapping instructions, you mean each one using the result of the previous, right? Not actually overlapping one instruction and args with another, right?

Yes, overlapping the actual instruction opcodes and args, it's a known and good trick to confuse some disassemblers or even debuggers. Anyway, you want to do a Google/board search for code obfuscation...

I tracked down some info in CrackZ's essays. Some of the suggestions involved using bit shifts and rotates, especially the ones using the carry flag, because it makes it harder to breakpoint and trace.

Has anyone seen any good examples of this I could learn from?

When you mentioned bit shifts it reminded me of this paper, part of which explains interesting opcode tricks:

Polymorphic Viruses - Implementation, Detection, and Protection
http://vx.netlux.org/lib/ayt01.html


Implementation of Polymorphism on the Intel 80x86

In almost every case we have examined, the polymorphic engine exploits the fact that certain computations can be performed using different registers and instructions. To step through encrypted portion of code, for example, one can use DI, SI, or BX registers. To increment or to decrement the index value, one can ADD to the index register, INC it, or use an implicit instruction that increments it (CMPSB is used in TPE for example).

Polymorphic engines can also rely on the availability of instructions that are coded using the same opcodes. On the 80x86, there are 11 opcodes used for several different instructions: 80, 81, 83, D0, D1, D2, D3, F6, F7, FE, and FF. When it is necessary to encode information about one operand, the middle three bits of the ModRM byte are used to distinguish operations. The ModRM byte follows some opcodes and can either extend the opcode or contain information about the operand and addressing modes of an instruction. It contains three fields: Mod (2 bits), Reg (3 bits), and R/M (3 bits). For example, for the opcode 80, the middle three bits of the ModRM byte, which make up the Reg field, have the following meanings:

ADD 000
OR 001
ADC 010
SBB 011
AND 100
SUB 101
XOR 110
CMP 111

The second operand of each instruction with an opcode of 80 is an immediate byte, so the ModRM fields in the second byte of machine code encode the first register or memory operand.

By flipping a few bits, it is possible to generate code achieving many different operations easily. Combined with a rich set of addressing modes, and a good random number generator, one can create very different-looking decryptors.
---------


Kayaker

Hi,

Another thought, you could consider resolving some of your SMC to floating point instructions, you can use the FPU for obfuscated comparisons and other operations as well.

For example, this will push 2 values onto the FPU stack, subtract them, and compare the result in ST0 with 0. The result of the comparison is set in the condition codes field of the 16-bit FPU Status Word register. You can transfer the result to AX and the FLAGS register and jump based on the zero flag. You could also use comparison instructions such as FCOMPP, FXAM, etc.



The FPU could also be used to store a value, say at the start of the SMC, by pushing it deep onto the FPU stack - it could later be popped off into another variable when needed. Keeping in mind any other FPU instructions which might alter the stack.
Curious thought - what API's might use the FPU for calculations?(GDI comes to mind), and more importantly, are the contents always preserved, i.e. FPU pushes = pops?


Another stupid pet trick: I actually used this in some graphics code to randomize a display. I suppose it could also be used to add a little polymorphic flavor to tracing SMC. Say you had 2 sections of (SMC) code which did exactly the same thing but which were composed of different opcodes (see previous post), a simple random ON/OFF generator could be added with this perhaps. Half the time (randomly) the code would call one or the other routines. Though the execution results are the same, it does add a {1,2,...n} dose of confuse-a-cracker to the mix...



Cheers,
Kayaker

I'm trying to decide where to spend most of my time: encrypting the code or encrypting the key. The Petri Net is quite cool, but still comes down to a single branch instruction at some point. It seems to make it not worth tracing the registration/key checking code, but once registered, it's still either go or no-go.

The SMC sounds good, and reminds me of the old C64/6510 undocumented opcodes tricks. ;-)

I just remembered: one of the weaknesses I am trying to avoid is where the cracker can simply break after each decryption, read the decrypted code, set the next breakpoint, and keep going.

Would it make any sense to incorporated RDTSC into the code? You could check for long delays between certain instructions (which would indicate single-stepping) and use that to foul up the decryption, while also could use the code itself as part of the key so it cannot be changed easily to remove the check for the timestamp.

right click find all commands rdtsc
right click set log break point on all commands
pause to always
when paused send the following commands to plugin

.a eip,db 0x90 0x90
.run

or same ways find cmp eax. blah .set eax =0 .run
or find sub [esp],eax .set [esp] = eax .run

rdtscs gone to hell

I once heard the MMX instruction set has a wealth of possibilties..


regards..

wizzard

I always think Virtual machines are fun to implement to calculate some algorithm's - people spend hours debugging your virtual machine and wondering the hell its doing rather than actually reversing your algorithm

By virtual machine I think you mean more toward bytecode interpreter - a small routine which "executes" the bytecode (use a lodsb and then a switch table arrangement for the best performance) and then the bytecode itself, which will indeed confuse less experienced reverse-engineers. This is the technique used on many of the 4096-byte "intros" (very fascinating several minutes long graphical animation effects often with sound as well) which contain almost the highest code density possible. Their bytecode interpreter also makes use of self-modifying code and tricks with the stack (for example, push an entry of a switchtable onto the stack and then execute a ret). Although this is more of a size optimisation than purposeful obfuscation, the code is still quite difficult to dicipher.