Some times ago I stumbled on a post at Offensive Computing where an user had some problems trying to figure out the encryption used by a malware (md5=36401cb9178232dde01b2788e8fc56f4).
The malware contains 2 files located in the resource section. The files are encrypted, how to find out the encryption scheme? Well, using a debugger I would say. I didn’t start any debugger btw, deciding to take a look at the files using a resource editor. It’s a common thing to store files inside resource section. Here are the starting bytes of the first file:

http://zairon.files.wordpress.com/2007/10/rot13_file.jpg

If you know which are the first common bytes in an exe file you should be able to figure out yourself which kind of encryption has been used. The presence of many 0×13 bytes is a nice hint, the file has been rot13h-ed (13h stands for 0x13).

In an old blog entry (http://zairon.wordpress.com/2007/07/11/ida-plugin-extract-and-analyze/) I talked about a little ida plugin able to extract and analyze an hidden file; I slightly changed the plugin adding the possibility to un-rot13h the hidden file. Take a look at the simple dialog:

http://zairon.files.wordpress.com/2007/10/rot13_dialog.jpg

You can download the plugin from: http://www.box.net/shared/static/1kzvon1x67.zip

You mean rot-0x13'd right? 13 is 0x0d.

Yes, sorry. I have to edit the post a little, just to make things clear.
Thx for pointing it out.