Hi,

I'm working on a project to try to learn more about how file encryption works. I'm using a program (FlipAlbum 7) to burn an encrypted CD containing some jpeg files. For those unfamiliar with FlipAlbum, when you burn a CD containing photos it also writes a small program to the disc that is used for viewing the photos on another computer, without the user having direct access to the jpegs. So after I burn the disc, I rip the files from it so that I have the .exe that launches when you insert the disc, some dll's, and a folder containing all the encrypted jpeg's. What I am attempting to do is open the .exe in OllyDbg, find the code that retrieves the encrypted photo, and then find the algorithm that the program uses to decrypt the files in order to view them.

So my question is, am I correct in assuming that the program actually decrypts the file in order to view it? I would love to be able to extract the algorithm that it uses to decrypt the files, and see how it works step-by-step. It's all fun and games, but I need to know if it's actually possible. What do you guys think, am I on the right track or am I missing something totally important about the way these programs work?

Follow the input. It likely calls GetWindowText/GetDlgItemText/etc. to get the decryption key/password when you supply it. Set breakpoints on those APIs and see what it does with your input to trace the code to the decryption algorithm.
PEiD's KANAL plugin might help as well.

A small bit of advice here, albeit on a tangent..

If you've just started learning reversing, a largish program with an encryption algo is probably NOT the right start at all. I'd suggest that you get familiar with Olly and a few basic concepts, solve a few very simple crackme's and then hit this. Lena's tutorials are a fantastic start. All IMO.

If you're ok with all that and understand what Olly is doing and why..then you need to think...

-- When will the program decrypt the files? Maybe when you view them..?
-- How can you view them? By clicking on some menu option somewhere?
-- What actually happens when you click 'View pictures'? Some functions called somewhere..which display that pic?
-- What functions could these be? Mostly specific to that particular language?
-- What language is the software written in? What API's in THAT language are used to display images? Google? MSDN? .. for these APIs?
-- Find out if these APIs are used.. Ctrl+N in Olly. Set Breakpoints on all references of this API.
-- Now click to view the image...see which breakpoints are triggered... one of them WILL be code to decrypt and hence display the pic

Hope this helps. If you feel all the above went over your head...I strongly suggest you take up a small, easy program, learn a little more and then hit this.

Good luck!

Arvind

Thanks for your replies. In response to disavowed, the user never actually supplies the encryption key to the program; rather, it fetches it from a text file saved to the disc.

I set a breakpoint on the ReadFile API, hoping that I could catch it reading the encryption key file by watching for the filepath in the stack, but no such luck. It gets called many times - probably hundreds - when the program is first opened, and I can see it reading the jpeg's but not the key file. Perhaps if I follow it more closely after it reads a picture file I can find out what exactly it's doing with it. Ideally I'd like to reverse the algorithm so that I can produce fully functional jpeg's from the encrypted files.

Here's a question: is there any way in OllyDbg to determine when a specific file is read by the program? Instead of stepping through hundreds of ReadFile API's and waiting for the right file to come up, can I just have the program break when the right file is accessed? This would be very helpful for another project I'm working on too.

EDIT: Just reread your question. You might want to inject some code into ReadFile that checks filename/filepath, and triggers an int 3 when it reaches it.

Good luck.

You should be able to set a conditional breakpoint, not sure of the exact Olly syntax but something like

bpx ReadFile if [ESP+4]==hFile
where hFile is the file handle retrieved from a previous bp on the appropriate CreateFile.

bpx CreateFile if [ESP+4]=="path/filename"


Sysinternals Process Monitor or a good API monitor might also help to sort out the CreateFile/ReadFile calls.