Is anyone aware of tools or scripts which produce a visual display of randomness in binary files to detect packed malware?


For example like the results in this paper

A Fast Randomness Test that preserves Local Detail

http://www.virusbtn.com/pdf/conference_slides/2008/Ebringer-Sun-VB2008.pdf

Or the quick summary of the paper:

http://lukenotricks.blogspot.com/2009/03/randomness-tests-for-packed-malware.html


Or

Figure 2: Information entropy gradients of insidious GIF

http://www.mcafee.com/us/local_content/misc/sc_sep08.pdf


I found one utility that does entropy calculations, but it only spits out a score that's a little difficult to interpret:

MANDIANT Red Curtain
http://www.mandiant.com/mrc


There have been attempts at visualization of file differences or file fuzzing, but that's not quite the same:

http://computer.forensikblog.de/en/2006/02/compare_binary_files_with_nwdiff.html
https://www.openrce.org/blog/view/922/Visual_Patterns_for_File_Format_Fuzzing


Cheers,
Kayaker

Im not exactly sure if that is what you are looking for, but:
Entropy Plugin for the Hexer Hex-Editor
http://www.the-interweb.com/serendipity/?/archives/99-Sample-Hexer-Plugin-Calculating-the-entropy-of-a-file.html ("http://www.the-interweb.com/serendipity/?/archives/99-Sample-Hexer-Plugin-Calculating-the-entropy-of-a-file.html")

It is based on work by Ero Carrera
http://blog.dkbza.org/2007/05/scanning-data-for-entropy-anomalies.html ("http://blog.dkbza.org/2007/05/scanning-data-for-entropy-anomalies.html")

It might be worth checking out his python scripts

That's precisely what I was looking for, a simple graphical way of displaying file entropy. Thanks! I remember seeing that before as it turns out, I just couldn't remember where. The Hexer plugin looks particularly nice and ready-to-go.

It was actually the infected gif file reported by evaluator in the following thread, and the entropy display of a similar gif in the mcafee link above, that got me thinking about this and wanting to play around with the idea. I was wondering what the technique would reveal about the malicious gifs.

http://www.woodmann.com/forum/showthread.php?t=12764

Thanks again.

Might also try

http://sourceforge.net/projects/ciat/

Algorithms even simpler than entropy calculation can provide a much better result for id'ing malicious code. All you're really doing by looking at entropy gradients is creating a crude form of a rolling hash, which is a hash that 'forgets' data as it moves along. Visualizing the data in some way is extremely helpful though in recognizing patterns. Things get even more interesting when you start making smart hashes that recognize certain small idioms that occur in code or executable formats, such as a function prologue, and use those as start points for your hash, or weight mini-hashes that include them more heavily, etc...

I found this quite useful.

Ent -- Entropy Level and FPU Density Measurement Tool
www : http://vexillium.org
http://gynvael.coldwind.pl

Ahem...

http://www.woodmann.com/collaborative/tools/Category:Entropy_Analyzers

Search the CRCETL before asking for tools, did you not read the FAQ?!?!

Been meaning to finish this up. I tried all the suggestions, thanks. I found the Hexer plugin the most useful. Clear output and since it uses the JFreeChart library, you are able to zoom in on areas of the graph and on mouseover get the precise byte offsets of a section of the entropy chart, which you can then immediately refer back to the hex editor (after converting x-coordinate to hex).

This Java plugin is included with the latest version of the Hexer hex editor, but needs to be compiled to its class files.
Install the Java SE Development Kit (JDK) from http://java.sun.com/javase/downloads/index.jsp and update the make.bat file with the current version numbers for the JFreeChart library, following the rest of the instructions in the link given above by fuex in post#2.
To open Hexer, point your Java JRE enabled browser to the Hexer.jar file in the main directory.


There is a good explanation on interpreting the entropy results for Ent here:

http://gynvael.coldwind.pl/?id=162


Having been justifiably scolded for not searching the tool repository to end all tool repositories, my pennance was to add an entry for Hexer to the CRCETL


An interesting technique, maybe not something one would use regularly, but nice to have in the reversing repertoire.

Here's another tool of this genre, just posted on OffensiveComputing.net:

Tool for visualizing encrypted and/or packed data with special focus on PE-files. It's capable of creating histograms for the spreading of byte-codes for a whole file as well as section-wise regarding PE-files. This will make the detection of crypted and/or packed data much easier. The tool (a windows and a linux version) and a decent description is available under our CERT-homepage:

http://cert.at/downloads/software/bytehist_en.html