A frequent request here is for both introductory and detailed information on setting up and using a safe malware analysis environment. I've created a sticky thread where I hope we can gather as many good articles as possible that touch on that subject.

Please add any noteworthy articles you find or are aware of that can help guide those of us who are interested in secure reversing of insecure targets.

The best of the articles will find its way into a larger knowledge resource that is currently being set up, so anything you can add will be a contribution to something far grander and permanent than this thread.


To start with, here are a few that have been mentioned before in the forums:


Capture, care and analysis of Malware made easy
http://www.linklogger.com/vm_capture.htm

Practical Malware Analysis
http://www.blackhat.com/presentations/bh-dc-07/Kendall_McMillan/Presentation/bh-dc-07-Kendall_McMillan.pdf


Setting up Windbg/VMWare:

Remote Debugging using VMWare
http://www.catch22.net/tuts/vmware

Driver Debugging with WinDbg and VMWare
http://silverstr.ufies.org/lotr0/windbg-vmware.html


Cheers,
Kayaker

I think any mal analyzer should know pros & cons of various tools for setting up an analysis environment in the first place;

http://www.sans.org/reading_room/whitepapers/threats/malware_analysis_environment_design_and_artitecture_1841

Interesting article: an automated malware analysis environment (already linked up somewhere on the forum I'm sure).
http://cert.at/downloads/papers/mass_malware_analysis_en.html

How Detect virtual machines :

www.symantec.com/avcenter/reference/Virtual_Machine_Threats.pdf

For malware analysis a good combination could be Sandboxie + Buster Sandbox Analyzer

Sandboxie: http://www.sandboxie.com

Buster Sandbox Analyzer: http://bsa.sandboxie.info

As soon as I finish coding next BSA feature probably I will write a paper about setting up a malware analysis environment.

Sandboxie is not a good one , because the new malwares can detect sandboxie.

I told Sandboxie + Buster Sandbox Analyzer.

http://bsa.sandboxie.info/frameb.htm

Another one from CERT.at





source: https://www.openrce.org/forums/posts/1279

As usual it uses a Linux distribution (Ubuntu) to do the work.

I always wonder the same and itīs one of the reasons of why I coded Buster Sandbox Analyzer: Why to analyze Windows malware Linux is used?

Why are required complicated installations?

Apart it uses VirtualBox, so lots of malwares will detect the virtual machine and will not work properly.

Whatīs the point of doing this project under Linux if you use VirtualBox, something already available under Windows?

Hmm, maybe in order to prevent malware that is capable of breaking out of the virtual machine from infecting the host?
Just an idea.

Could be the reason but I donīt think so.

Anyway there are solutions to prevent anything breaks out of the virtual machine.

e.g.: if Iīm not wrong Sandboxie is able to sandbox VirtualBox.

That's exactly *the* reason!


Hm, a few things that need to be mentioned:

*) As already mentioned above, usually another OS is used for the base (and therefore not necessarily Linux), just in the case of an escape. In the case of Minibis (which is CERT.at's implementation of a concept posted earlier) there has to be a place that is declared to be save. That's because of all the monitoring-data there which has to stay "trusted" at least with adequate efforts.

*) Regarding VirtualBox I have to say that it is one of the least detected VM-solutions, though, to be more precisely, as it's built upon QEMU it's more of an emulator than a typical virtualization.
Anyway, Minibis is primarely used for mass malware analysis, so a few samples that won't execute do not mess up the results of let's say 25k samples.
Furthermore, nowadays malware is changing its characteristics regarding VM-detection. Actually the trend of VM-detection is massively falling down. That's because of the fact that virtualization became daily business for productive machines. As malware's major goal is to run to make money that is just a logical implication.

*) For anyone whose paranoia-mode is in god-mode: There's also a possibility to use native machines instead of virtual machines, that's usually done with data-recovery cards, forensic writeblockers and automated re-imaging.

Hope that brought some light into the discussion.

Cheers,
Chrisu.

If Iīm not wrong Sandboxie is able to sandbox VirtualBox so in the rare case of an escape from the virtual machine the malware would land inside sandbox folder.

Other solutions like Deep Freeze, Returnil, Shadow Defender, ... would make the work too.

It really depends on what your goal is. Just as a prevention-layer for emailing and surfing I agree with you. But if comes to professional behavioral analysis of malware those sandboxes are just the wrong tool. That is because of the differences between sandboxes, VMs and emulators regarding their characteristics.
When you do behavioral malware analysis you *want* the underlying OS to get infected, furthermore (besides VM-detection, I already mentioned that above) you want to have "normal" looking, full OS, so in other words, any thing on the system that is additional might change your monitoring results.
Another thing is the fundamental diffrence in the approach of sandboxes and VMs: You can compare it with "white"- and "black"-listing. Sandboxes do blacklisting - they try to slap an executable on any thinkable way it could act evil. VMs won't let the guest do anything until you give it the ability.
There are just too many drawbacks sandboxes have in comparison to VMs when it comes to professional behavorial malware research, but as I said, there are definitely use-cases for sandboxes, though.
As for me, I would prefer to bring my own trusted oxygen with me when I enter a room with unknown bacterias and viruses than breathing the air in that room through a filter ... but that's just my opinion.

Cheers, Chrisu.

I agree with you, Chrisu, I only disagree in one thing: if you are going to use VirtualBox for malware analysis, why to use it under Linux if VirtualBox for Windows exists?

The reason you give is "to prevent malware that is capable of breaking out of the virtual machine from infecting the host".

There are several methods to prevent such situation, so using Linux is like killing flies with cannons.



You can declare a place to be save under Windows with a minimum effort.

I understand your veto on that. Actually it's (nearly) perfectly fine if you use Windows as the host-system. The fundamental concept behind Minibis furthermore gives you free decision in all of it's components, so, also in this. However, having said this I still would recommend to have another OS as host (might also be Mac OS), just massively reduce the risk that if something escapes that it will find its native environment again. You can take the following scenario as an example:

Usually you have networking capabilities in your malware lab environment. That's a must as you want to know what the sample is trying to do over the net. Under the forensical aspect to not unnecessarily add software to the samples playing field you usually position you network-monitor (sniffer) outside of it. So if you have the typical scenario of a host and a guest (in VM-jargon) your sniffer would sit on the host. The network in that scenario is definitely the place of the highest risk for anything to escape. So if you analyse samples like Conficker (and so on) which try to spread themselves over network you're in a very bad situation if you have Windows as host system, especially if it's an unknown sample that might have some rpc-zeroday-exploit-payload onboard. In the worst case it would infect your host, which wouldn't even be the fault of the VM-solution.

To stay on this rpc scenario: that's also a point for sandboxes. Sandboxes do hook (and so on) more or less of the OS' functionalities to hinder a process from attacking un-sandboxed components including inter-process-communication (rpc, pipes, mailslots, ...). But the network communication is usually not "influenced" as sandboxes' primarily goal is to have a prophylaxe against infections during surfing and emailing, in other words, during internet-communications. But, all this techniques (rpc, pipes, mailslots) I mentioned above are done via "cable". So it's easy to circumvent the sandbox by implementing them directly in the malware and not calling the according OS-functionalities which are controlled by the sandbox.

So, as a ... résumé: If you're about to lock up an alien from planet "Nitro" and you know that it needs pure nitrogen for breathing wouldn't it be smarter to surround its cell with pure oxygen that nitrogen?

Cheers,
chrisu.

Tell me when was the last time a malware was able to escape from VirtualBox and infect the host, please.



I donīt understand how is possible a malware can not spread over network directly from VirtualBox but it can escape from the VM and do it from the host. Thatīs just amazing!!! I want to see that!!!

Could you provide an example of a program able to do that?

I understand you want to defend your decission of using Linux, but I think your arguments are weak if not completely out of the reality.

I asked how you avoid that malwares detect the VM and abort execution. Your reply was:



Why donīt you apply the same criteria to the malwares able to escape from VirtualBox and infect the host? You just need an disk image solution. In the very very rare case a malware escapes you just need to recover from the image.

With all these argues I just pretend that people building malware analysis tools consider seriously using Windows as platform for them. The security of the host can not be the excuse to build them under Linux.

Buster Sandbox Analyzer, apart of the ultra expensive Norman Sandbox Analyzer, is the only malware analysis solution for Windows users. I would like there are other alternatives.

As my automated lab is built up as I sketched it before, I had none. If my host was Windows-based and not up-to-date at that moment (which is not of relevance in this example as it's only a question of "Is it possible?" Conficker would have nailed it.


Misunderstanding on your side or I explained it to chaotic. If the latter's the point, sorry for that. Firstly, a malware lab should not be connected to the real internet (exception do exist, though). Secondly, malware *can* spread over network from VirtualBox, but under a controlled and secured environment it won't find a native (same OS) door on the others in the network, which is only my host.


I think I already did - but keep in mind, my scenario doesn't make use of what is usually understood regarding "VM-escape".


Not at all, just thought there where a complete misunderstanding why Linux can make sense in case of automated malware analysis.


Hm, I'm sorry but that's my daily reality as a malware-analyst and reverse-engineer. And regarding this I have to take all counter measures that nothing escapes and/or ruines my work. And trust me, I've seen a lot crazy things ;-)


And this is totally true. Regarding the initial goal of Minibis it was planned to analyze thousands of samples. That's why a "few" won't effect the statistically result in an extraordinary way.


You're merging up things. In the first case you have a few samples not running as usual and therefore bringing up a handful misleading results. In the second you have a sample (or samples) escaping its cell eventually attacking my "save place" and ruining my work. These are completely different things.


Again merged up: For my "save place" a backup image would delete all of my progress. For the "playing fields" in the case of VMs technique you can say that you're already recovering an "image" by the revertion of the guest.


Wrong and write, just mixed up:
YES, for manual analysis and reverse-engineering you will choose Windows - of course - and Windows-based tools and recover afterwards from a clean image, a data-recoverycard, and so on.
NO, for automated scenarios you need some software to play "your" role and that must not be infected in any way.


I'm sure it's a good tool - I never wanted to offend it or you. Actually, Minibis is more a ... framework, which you can use according to your imagination. So, we're talking 'bout oranges and apples ;-)

Cheers,
Chrisu.

Something pretty simple to do under Windows so there is no point doing automated malware analysis under Linux just in the name of security.

I can setup such scenario under Windows any day of the week with a minumum resource impact over the system and fully secure.

Thatīs my point to critice Linux based malware analysis tools.

One question: Did you ever try configuring such environment under Windows or you simply didnīt try it?

Other question:



Whatīs the point of having a network if Windows malwares can not find a Windows OS?

Whatīs the difference with not having a network at all?

You need a network to monitor the regarding activity, and furthermore give the sample the impression that it gets what it wants. The latter is usually more the case for manual analysis as you're focusing really in details on the specific sample.

Besides that a lot of malware would die or switch to standby if no network is available.

Cheers,
Chrisu.

But itīs a network where VirtualBox computer is alone, no other Windows OS can be found, so itīs like if you run VirtualBox under Windows and you got the network configured but you donīt have connected any other computer. The result is the same, so again, whatīs the point of using Linux for security reasons if the solution under Windows is as simple as not having any other computer connected to the network?

btw... you forgot to reply a question: Did you ever try configuring a secure environment under Windows or you simply didnīt try it?

If read my post again ... I have 2 (logical) computers in the net, the guest and the host.


Didn't know that this is an interview? ;-)
Anyway, what's the point on that? Securing Windows bit by bit (as I said, like Blacklisting) though there's a solution that has the characteristics of Whitelisting? I'm sure that Windows can be secured, and btw., I'm primarely a Windows guy (otherwise I wouldn't have the knowledge how to reverse engineer Windows malware, don't you think). In the concrete scenario we're talking about the easiest, most stable and forensically nearly "authentic" way is how Minibis is set up. To have it forensically perfectly acceptable you could even replace the guest by a real physical, native Windows box. But for automatism the "host" would stay.

Solution: You can run two instances of VirtualBox so you got also 2 (logical) computers in the net.

This is not an interview. This is a thread about setting up a malware analysis environment and seems like here we are the only two persons that did some deep research about the topic. So I consider very interesting for us and for the rest of users if we share our experience and our thoughts about the issue.

Iīm sorry if I make feel you are in a Gestapo interrogation but I consider the question-reply method as a good way to get concret replies about interesting topics. I felt you would not tell me if you tried to get a secure environment under Windows so I had to ask it.

Do you think a Minibis port for Windows could be released?

LOOOL ... no problem


It definitely could. If I will? ... I've got to think through this if it's worth the effort. Please do not misunderstand this, but just for one person ... hm. You have to understand that I'm actually in a very hot phase of development regarding a new disassembler/code analyzer. But, we'll see.

Anyway, if you like to learn more 'bout Minibis I can recommend reading the following things:
*) Mass Malware Analysis: A Do-It-Yourself Kit (http://cert.at/downloads/papers/mass_malware_analysis_en.html)
*) My according article in HITB eZine (https://www.hackinthebox.org/misc/HITB-Ezine-Issue-002.pdf)
*) Minibis' website (http://cert.at/downloads/software/minibis_en.html)

Cheers,
Chrisu.

Do you really consider Minibis has more potential users if itīs released using Linux as host OS than Windows? I would have to disagree about that.

Thanks for the links and for sharing your thoughts with me!

Well, there may be an additional reason why quite a number of these malware analysis tools are running on Linux - they are coming out of an academical environment. Windows is NOT a very important OS in universities. For instance at my faculty (computer science) there are only two(2!) comps in the Computer Graphics lab running Windows (and I've never seen them being switched on), although we are a member of MSDN-AA. Every other lab is running Linux or BSD or Solaris. As far as I know, it's the same in many other unversities around here. I must admit, I know nothing about the situation in the US, maybe it's totally different there.
Again, that's just an idea, but maybe it's related.

Best regards
darkelf

Darkelf: interesting point of view!

We could discuss about next questions:

When a malware analysis tool is developed, for who is created? Is it created for personal use (even if itīs shared publicly) or for other persons?

If itīs for other persons, in theory what are the kind of persons that we can consider in the scope for the usage of the tool? (excuse my poor english)

chrisu: Do you prefer that Minibis is used by as many users as possible or you prefer that is used by less users but with a more advanced profile, like letīs say IT professionals, system admins, etc?

I can talk about my experience with Buster Sandbox Analyzer.

I didnīt have the need of a malware analysis tool for personal use because as advanced user in Windows computer security I didnīt need something like that. But even if I didnīt have the need for it I wished I had the opportunity to try one when I want.

I didnīt like that under Windows there was only one option and a very expensive one, so with the help of Sandboxie, as I had the coding skills and the experience required to develop such tool I did it.

So I made BSA for other people, not for myself, and I did it for Windows because it didnīt exist such tool publicly available.

I try to approach the malware analysis to normal users. Users that usually use just an antivirus. Itīs not an easy task because most of Windows users are used to the "install-and-forget" security solutions.

Resuming: BSA is a malware analysis tool for the masses.

Minibis is definitely NOT for the masses. Its constituency are malware-researchers, certs, antivirus-companies and let's say the "advanced" user that already knows how to manually analyze malware.
So, to be precisely, Minibis is a very flexible and customizable framework to automate the manual activities of a researcher if he or she needs to analyze thousands of samples i.e. to produce a database for statistical statements, to identify trends, and so on.
BUT, it's also usable as an initial quick-check in case of a new sample.

That's what Minibis is - not more - not less.

The future will also bring some new possibilities that haven't been around in this way. According to this the actual characteristic (host/guest) is a fundamental must to Minibis. But please understand, that I cannot tell more about this yet, I'm still in proof of concept - I just wanted to give another explanation for my underlying concept.

Cheers,
Chrisu.

Minibis could be seen as not for the masses because it was designed to analyze thousands of malwares but the same it can analyze 25k samples it can analyze 1.



Do malware-researchers really need to analyze thousands of malware samples? I donīt think so. They usually analyze malware samples one by one and manually, using decompilers like IDA or debuggers like OllyDbg.

Do advancer users have 25k malware samples? Donīt think so, but even if they do... do they need to analyze them? Again, I donīt think so. I donīt see a reason for that.

Being realistic mass malware analysis tools are intended for antivirus companies that need to filter between the big amount of files they receive to discard between harmless and potentially dangerous files. Checking all they get one by one would be impossible nowadays.

If an antivirus-company must do mass malware analysis, on what option will they rely? Probably in their own solution or on a professional solution like Norman Sandbox Analyzer.

So I think a good question is: Is there a "market" for public malware analyzers? It exists but itīs very very little.

Then who will be using public malware analyzers? Mainly advanced users, not malware researchers because they donīt need that neither antivirus companies because they will use or their own solution or a professional one.

So in my opinion the scope of the publicly available malware analysis tools (mass analyzer or not) are the advanced users.

I will not comment about CERTs because I donīt really know if they process big amounts of samples or they mainly work with honeypots.

How do most of the advanced users prefer to make malware analysis? Probably using online malware analyzers like Anubis, ThreatExpert, JoeBox, etc. Why? I think because they are afraid of possible infections so they are safe using online tools.

From the advanced users that donīt mind hosting a malware analyzer, what do they prefer: a Linux or a Windows based malware analyzer tool? Windows, of course, because they want to check if a program is trustable to later install it in their system. Having to make the analysis under Linux to analyze a Windows application is not practical for them.

For all the above reasons is why I think malware analysis tools must be hosted under Windows. The few persons (letīs be realistic, probably just the 1 or 2% of computer users use them) that will use that kind of tools work with Windows.

Correct!


It depends on what's needed. Usually a researcher initially doesn't look into the code - that's what Ida, Olly, etc. are for. First you just want to know in a quick way with what you're dealing with. Then, if it's necessary or relevant you might take a look at the code.
Btw., IDA is no decompiler, though hexrays are selling an decompiler-plugin for their Disassembler/Debugger IDA Pro. So, don't merge this up.


I wouldn't want to tell the advanced user that is really interested in my tool what to do.


That's just ONE scenario where mass analysis can make sense. There are way more than this. How do you think are Malware trends identified. How do you think statistical data is produced for list i.e. the top-ten of Windows autostart-possibilities used by malware-authors. How do you think it's possible to find out eventually other malware that seem to be created by the same developer, or the same frameworks/tools. And so on - there are really a lot.


That really depends on too much influences.


There is a market - that's it's big I never mentioned.


Mainly CERTs - that's why I made it public. It's a common approach in the CERT community to share instrumentation.


No, see above.


I'm from a national and government CERT, I guess I know what our branch is doing. ;-)
And to answer your question: They do, one more the other less, that depends on many things.


That's correct for the normal advanced users. But for CERTs and AV-vendors from time to time, too, there are periodicly scenarios where nothing is allowed to become public - so, no Anubis and so.


I don't really care a lot regarding this, as they are not my main-constituency. I just decided to let also the public (not CERT or researcher) guy participate in my work.


You're still merging up two different things. Instruments for fast analysis of lots of samples; and indepth code-analysis (mainly) on Windows PE files (executables).

Cheers,
Chrisu.

I meant disassembler not decompiler, sorry.

Then we must difference between independent malware-researchers and malware-researchers working for antivirus-companies.

The independent malware-researcher doesnīt need the quick way to check with what heīs dealing. Most of the time he will work on samples already known to be malware.

The malware-researcher working for an antivirus-company will receive samples already filtered by the own malware analyzer tool or the third part professional tool.



Who does that work? Antivirus-companies.

What tools are being used to do such work? Internal tools or professional ones like the ones developed by Zynamics ("http://www.zynamics.com/").



How do CERTs get samples?



Iīm talking about malware analysis tools producing results with human intervention or automatically and independently if they can process lots of samples or only one at a time. And now Iīm discussing if itīs better to build them under Linux or Windows depending of the people that may use them.

With the indepth code-analysis (mainly) on Windows PE files (executables) I meant that the people doing that work doesnīt need of tools like Minibis or BSA. That people work on samples already filtered.

Adding..

5 Steps to Building a Malware Analysis Toolkit Using Free Tools

http://zeltser.com/malware-analysis-toolkit/

Doubtful, as Zynamics's VxClass doesn't scale to real-world scenarios.

Some time ago they wrote me telling they were working on scalating to match real-world scenarios. Donīt know how much they have advanced on the task since then.

I haven't seen a live demo since VxClass came out, so it's entirely possible that it does scale to millions of samples now. Not sure how often Halvar reads this forum, but would be nice to get some input from him on this.

Ah, cool, thanks for notifying me.

When we speak about scalability, we have to look at two angles: Processing the stream of incoming samples, and processing the set of legacy samples accumulated over time.

I will write a bit about stream processing here:

A rough measure of how many files you need to process per day is roughly 40k or so (measured by MD5sum).

When it comes to processing legacy samples, were quickly speaking millions of files, but I will talk about this later.

So, core point is: We have spent the last year distributing VxClass, and we now regularly run a compute cluster where we process approx 1k executables per compute-node per day -- right now we run an 8-machine cluster. That puts us at processing 20% (8000 files) of the malware that needs to be processed on a (computing) budget of roughly 800 USD / month. Scaling further would be no problem from our end, but the database server that we're using tends to corrupt tables if we try to push it further (sigh, it seems to be hard to write decent software).

Now, a second point to consider: The full VxClass run is meant for -correlation- -- e.g. it is designed to be favour accuracy over speed. There are two (trivial) tricks to push performance higher:

Disable expensive comparisons if approximate comparisons yield high similarity -- VxClass contains a -very- fast approximate comparison that is used to schedule more expensive comparisons. If the approximate comparison detects high similarity, this is essentially sufficient for the AV scenario -- not necessarily for the correlation scenario though.
Use the automated signature generation for pre-filtering. VxClass can automatically generate "smart" signatures (which is AV-speak for byte signatures with wild cards -- AVs use a lot of hashes these days, making wildcards seem "smart". The way this works is that we generate byte signatures on the fly, and then only perform expensive comparisons on those executables -not- matched by existing signatures.

Summary: Give me 40 machines and a decent database server, and I will make VxClass process 40k samples / day.

Now, an interesting question comes up: What do most AV labs actually do ?

From what I could gather, their approach is essentially a combination of behavior-monitoring and hash generation -- e.g. they run large farms of virtualized environments, inject malware, and then observe behavior. If the behavior is bad, they add a "hash signature" to their signature DB and roll it out.

Kaspersky seems to be doing something that has relationships to image processing (they seem to use a lot of GPU code), the details are sketchy tho.

Regarding dealing with the 30m or 60m legacy samples: The actual quantity of samples that you need to work on is going to be -drastically- less: Process the first 10k, generate signatures, sieve out from the 60m those caught by the signatures, repeat. I would be surprised if you need to perform expensive processing on more than 1m files.

I hope this clarifies a bit. A lot of the issues we are wrestling with are those induced by our small size: An AV company spends more on executive cab fares per month than what we can afford to spend on computation. Most large AVs have -hundreds- of machines processing incoming malware, we have 8

Cheers,
Halvar

Thanks for the response, Halvar!

Thank you very much for the detailed explanation!

A new blog post by Lenny Zeltser summarizing the topic:

How to Get Started With Malware Analysis

http://blogs.sans.org/computer-forensics/2010/11/12/get-started-with-malware-analysis/

Yup, and quoting our board =)

That was nice of Lenny .

Woodmann