Anomaly Detection Systems - Part 1

by Thuull

In order to talk about detection systems, we must first explore the intent behind what detection is all about.  The whole idea is to identify attacks against your network, primarily to determine whether or not an attack may have been successful and to get a handle on what is currently being done "on the other side of the fence," so to speak.

Intrusion detection systems have primarily been compartmentalized into four distinct camps, which in themselves are defined by a combination of two factors.

First, a system can be "Active" or it can be "Passive."

Second, it can be "Host Based" or "Network Based."  So, when combined, you can have an intrusion detection system that is "Active/Host Based," "Passive/Host Based," "Active/Network Based," or "Passive/Network Based."  There are obviously other ways that IDS systems can be categorized, but this paradigm set forth by Internet Security Systems pretty much covers all the bases.

In order to be classified as an "Active" IDS, the system must be capable of real-time (or near real-time) response to an identified incoming attack, such as updating firewall rules based on the attack, or notifying a command console of the activity immediately after it occurs.  "Passive" systems generally record the activity and store it for easy reference at a later date.  "Host Based" systems are exactly that; they reside on the individual hosts that are being targeted.  "Network Based" systems sit somewhere on the network between the attacker and the target, and spy on the traffic as it flows by, looking for attacks.  Generally, network based systems reside either in a demilitarized zone (DMZ), between a network's firewall and their upstream provider, between the network's firewall and the rest of the internal network, or any combination of these three.

Now, let's talk a little bit about trends.  Since the inception of intrusion detection systems as we know them today, they have generally been based around the concept of "attack signatures."  That is; every attack has a signature that distinguishes itself from other normal network traffic and from other attacks.  This is done very similarly to the way that most popular virus scanners are designed.  The system scans all the traffic, and when it sees a pattern that matches that of a known attack, it does whatever it was set up to do (page an admin, update firewall rules, notify a console, etc.).

An oft unrecognized means of accomplishing intrusion detection is "Anomaly Detection."  With an anomaly detection system, traffic that normally can be found on the network is ignored, and bits of traffic that are not normally seen are highlighted and brought to the network owner's attention.  This has distinct advantages, as outlined below.

We all know that there is no such thing as a "secure" system.  Every machine that is attached to the Internet today can have its security defeated.  What keeps this from happening in most cases is that the vulnerabilities that are on the systems have not yet been found.  But they're there, you can bet on it.  So, what happens when a new vulnerability is found?  The individual that found it will likely create some exploit code for it, to take advantage of the vulnerability.  This code is then shared with friends, or kept to oneself for a certain period of time.

Eventually, it will probably end up in the hands of the security community as a whole, and a fix for the vulnerability will be coded.  Now, between the time that the exploit is coded, and the fix is coded, what good are intrusion detection systems based on attack signature?  None, whatsoever.  Simply because of the fact that in order to be able to define a signature that identifies a discriminate attack, one must know what that attack "looks like" as it crosses the wire, or finds itself on its target system.

What I plan to set forth with this article is an alternate means of "visualizing" security on your network, be it four Linux machines sitting behind a dual-channel ISDN, or the largest banking network in the world.

Let's make some assumptions:

A.)  You cannot keep someone who wants access to your network from obtaining access, short of unplugging the machine.

B.)  You cannot stop someone from wanting to gain access to your network.

C.  You have limited resources to accomplish your security (don't we all?).

With these assumptions in mind, what can you do?  Well, you can throw man-power and resources at solving the problem - purchase clustered firewalls, intrusion detection systems, secure all of the machines in the network, etc.  But, what is the best that you can really hope to accomplish?

The best you can really do is make it difficult enough for the attacker to get in so that it takes him more time to do so than he intended.  Second, you can identify the initial scanning that must take place in order to determine what services exist on your network that may be vulnerable.  And, third, you can take actions, either aggressive or passive, to ensure that the traffic no longer continues to be able to access the machines that may be vulnerable.

How can you do this?  How can you identify all traffic that may be questionable, even exploits that were coded yesterday?  Anomaly Detection.

An extremely effective Anomaly Detection system can be built on any Linux platform with simple freeware tools and a little modification.  These tools consist of IPChains/ipfwadm, PortSentry, logcheck, Gnumeric, and an email address.  Here's how the system works...

On every system, IPChains/ipfwadm is set up to log all traffic going to ports that are not listeners.  If it's a web server and you use SSH, have IPChains log every packet that goes to any port other than 22/TCP or 80/TCP.

Modify PortSentry to execute logcheck anytime that PortSentry trips.  Use: portsentry -actp

Modify logcheck to e-mail you any unusual activity that appears in the logs to your email address.

Use Gnumeric, or any other spreadsheet that you like, to maintain a record of every rogue packet on each machine.

Maintain IP address, date and time of the activity, ports involved (including source port), DNS resolution of the offending IP address (if available), and contact information or the owners of those IP addresses.

With this system in place, you will see every packet that enters your network that does not belong on your network.  Every packet.

Face it, for an attacker to be able to compromise your system, he must know what services are running, what OSes you use, etc.  He must do some preliminary checking to determine what is on your network.

Slow him down, give yourself the ability to see it happening, and give yourself some time to respond.

The response, of course, I leave up to you.