Qmail-Scanner: Content Scanner for Qmail

Description

Qmail-Scanner, (also known as scan4virus) is an addon that enables a Qmail Email server to scan all gatewayed Email for certain characteristics (i.e. a content scanner). It is typically used for its anti-virus protection functions, in which case it is used in conjunction with commercial virus scanners. but also enables a site (at a server/site level) to react to Email that contains specific strings in particular headers, or particular attachment filenames or types (e.g. *.VBS attachments). It also can be used as an archiving tool for auditing or backup purposes. Qmail-Scanner is integrated into the mail server at a lower level than some other Unix-based virus scanners, resulting in better performance. It is capable of scanning not only locally sent/received Email, but also Email that crosses the server in a relay capacity.

Features

Uses any commercial Unix command-line virus scanner.
Can call more than one virus scanner for each mail message
Has its own internal scanner that can be used to pick up viruses for which commercial scanner updates are not yet available
The internal scanner can also be used to block Email based on attachment types, or Email with certain Email headers... Need to stop *.mp3 files or "Subject: ILOVEYOU" Email getting onto and off your LAN - can do! :-)
Auto-detects Email from "postmaster"-style and mailing-list addresses - and doesn't send virus reports to them (i.e. attempts to act more like a responsible net citizen)
Each message is tagged via a new Received: header with a virus report showing whether it is clean or not and virus scanner version numbers/etc
Messages with viruses are moved into a "maildir" mail folder for later perusal by the appropriate staff
Can optionally add a descriptive header: X-Qmail-Scanner to every Email that passes through the system to allow users to see that a scanner has run over their messages
Messages caught by Qmail-Scanner generate an Email message (currently supports English, Italian, German, Spanish, Turkish, Lithuanian, French, Portuguese and Chinese messages) to a configurable combination of the sender, recipients and a "quarantine-admin" address explaining why their message was rejected
Can archive all processed Email into an archive maildir. Useful when debugging Email-based apps, for backup purposes and for audit policy reasons. Currently the mail envelope headers (the "rcpt to:" and "mail from:" headers) are appended to the bottom of each message.
Can report via syslog or to a file, a one-line description of each processed message, giving information such as subject line, attachment filenames, sizes, etc.
Redundant scanning. Not only does it unpack each message before running the scanners over it, it can also scan the original Email message as well as the unpacked messages (if you think a particular scanner can do a better job than Qmail-scanner's internal systems allow.)

Download

The latest release is 1.10 (via http), and is kindly housed by SourceForge. GnuPG signature of qmail-scanner-1.10.tgz is also available. Of course, you'll be needing my GPG Public Key to verify that.

Requirements

Qmail 1.03 (there's a patched src RPM for Linux users available that contains the QMAILQUEUE patch amongst other things - just "rpm --rebuild" as root to build your own i386.rpm. NOTE: I cannot vouch for it - I do not use it. Please ensure you know how it works before installing Qmail-Scanner.)
reformime from Maildrop 1.3.6+
Perl 5.005_03+
Perl module Time::HiRes
Perl module DB_File (most distributions come with it pre-installed)
Barely Optional: Mark Simpson's TNEF unpacker. Can decode those annoying MS-TNEF MIME attachments that Microsoft mail servers just love to use. If you don't have this, there are several classes of Email that you basically won't be able to detect viruses in.

Patches

Bruce Guenter's QMAILQUEUE patch is required to enable Qmail to call a different qmail-queue program than the one compiled in by default. Qmail-scanner's qmail-scanner-queue.pl perl script is used instead of Qmail's qmail-queue binary. After qmail-scanner-queue.pl has run, it calls the original qmail-queue binary to resubmit the message back into the system.

Note 1: Qmail-scanner can be installed without the QMAILQUEUE patch. This is not recommended as it would be easy to trash an otherwise working scanning system when upgrading/reinstalling Qmail for example. See Manual Installation page for details.
Note 2: Bruce Guenter also provides a patched qmail-1.03 RPM for Linux systems that contains the above patch plus a bunch of other bits and pieces. Linux users may find it easier to use that. As mentioned above, please ensure you have Qmail working before installing Qmail-Scanner. On the Qmail-Scanner mailing-lists, we are seeing too many cases of users having "problems" - which end up being Qmail configuration/understanding issues...

Supported Virus Scanners

The following virus scanners are known to work with qmail-scanner. Other Unix-based scanners should be simple to add support for.

Trend's Virus scanner
Sophos's "sweep" virus scanner
H+BEDV's antivir scanner
Kaspersky's AVPLinux scanner
MacAfee's (NAI's) virus scanner
Command's virus scanner
F-Secure Anti-Virus scanner
F-Prot Anti-Virus scanner
InocuLAN Anti-Virus scanner
Sophie: Daemon front-end to Sophos Sweep (see FAQ for details)
Trophie: Daemon front-end to Trend iscan (see FAQ for details)
Spam Assassin Daemon (see FAQ for details)

CHANGES

There is a separate page listing changes that have been made between releases

TODO

There is a separate TODO page.

FAQ

There is a separate FAQ page.

Performance/Resource Usage

Adding content/virus scanning to an Email server will considerably add to the resource usage of that server. As this "wrapper" is written in perl instead of low-level C, quite a lot of memory and file opens/stats occurs just to get it going. Adding to this the actual scanners memory and CPU usage and it becomes quite complicated (certainly the debugging info shows that the scanner harness spends more time running the commercial scanners than it does doing things itself [that is to be expected as they do quite a lot of thinking...]).

As a "rule of thumb" I'd suggest you look at how many simultaneous SMTP sessions you are willing your box to have going at any one point in time. Each SMTP session can invoke up to 'n' different virus scanners (although they run one after the other - not simultaneously) and I'd estimate that leads to around 5-6Mb of memory usage per SMTP session. Thus if your dedicated SMTP host has 256Mb RAM + 256Mb swap - that should mean you can handle - well heaps ;-) The scanners cause the CPU to be thrashed while they're running, so I'm making sure for our site that our Qmail server will only accept up to 30 incoming SMTP sessions at any one time - that way I know the box will handle it. As this leads to an increased memory usage, don't forget Qmail's memory limits will need to be increased to deal with it (set via ulimit or softlimit calls with Qmail system startup scripts).

One thing you should test for is what happens if connectivity between this server and another local SMTP server is down for any length of time (due to failure/power outage). When the link is restored, can your server handle the other trying to dump 1,000's of Email msgs onto it at once? You need to use softlimit and tcpserver's limit options to ensure your box doesn't get killed. Note that this resource issue isn't caused by Qmail-Scanner. The same thing will happen with a pure, untouched Qmail (or any other) system - it will just happen sooner...

After that scare-mongering I should say that I have tested Qmail-Scanner under ridiculously low resource conditions - and it reacts as it should - so at worst your system should start deferring Email. Thankfully DJB's layering of programs is such that this is easy to accomplish :-)

Installation

Unpack Qmail-Scanner and run ./configure --help. This will show you what options are available to you.
Run ./configure ... [with your options], it will autodetect what software is installed on your system, and will generate a script specific to your system. If you don't see any errors reported, then the build is (probably) successful.
Run ./configure again, this time include "--install" along with the options you chose, this will do the same as the previous line, but will also create the directory structure required, and install qmail-scanner-queue.pl
If you want to manually install it, see the Manual Installation page.
Before going any further, you can test the installation by running ./contrib/test_installation.sh. This will send three Emails: one normal and two "infected" with the EICAR test virus - to the administrator Email address. Obviously Qmail-Scanner should catch them, quarantine them and send messages to the administrator stating that such messages had been caught. The EICAR is a test virus - not a real one - so don't get too concerned! :-) If you receive the original messages instead of "Virus found" messages, then something is wrong with your install.

At this stage the Qmail startup script(s) (e.g. /etc/rc.d/init.d/qmail ) will need to be updated so that Qmail knows to use qmail-scanner-queue.pl instead of qmail-queue.

echo -n "qmail-smtpd, "
QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl" export QMAILQUEUE

(softlimit -a 5120000 tcpserver -l`hostname -f` -c20 -b30 -P -h -R -t10 \
 -O -Q -v -x/var/qmail/control/tcprules.cdb \
 -uQMAILDUSER -gQMAILDGROUP 0 25 qmail-smtpd 2>&1) | splogger tcpserver &

..or whatever yours is like. Basically set QMAILQUEUE just before you invoke qmail-smtpd - that way only it runs with a "different" qmail-queue - everything else carries on running the standard qmail-queue.

BE VERY CAREFUL ABOUT THAT - YOU DON'T WANT AN INFINITE LOOP FORMING!!!

This will mean from now on all incoming SMTP mail will be scanned for viruses/etc.

If you are using daemontools to control your Qmail installation, you will instead need to do something like:

echo "/var/qmail/bin/qmail-scanner-queue.pl" > /service/smtpd/env/QMAILQUEUE

...and restart Qmail.

You must increase the amount of memory your system allows qmail-smtpd to run with, as it it now running the entire perl interpreter PLUS virus scanners. Typical installs of Qmail have system rc/startup scripts (e.g. /etc/rc.d/init.d/qmail or /service/smtp/run) that limit the amount of RAM qmail-smtpd can use via ulimit or softlimit. You must increase that to around 5-6Mb. If you don't qmail-smtpd will crash with a "qq" error on the receipt of the very first message... The actual amount is dependent on the OS in question as well as the virus scanners being used, so be prepared to experiment a little. Whatever you do, don't just set it to something stupid like 100M "just to be sure". The whole point about limiting RAM usage is so that "unusal" mail messages (e.g. from spammers or hackers) can't cause your system to become unusable by making it run out of RAM.

If you want finer control, you can define QMAILQUEUE within the tcpserver rules file (man tcpserver for details), so that it only gets defined for certain IP addresses/ranges. That way you could have virus scanning only performed on mail coming from your Exchange server (as an example), and not from your Unix servers.

To scan all mail sent by local shell users, the QMAILQUEUE will also need to be defined within /etc/profile or the like so that when they send mail, it will be affected as well.

Although as they are obviously not Windows users, you may want to save your system the effort and explicitly NOT do that! :-)

If "$DEBUG=1" (the default) is set within qmail-queue-scanner.pl, then every transaction will be logged to /var/spool/qmailscan/qmail-queue.log - so you'll see how it goes. Regardless of debugging, errors (and attachment info if enabled) should also be recorded in the qmail logs (probably via syslog) - just look for entries containing the string "X-QmailScan".

Any SMTP sessions that are dropped (due to network outages/etc) may lead to files lying around in /var/spool/qmailscan . Running /var/qmail/bin/qmail-scanner-queue.pl -z at least once daily will ensure such files are deleted when they're over 30 hours old - make a cronjob to do that. Also realise that /var/spool/qmailscan/qmail-queue.log will grow without bounds. At some stage turn debugging off ($DEBUG=0) and delete the logfile. Personally, I like the logfile, so I run a cronjob that just does "mv -f qmail-queue.log qmail-queue.log.1" at 3am every morning. That way logs don't grow without bound, but you still end up with the logs from the past two days.

Qmail-Scanner contains an internal scanner which allows you to reject Email based on attachment filenames and/or Email headers. Read the minimal document on it for details.

When Viruses Strike...

When Qmail-Scanner finds a message with a virus, it moves it into a local "quarantine" mail folder (maildir format) - by default /var/spool/qmailscan/quarantine/. This means the message can be read in its pure "adulterated" state (e.g. still containing viruses) by maildir clients like mutt - or via IMAP (if maildir format supported - you'll have to work that out for yourself). At worse you can just read it - it's just a MIME file...

If you want a good IMAP server that supports maildir natively - try Courier-IMAP.

I made the decision to write it into maildir format for performance and reliability reasons - and it expressly makes it difficult for any Windows admin to click on it with their vulnerable Windows mailer and read it :-) Qmail actually comes with a program called /var/qmail/bin/maildir2mbox which can do just that... (you could run it from cron to automatically suck all the new mail messages from /var/spool/qmailscan/quarantine/new/ into a mbox.)

Also this event is logged in /var/spool/qmailscan/quarantine.log in a tab-delimited format (for post-processing). A good script is needed to convert this file into some nice graphs for management :-). See QSS for an example of one way of generating stats.

If Qmail-Scanner was configured with the "--log-details" option, then a one-line summary of every message processed is recorded either in mailstats.csv or via syslog. e.g:

Jan 16 10:42:51 mailhost perlscanner: BadTrans 0.967762 xuy@213.3 remote@addr Franky_says_RELAX <> 1646 Wed,_16_Jan_2002_10:42:42_+1300 1011130969.31369-0.mailhost:117 1011130969.31369-1.mailhost:139  
Jan 16 10:42:52 mailhost perlscanner: Clean 0.2322 jhaar@users.sourceforge.net jhaar@users.sourceforge.net Testing_testing <08008383883@mailhost> 2345 Wed,_16_Jan_2002_10:42:42_+1300 1011130969.31369-0.mailhost:117 1011130969.31369-1.mailhost:139

Support

This software is released under the GPL as found in the COPYING file enclosed.

This package is housed on SourceForge.

Any questions, suggestions, etc to the mailing-list set up to discuss this, subscribe via http://lists.sourceforge.net/mailman/listinfo/qmail-scanner-general , or subscribe to the announcements-only list via http://lists.sourceforge.net/mailman/listinfo/qmail-scanner-announce.

Last Updated: