Centralized System Monitoring With Swatch Stephen E. Hansen and E. Todd Atkins Distributed Computing Group Stanford University Stephen.Hansen@Stanford.EDU Todd.Atkins@Stanford.EDU
What every good system administrator tries to do is keep an eye on the health of each of the systems in his or her care. The health of a system should be reflected in the log messages generated by the kernel and the various daemons and utilities. These messages should also include information relevant to system security. With most systems we have seen, the system's log information is not generally made available to the system administrator in a way that is either secure or convenient. The Unix syslog facility, regardless of the original intent, has traditionally been used as more of a debugging aid than as a tool for system management. The assumption seems to be that system log files are only to be consulted after the fact, to help determine what happened rather than what is happening or what is about to happen. All modern Unix systems do some logging. This usually consists of memory, disk, or tape errors, jammed ethernet notices, and the like. With the exception of login failures, few systems log the kinds of information that might indicate a security related probe or attack. Even when this information is logged, it is often hidden away in places that are either not monitored regularly or are susceptible to deletion or modification by a successful intruder.
Program | Description |
---|---|
fingerd | Reports the originating
host and the finger target(s) to syslog. |
ftpd | Reports originating host to syslog. Reports file transfers to a local log file along with the local user name and, if the user is ``anonymous'', the password. |
ruserok | Used by rshd and login when called by rlogind. Disallows and reports to syslog any attempts to use a /etc/hosts.equiv or ~/.rhosts file that contains a `+'. |
rshd | Reports the access status, local user, remote user and host, and the command issued to a local log file. |
login | Number of tries reduced to three. Reports to syslog on `INCOMPLETE LOGIN ATTEMPT', `REPEATED LOGIN ATTEMPT', and `ROOT LOGIN REFUSED'. Includes the account names attempted and the originating host. |
At our site we were fortunate enough to have access to the vendor's source code for all our utilities. While this is not possible for everyone, each of the utilities listed in Table 1 are available from various network archive sites. In a few cases it might be preferable to use the public version instead so as to improve portability. Another source of security related information is from the tcp wrapper code written by Weitse Venema[1]. Besides providing access control for those network services run out of inetd, it generates information via syslog about the connections it mediates. One important utility not listed in Table 1 is sendmail. Even without modification sendmail can be configured to generate a plethora of status information. Unfortunately, sendmail isn't very discriminating in what it reports, assigning every status message the same priority. Modifying sendmail to assign appropriate priority levels to its status messages is on our to-do list.
The way our facility is set up, each server system keeps its own copy of most of the syslog messages in the file /var/log/syslog. Syslog files are rotated on a daily basis, compressed, and kept online for about a week. Log messages that might reflect a system's health or potential security problems are also forwarded to a central log host, the logmaster. In practice this means that almost everything except sendmail status messages are sent to the logmaster. Leaving the sendmail status messages on the servers cuts down on the network traffic due to syslog without significantly affecting our ability to monitor. On our systems the sendmail messages can account for as much as 90% of a host's log messages, although 50% is more common. Appendix A shows the syslog configuration file (/etc/syslog.conf) for a host being monitored. The last two lines in the file are responsible for sending data to the logmaster.
Copying the syslog information to a central site is done for several reasons. First, it provides security and redundancy. If the log files on the originating host are destroyed or modified, either accidently or by malicious intent, those on the hopefully more secure central site will be left intact. Second, it simplifies the monitoring of all the log information. By collecting information from a number of systems in a single ordered file, problems may be found that would be missed if viewed in isolation. For example, a single failed log in attempt on one system might be attributed to a typing error. The same failed log in attempt occurring on several systems in sequence could indicate an intruder trying to break in. Collecting information from several different system utilities as well as from more than one system can provide information indicating a pattern of attack. Several fingers followed by a failed login or rsh command is a common pattern revealed by this type of monitoring. Logging the target of the finger requests has been extremely useful in exposing undue interest or, in the case of multihopped fingers, attempts to hide a cracker's staging area.
Swatch's most powerful function is in examining information as it is being
appended to a log file. We use swatch to look at messages as they are being
added to the syslog file, alerting us immediately to serious system
problems as they occur. Using a tail(1)
of /var/log/syslog is
the default action for swatch but another file can be ``tailed'' by using
the -t command line option as in
% swatch -t /var/log/authlog
Receiving timely notification of certain types of probes or attacks often enables us to find out which users are logged on to the originating system. Finding out such information can help identify hackers or compromised accounts.
Using the -f option, swatch can be made to read in and process a file from beginning to end. This single pass feature can be used to examine old syslog or other text files.
% swatch -f /var/log/syslog.0
This option can be used to catch up on the contents of log files after being away from the computer for a while (like after vacationing in Hawaii for a week). This feature is often used to filter through several megabytes of old syslog files to look for evidence of suspected system and network related problems as well as system probes and break-in attempts. Having swatch examine the output from a program is also useful. For example, one might want to sort through process accounting or other audit information that is not kept in a plain text file and requires special processing to read.
% swatch -c swatchrc.acct -p lastcomm
/pattern/[,/pattern/,º] action[,action,º] [ [ [ HH: ] MM: ] SS [ start:length ] ]
The patterns must be regular expressions which Perl will accept, which are very similar to those used by the Unix egrep program. Each string to be matched is compared, in order, with the expressions in the configuration file and if a match is found the corresponding actions are taken. A copy of the Unix manual page for swatch's configuration file is listed in Appendix C. The time interval can be used to help eliminate redundant messages. For example, on our systems ``file system full'' messages tend to come at the rate of several dozen per minute. We specify an interval of five minutes which will usually eliminate hundreds of redundant notifications. The time stamp location information is optional and can only be used when a time interval is specified. Swatch uses it strip away the time stamp in order to compare it to other messages which are stored in its internal history list. Lines beginning with the `#' character are treated as comment lines and are ignored.
$N
will be replaced by the Nth
field in the matched line. A $0
or a $* will be replaced by
the entire line.
logger(1)
program periodically via
cron(1)
to send a message which contains the string ``test
pager.'' This causes swatch to attempt to page our on call systems
administrator. The second pattern/action line looks for a /bin/login syslog
message of the form
Jul 30 13:49:47 Sierra login: REPEATED LOGIN FAILURES ON ttyq0 FROM cert.cert.org: root, anonyme, anonyme
The string REPEATED matches the pattern and swatch executes a script to finger the host that initiated the failed login and store the information for later examination. We are occasionally able to detect compromised accounts with this information.
The rest of the file contains pattern/action lines that are grouped by specific names of machines. It watches out for kernel messages which indicate a potentially serious problem, such as a machine crash or an unexpected reboot. It also looks for messages from the room temperature monitor. When these types of messages are encountered, swatch sends a mail message to our systems administrator mailbox and executes a script to call a pager with a code indicating the system and message type.
FIGURE 1. Swatch configuration file for continuous monitoring # # Swatch configuration file for constant system monitoring in the background # # # Test the pager every once in a while # /test pager/ exec="/etc/call_pager 5551234 123" # # Bad login attempts # /INVALID|REPEATED|INCOMPLETE/ exec="/etc/backfinger $0" # # CAST # /CAST.*(panic|halt)/ mail=action,exec="/etc/call_pager 5551212 0911" 05:00 0:16 /CAST.*reboot/ mail=action,exec="/etc/call_pager 5551212 0411" 05:00 0:16 /CAST.*SunOS Release/ mail=action,exec="/etc/call_pager 5551212 0411" 05:00 0:16 /CAST.*file system full/ mail=action,exec="/etc/call_pager 5551212 0611" 05:00 0:16 # # Sierra # /Sierra.*WizMON/ mail=action,exec="/etc/call_pager 5551234 1666" 05:00 0:16 /Sierra.*(panic|halt)/ mail=action,exec="/etc/call_pager 5551234 1911" 05:00 0:16 /Sierra.*reboot/ mail=action,exec="/etc/call_pager 5551234 1411" 05:00 0:16 /Sierra.*SunOS Release/ mail=action,exec="/etc/call_pager 5551234 1411" 05:00 0:16 /Sierra.*file system full/ mail=action,exec="/etc/call_pager 5551234 1611" 05:00 0:16
FIGURE 2. Personalized swatch configuration file # # Personal Swatch configuration file to be run in a window on a workstation # # # These probes should be harmless, but who knows? # /fingerd.*(root|[Tt]ip|guest|atkins)/ echo,bell,exec="/bin/date >> /home/atkins/tmp/finger.log",exec="/usr/local/etc/backfinger @$6 >> /home/atkins/tmp/finger.log" # # This should never happen # /su: atkins/ echo,bell /su: .* failed/ echo,bell=3 /[dD]enied/||/DENIED/ echo=boldunderline,bell # Alert me of bad login attempts and find out who is on that system /INVALID|REPEATED|INCOMPLETE/ echo=underline,bell=3 # Important program errors /LOGIN/ echo=boldunderline,bell=3 /passwd/ echo=bold,bell=3 /ruserok/ echo=bold,bell=3 # Ignore this stuff /sendmail/,/nntp/,/xntp|ntpd/,/faxspooler/ ignore # Report unusual tftp info /tftpd.*(ncd|kfps|normal exit)/ ignore /tftpd/ echo,bell=3 # Kernel problems /(panic|halt|SunOS Release)/ echo=blink,bell 3:00 0:16 /file system full/ echo=bold,bell=3 5:00 0:16 # Try to ignore uninteresting kernel messages /vmunix.*(at|on)/ ignore /vmunix/ echo,bell 1:00 0:16
FIGURE 3. Output from swatch using the configuration file in Figure 3 over the course of 6 hours and more than 2300 lines of input Sep 13 05:07:23 Sierra vmunix: ie0: no carrier Sep 13 07:32:07 Sierra ftpd[17910]: FTP LOGIN FROM thermo-amy.Stanford.EDU [36.65.0.83], eaton Sep 13 07:35:58 Sierra ftpd[18015]: FTP LOGIN FROM thermo-amy.Stanford.EDU [36.65.0.83], eaton Sep 13 07:58:30 gloworm login: INCOMPLETE LOGIN ATTEMPT ON ttyp2 FROM deis17.cineca.it Sep 13 08:15:35 loading-zone.Stanford.EDU vmunix: /loading-zone: file system full Sep 13 08:15:35 stjames vmunix: NFS write error: on host loading-zone remote file system full Sep 13 08:53:25 Sierra login: REPEATED LOGIN FAILURES ON ttypb FROM uwmfe.neep.wisc.edu: help, newuser, d Sep 13 09:26:59 Sierra su: 'su root' failed for quinn on /dev/ttyp9 Sep 13 09:45:04 espresso.Stanford.EDU login: ROOT LOGIN ttyp0 FROM coffee Sep 13 10:04:50 Gordon-Biersch vmunix: pid 16100: killed due to swap problems in exec: I/O error mapping pages Sep 13 10:05:20 Sierra ftpd[21910]: FTP LOGIN FROM thermo-amy.Stanford.EDU [36.65.0.83], eaton Sep 13 10:06:25 Gordon-Biersch vmunix: /tmp: file system full, anon reservation exceeded Sep 13 10:06:43 Gordon-Biersch vmunix: pid 16118: killed due to swap problems in exec: I/O error mapping pages Sep 13 10:07:02 Gordon-Biersch vmunix: pid 16124: killed due to swap problems in exec: I/O error mapping pages Sep 13 10:09:34 Sierra ftpd[22085]: FTP LOGIN FROM thermo-amy.Stanford.EDU [36.65.0.83], eaton Sep 13 10:33:55 Gordon-Biersch fingerd[16484]: pudleys.Stanford.EDU (36.2.0.92.1654) -> "atkins" Sep 13 11:35:13 espresso.Stanford.EDU vmunix: SunOS Release 4.1.1 (ISL_CLIENT) #1: Mon Jan 13 08:58:58 PST 1992 Sep 13 11:35:13 espresso.Stanford.EDU vmunix: Copyright (c) 1983-1990, Sun Microsystems, Inc. Sep 13 11:35:13 espresso.Stanford.EDU vmunix: mem = 24576K (0x1800000) Sep 13 11:35:13 espresso.Stanford.EDU vmunix: avail mem = 22630400 Sep 13 11:35:13 espresso.Stanford.EDU vmunix: Ethernet address = 8:0:20:b:67:21 Sep 13 11:35:13 espresso.Stanford.EDU vmunix: cpu = Sun 4/40 Sep 13 11:35:13 espresso.Stanford.EDU vmunix: sd0: <SUN0207 cyl 1254 alt 2 hd 9 sec 36> Sep 13 11:35:13 espresso.Stanford.EDU vmunix: sd2: <Fujitsu M2624F cyl 1463 alt 2 hd 11 cyl 1463 alt 2 hd 11 sec 63> Sep 13 11:54:22 espresso.Stanford.EDU vmunix: rebooting... Sep 13 11:56:40 espresso.Stanford.EDU vmunix: NOT BLOCK: GOTO REQUESTLOOP Sep 13 11:56:50 espresso.Stanford.EDU vmunix: zs3: silo overflow Sep 13 12:06:05 Sierra ftpd[28258]: FTP LOGIN FROM vali.Stanford.EDU [36.59.0.32], fanning Sep 13 12:11:10 Sierra ftpd[29236]: FTP LOGIN FROM me-bradshaw.Stanford.EDU [36.65.0.71], bradshaw
2. L. Wall and R. Schwatz. ``Programming Perl'', O'Reilly and Associates, Sebastopol, CA. 1991.
# syslog configuration file. # # Master syslog configuration file. # # This file is processed by m4 so be careful to quote (`') names # that match m4 reserved words. Also, within ifdef's, arguments # containing commas must be quoted. # # Note: Have to exclude user from most lines so that user.alert # and user.emerg are not included, because old sendmails # will generate them for debugging information. If you # have no 4.2BSD based systems doing network logging, you # can remove all the special cases for "user" logging. # *.err;kern.debug;auth.notice;user.none /dev/console *.err;kern.debug;daemon,auth.notice;mail.crit;user.none /var/adm/messages lpr.debug /var/adm/lpd-errs # You may want to add operator to the following if your operator # is a traditional Unix style operator. *.alert;kern.err;daemon.err;user.none root *.emerg;user.none * ifdef(`LOGHOST', # for loghost machines, to have authentication messages (su, login, etc.) # logged to a file, un-comment out the following line and adjust the file # name as appropriate. auth.notice /var/log/authlog daemon.info;auth.notice;mail.debug;kern.debug /var/log/syslog *.err;daemon.none;mail.none;kern.none;auth.none;user.none /var/log/syslog ) # following line for compatibility with old sendmails. they will send # messages with no facility code, which will be turned into "user" messages # by the local syslog daemon. only the "loghost" machine needs the following # line, to cause these old sendmail log messages to be logged in the # mail syslog file. # ifdef(`LOGHOST', user.alert /var/log/syslog ) # # non-loghost machines will use the following lines to cause "user" # log messages to be logged locally. # ifdef(`LOGHOST', , user.err /dev/console user.alert root ) # Send most everything to the LogMaster. If this is the logmaster, comment our the # following two lines *.info;kern.none;mail.none @logmaster kern.debug;mail.err @logmaster
SWATCH(8) MAINTENANCE COMMANDS SWATCH(8) NAME swatch - simple watcher SYNOPSIS swatch [ -c config_file ] [ -r restart_time ] [[ -f file_to_examine ] | [ -p program_to_pipe_from ] | [ -t file_to_tail ]] DESCRIPTION Swatch is designed to monitor system activity. Swatch requires a configuration file which contains pattern(s) to look for and action(s) to do when each pattern is found. OPTIONS -c filename Use filename as the configuration file. -r restart_time Automatically restart at specified time. Restart_time can be in any of the following formats: +hh:mm Restart after the specified time where hh is hours and mm is minutes. hh:mm[am|pm] Restart at the specified time. You may specify only one of the following options: -f filename Use filename as the file to examine. Swatch will do a single pass through the named file. -p program_name Examine input piped in from the program_name. -t filename Examine lines of text as they are added to filename. If swatch is called with no options, it is the same as typing the command line swatch -c ~/.swatchrc -t /var/log/syslog SEE ALSO swatch(5), signal(3) FILES /var/tmp/..swatch..PID Temporary execution file AUTHOR E. Todd Atkins (Todd.Atkins@Stanford.EDU) EE Computer Facility Stanford University NOTES Upon receiving a ALRM or HUP signal swatch will re-read the configuration file and restart. Swatch will terminate gracefully when it receives a QUIT, TERM, or INT signal. Sun Release 4.1 Last change: 30 July 1992 1
SWATCH(5) FILE FORMATS SWATCH(5) NAME swatchrc - configuration file for the simple watcher swatch(8) SYNOPSIS ~/.swatchrc DESCRIPTION This configuration file is used by the swatch(8) program to determine what types of expression patterns to look for and what type of action(s) should be taken when a pattern is matched. The file contains two TAB separated fields: /pattern/[,/pattern/,...] action[,action,...] A pattern must be a regular expression which perl(1) will accept, which is very similar to the regular expressions which egrep(1) accepts. The following actions are acceptable: echo[=mode] Echo the matched line. The text mode may be normal, bold, underscore, blink, inverse. Some modes may not work on some terminals. Normal is the default. bell[=N] Echo the matched line, and send a bell N times (default = 1). exec=command Execute command. The command may contain variables which are substituted with fields from the matched line. A $N will be replaced by the Nth field in the line. A $0 or $* will be replaced by the entire line. ignore Ignore the matched line. mail[=address:address:...] Send mail to address(es) containing the matched lines as they appear (default address is the user who is running the program). pipe=command Pipe matched lines into command. write[=user:user:...] Use write(1) to send matched lines to user(s). SEE ALSO swatch(8), perl(1) AUTHOR E. Todd Atkins (Todd.Atkins@Stanford.EDU) EE Computer Facility Stanford University Sun Release 4.1 Last change: 30 July 1992 1