You trust your users right? Me neither. Ever watched top/psrinfo or repeatedly use the w command to attempt to watch users on your system? We all have at some point. There are a lot of ways to see what users are doing, from checking their .bash_history to using DTrace (see 'shellsnoop'). But this is all kids stuff. There are times you not only want to watch your users but you really need to know what their doing. If you want to know exactly what people are doing on your systems, I'm glad to say there is a solution: Solaris Auditing, otherwise known as the Solaris Basic Security Module (BSM). Its a valued part of Trusted Solaris and available to you in Solaris 10 out of the box (and Nevada Solaris Express).
When auditing is enabled the system will log as much detail about whats occuring on the system as you wish. You can see everything from logins and logouts to executions to process creation to file access. If it happens, you can see it. When an action occurs that you want to capture the action is recorded and placed into an "audit trace file". Later, these trace files can be used to provide all sorts of details about system activity. The possibilities are nearly endless.
You'll find the auditing configuration files in /etc/security/. In that directory are several config files for auditing, some scripts for enabling or disabling auditing, as well as some RBAC related files that we're not worried about here. In particular, you'll see the following files:
- audit_class: Defines classes of audit events that are packed together for easy use.
- audit_control: Primary audit configuration file
- audit_data: Datafile maintained by auditd, don't edit this file.
- audit_event: Defines all the auditable events
- audit_record_attr: Defines the record format for various types of events
- audit_startup: Script for starting BSM/Auditing services
- audit_user: Define user specific audit flags (from audit_class)
- audit_warn: Auditd warning notification script
- bsmconv: Script for enabling BSM/Auditing
- bsmunconv: Script for removing BSM/Auditing
To get started on your auditing journey, you need to execute the /etc/security/bsmconv script as root. Once you run this you'll need to reboot so shut down most things before your run the script and when it completes do the reboot. The docs suggest you do this in single-user mode, but unless the system is in production I wouldn't worry about it. Following the reboot, check the auditing service svcs -a auditd, if its not enabled you can do so with svcadm enable auditd. You'll see the "auditd" process running via ps -ef.
When defining what events you want to record, you'll want to look at the classes defined in audit_class, and to better understand those classes you'll want to look at audit_event. Classes are represented by 2 letters, for instance "fr" represents "File Read". You can string together these classes to get just the sort of records you want.
In the /etc/security/audit_control configuration file there are two lines in particular that your interested in: flags and naflags. The "flags" line specifies the default event classes to audit. These flags can be over-riden by specifying flags for a user by name in audit_user. The "naflags" specifies event classes to audit only when the event can not be attributed to a specific user (naflags meaning "non-attributable flags"). You can modify the flags you specify with a + or -, where +flag means only record the event on success and -flag means only record the event on failure. A good example might be if you only want to audit file writes that fail or only logins that succeed.
Some interesting audit classes include (see the list in audit_class:
- fr: file read
- fw: file write
- nt: network
- lo: login or logout
- fc: file create
- fd: file delete
- xx: All X events
The following is an example audit_control (refer to the audit_control and audit_user man pages for more details):
Naturally, storing all this data isn't easy. Events are recorded into an "audit trail", by default these audit trails are stored in /var/audit, although for security purposes its recommended that they be places on a remote mount point (more on this later). Audit trails can get really really large based on how much your auditing so be careful to watch them closely for a day or two following enabling it. In order to speed things up, trails are written in binary, which means that you need to "close" them before moving them around and you can't just cat or tail the trails. The currently active audit trail will have "not_terminated" in its file name. Before doing analysis you should "close" the trail, which can be done by stopping auditd or by using auditreduce -O and then removing the not_termianted trail.
In order to read the binary audit trails we can use praudit. This tool can output audit trails in raw format (-r), short form (-s) with one line per record (-l), or in XML form (-x). The short form is useful for digging through manually, while the XML form is handy if you want to use an XML processor to convert it into another more interesting and presentable format. I recommend XML form for those new to audit trails because all the output data is enclosed in tags that describe what your looking at which is handy for learning.
Another useful tool is auditreduce, which can merge and select audit records from audit trail files. This is handy when you want to consolidate one or more audit trails or for moving audit trails around. Its handy if you want to, say, move audit trails off to another system on a regular basis instead storing the trails directly on NFS this would be the tool for you, or even if you wanted to take audit trails from multiple effected systems (such as during a breakin) and create a single audit trail to search through.
In Solaris10 a really kool feature was added to BSM: plugins! Using a plugin (shared lib) you can now output audit events directly to Syslog! This means that you don't need to leave audit trails on disk, NFS or local, but can send them, as they happen, off to a secure system. To enable it just add a line such as the following to your /etc/security/audit_control:
Note that the flags specified on this line don't define what gets audited. The flags are there to define what events should be passed via Syslog, effectively allowing you to filter certain flags to be syslog'ed even if you use other flags elsewhere. Please see Martin Englund's blog for more info, or read the audit_syslog man page.
Once you've enabled the syslog plugin for BSM, redirect the Syslog to a remote server with a line like this in /etc/syslog.conf:
Auditing provides you with a lot of useful capabilities. Security, of course, is the key, but it goes further than that. Ever get tired of a user saying "I don't know what I did, but..."? If you used auditing you could look back through the users execs() and see what they had done. Curious if users are snooping through files? You could look through the audit trail at all file reads on a certain file.
You can even use auditing for troubleshooting! As an example, when I went to the MySQL Users Confrence I took my dev workstation with me for demos. At home I use an LDAP server so when I booted the system at the show it got really angry that LDAP wasn't present so I disabled the LDAP client service. When I got home I forgot about it and didn't re-enable LDAP although most of the user accounts are still in local files so the system was uneffected and I didn't notice. Because I didn't remove ldap from the nsswitch.conf, the system still was constantly trying LDAP lookups despite not having a client to preform the lookups on its behalf and I never realized it untill I was setting up auditng for this blog entry and realized that I kept seeing a lot of failed read requests to the Solaris Door of the name service... d0h! I'm not saying auditing is your first choice in troubleshooting tools, but it sure came in handy for me that time. The point simply is, auditing isn't just about security, it has a wide range of uses.
Please realize that by the nature of auditing your writting data to disk (or NFS) every time your system preforms and event that it needs to audit. It doesn't take a genius to realize that this is going to effect performance negatively. In my test its not generally a significant hit, but if you are going to consider using Solaris Auditing in production limit your audited events as much as possible and carefully monitor server and application performance for a day or two following the time that you enabled it.
Auditing can seem difficult. I avoided it for years because it seemed so hard... but it really isn't. The config files have like a whopping 4 lines, so if you've never tried out auditing, give it a whirl and see if its for you.