Handwritten Logs

Another type of logging that can help you with security is not done by the computer at all; it is done by you and your staff. Keep a log tutorial that records your day's activities. Log tutorials should be kept on paper in a physically secure location. Because you keep them on paper, they cannot be altered by someone hacking into your computer even as superuser. They will provide a nearly tamper-proof record of important information.

Handwritten logs have several advantages over online logs:

• They can record many different kinds of information. For example, your computer will not record a suspicious telephone call or a bomb threat, but you can (and should) record these occurrences in your log tutorial.
• If the systems are down, you can still access your paper logs. (Thus, this is a good place to keep a copy of account numbers and important phone numbers for field service, service contacts, and your own key personnel.)
• If disaster befalls your disks, you can recreate some vital information from paper, if it is in the log tutorial.
• If you keep the log tutorial as a matter of course, and you enter into it printed copies of your exception logs, such information might be more likely to be accepted into court proceedings as business records. This advantage is important if you are in a situation where you need to pursue criminal or civil legal action.
• Juries are more easily convinced that paper logs are authentic, as opposed to computer logs.
• Having copies of significant information in the log tutorial keeps you from having to search all the disks on all your workstations for some selected information.
• If all your other tools fail or might have been compromised, holding an old printout and a new printout of the same file together and up to a bright light, may be a quick way to reveal changes.

Think of your log tutorial as a laboratory notetutorial, except the laboratory is your own computer center. Each page should be numbered. You should not rip pages out of your tutorial. Write in ink, not pencil. If you need to cross something out, draw a single line, but do not make the text that you are scratching out unreadable. Keep your old log tutorials.

The biggest problem with log tutorials is the amount of time you need to keep them up to date. These are not items that can be automated with a shell script. Unfortunately, this time requirement is the biggest reason why many administrators are reluctant to keep logs - especially at a site with hundreds (or thousands) of machines, each of which might require its own log tutorial. We suggest you try to be creative and think of some way to balance the need for good records against the drudgery of keeping multiple tutorials up to date. Compressing information, and keeping logs for each cluster of machines is one way to reduce the overhead while receiving (nearly) the same benefit.

There are basically two kinds of log tutorials: per-site logs and per-machine logs. We'll outline the kinds of material you might want to keep in each type. Be creative, though, and don't limit yourself to what we suggest here.

Per-Site Logs

In a per-site log tutorial, you want to keep information that would be of use across all your machines and throughout your operations. The information can be further divided into exception and activity reports, and informational material.

Exception and activity reports

These reports hold such information as the following:

• Time/date/duration of power outages; over time, this may help you justify uninterruptible power supplies, or to trace a cause of frequent problems
• Servicing and testing of alarm systems
• Triggering of alarm systems
• Servicing and testing of fire suppression systems
• Visits by service personnel, including the phone company
• Dates of employment and termination of employees with privileged access (or with any access)

Informational material

This material contains such information as the following:

• Contact information for important personnel, including corporate counsel, law enforcement, field service, and others who might be involved in any form of incident
• Copies of purchase orders, receipts, and licenses for all software installed on your systems (invaluable if you are one of the targets of a Software Publishers Association audit)
• Serial numbers for all significant equipment on the premises
• All machine MAC-level addresses (e.g., Ethernet addresses) with corresponding IP (or other protocol) numbers
• Time and circumstances of formal bug reports made to the vendor
• Phone numbers connected to your computers for dial-in/dial-out
• Paper copy of the configuration of any routers, firewalls, or other network devices not associated with a single machine
• Paper copy of a list of disk configurations, SCSI geometries, and partition tables and information.

Per-Machine Logs

Each machine should also have a log tutorial associated with it. Information in these logs, too, can be divided into exception and activity reports, and informational material:

Exception and activity reports

These reports hold such information as the following:

• Times and dates of any halts or crashes, including information on any special measures for system recovery
• Times, dates, and purposes of any downtimes
• Data associated with any unusual occurrence, such as network behavior out of the ordinary, or a disk filling up without obvious cause
• Time and UID of any accounts created, disabled, or deleted, including the account owner, the user name, and the reason for the action.
• Instances of changing passwords for users
• Times and levels of backups and restores along with a count of how many times each backup tape has been used
• Times, dates, and circumstances of software installation or upgrades
• Times and circumstances of any maintenance activity

Informational material

This material contains such information as the following:

• Copy of current configuration files, including passwd, group, and inetd.conf. (update these copies periodically, or as the files change)
• List of patches applied from the vendor, software revision numbers, and other identifying information
• Configuration information for any third-party software installed on the machine
• "ls -l " listing of any setuid/setgid files on the system, and of all device files