What to do if you find a security problem

We spend time and energy securing our systems, but what do we do if we find a breach? In this unfortunate instance the solution may or may not be extreme, according to Aeleen Frisch's book Essential System Administration, O'Reilly and Associates, www.oreilly.com.

If one of the security monitoring tools you use finds a problem, there are two concerns facing you: preventing further damage and correcting whatever the current problem is. How strongly you react depends to a great extent on the security requirements of your site; everyone needs to investigate every unexpected change to the system uncovered in a security check, but how quickly it has to be done and what to do in the meantime will depend on the problem and how much of a risk you and your site are willing to assume.

For example, suppose Tripwire finds a single change on the system: the group owner of /usr/local/bin has been changed from bin to system. Assuming you've set up an appropriate configuration file and are running Tripwire nightly, you can probably change the group owner back and find out which system administrator made this silly mistake. At the other extreme, if the one change is a replacement of /ect/passwd, and you are doing only minimal security monitoring -- checking file ownerships, modes, sizes and modification dates -- you've got a much bigger problem. You can no longer really trust any file on the system because the data you have isn't good enough to determine which files have been altered. In such an extreme case, this is the right -- if extremely painful -- thing to do:

Reboot the system immediately to single-user mode, in order to attempt to get rid of any malignant users or processes (Note: There are more complex strategies for handling an intrusion-in-progress; however, they are not recommended for the uninitiated or the faint-hearted.). Disconnect the system from any unsecured network (which is pretty much any network).

Back up any files you cannot afford to lose (but be aware that they may already be tainted). Back up all log and accounting files to aid in future investigation of the problem.

You may want to keep the system down while you investigate. When you are ready to bring the system back online, reinstall the operating system from scratch (including remaking all filesystems). Restore other files manually and check them out carefully in a secure filesystem. Rebuild all executables for which you have the source code.

The severity of this cure should emphasize once again the importance of formulating and implementing an effective security monitoring process.