Linux System Administration Best Practices

You need to maintain and monitor Linux machines to keep them secure. Otherwise, the work being done on the machines is at risk. It is best if each machine has one person that is responsible for managing and maintaining the machine. We have seen machines where every disk was wiped clean by someone who had broken into the machine. Also, someone who has broken into your machine can use it to launch attacks that can threaten the other machines on the network and the performance of the network itself.

At any given moment, several automated scans of the Penn network are running, searching for potentially vulnerable machines to attack. No matter how "low-profile" your machine is, if it has any vulnerabilities, and it is on the net, it will be cracked.

Someone who gets root access to your machine may install hidden "back doors" which will allow them to get back into the machine after the original break-in is discovered and fixed. The only sure solution is to keep the machine off-line while you wipe the machine, re-install the operating system from scratch, reinstall all applications from a trusted source, and then carefully restore user files from backup as needed. You will also need to contact everyone whose passwords may have been compromised (so that they can change their passwords) and Penn policy may require other steps.

This is a time consuming process, which can be disastrous if there are research deadlines approaching. It is much safer to keep a machine secure in the first place, than to recover from a break-in.

This guide will describe the decisions and tasks involved in running a secure Linux or unix-based system. It does not contain step by step instructions, because these vary between systems. You can use an Internet search engine to find detailed instructions. If you are affiliated with SEAS, you can contact CETS for advice, but we are not familiar with every operating system, distribution, and version.

Backups

If you are responsible for a machine, you are responsible for the files stored on that machine. You must choose a backup model. If you are taking responsibility for a system that someone else set up, you should review the backup decisions to make sure that you understand them, and that they are still appropriate. A yearly test restore and review of your backup model is a good idea.

Backups allow you to recover if your machine is compromised, or if there is a disk failure or other hardware failure. Disks often fail suddenly and without warning, and events such as fires, electrical surges, and theft can cause data loss with no warning. Mirrors or RAIDs do not protect data from fire, electrical surges, physical theft, or compromised security (cyber break-ins).

In the past, people in SEAS have lost many months of work because they did not have a reliable backup. This sort of disaster can result in losing valuable research data, papers ready to submit for publication, grant proposals in preparation, etc. Many grants require that the computers involved in the research be administered according to currently accepted "best practices" which include regular backups and secure configuration.

There are five basic approaches to backing up your machine:

1. Store important files on the SEAS file server, and only store temporary files, the OS and applications on your machine. If necessary, the OS and applications can be reinstalled from original media or from a trusted source.
2. Store important files on your local disks, and contract with CETS to perform automatic network backups (see the "Help from CETS" section below).
3. Store non-volatile data (eg, input data) away from the machine on DVD or other media, and have a working copy on the machine's disks.
4. Have two machines, preferably in different rooms or buildings, and automatically copy data from one to the other. This doesn't help if the deleted or corrupted data is automatically copied before you notice the problem. Ideally you would have more than one remote copy, perhaps a daily copy and a weekly copy.
5. Copy your data periodically to a DVD or other media and store the copies away from the machine. I don't recommend this method. In my experience, when something goes wrong and you need the backups, you discover that the person who was supposed to do the backups graduated and your most recent backup is six months old.

Maintaining Security

If possible, configure your software to automatically install security patches. Security patches are often released in response to attacks that are already being used actively throughout the Internet. Attackers know that they have a small window of time before machines are patched, so they try to attack as many machines as possible as quickly as possible. These attacks are automated, with each newly compromised machine joining in the attacking force. It is a good idea to install security patches the day that they are announced. Unless you have a team of sysadmins or you want to install patches when on vacation or over break, it's best to install them automatically.

Subscribe to the security announcement mailing list for your operating system version. This list will announce when new security patches are available. Install them the day that they are announced. If your software installs patches automatically, confirm that the patch is in place when you receive the announcement.

Subscribe to the security announcement mailing lists for any applications that you have installed that are not part of the OS distribution. Any application can have a security hole. For example, some major security problems were in 3rd party software such as Oracle. Install any patches the day that they come out.

You have to worry about backups and security for every OS installation, which includes dual boot and virtual installations. It is very difficult to keep a machine up to date if it is the "unbooted" side of a dual-boot machine, so when it is booted it is often immediately compromised. We don't support dual boot systems for this reason - we simply can't keep them secure.

Accounts and passwords

Set a root password that is unusual.

Do not use the root account for anything except systems administration. No GUI logins (KDE, gnome), no playing music, no writing reports, no research work.

Close accounts when people are no longer using them, or when people are no longer affiliated with your project. Once a year, contact each account holder by phone or in person (not by email). If you can't contact someone, close their account.

Use different passwords to contain compromises within groups of similar machines. You don't want a problem to jump from a vulnerable machine to one that was safe, because the same password was used on both. Make sure that ssh agent forwarding is turned off. Use kerberos to avoid typing in different passwords all the time.

Block Network Attacks

Turn off all unnecessary services on your machine. Some services are started when the machine boots, and others are started by inetd or xinetd. You should be aware of all network services offered by your computer, and you should know how each is used and why it is necessary.

Use an internal firewall (eg iptables) to restrict services as much as possible. For example, it's better to restrict SSH to all machines at Penn, Stanford, and CMU, than to leave it open to the entire Internet. Of course, it's better to restrict SSH only to specific machines at each institution if you can. FTP, X Windows, rsync, etc should always be tunneled through SSH, so you should not have ports open for those services. Have CETS scan your machine and tell you what ports are open on your machine.

Find a guide for 'Hardening' the security setup for your distribution and follow the suggestions where possible.

If you have several machines which need to be reconfigured often, it is difficult to keep them all secure. One solution is to have one securely installed machine with two network interfaces, and only SSH enabled. All of the "experimental" machines are attached to a switch which is attached to one network interface on the secure machine. The other interface is attached to SEASnet. The only way to attack the experimental machines is through the secure machine. If you are considering this approach, contact CETS for help with the networking and IP addresses.

Don't install compilers if they are not necessary. Worms and viruses can use them to install themselves on your system.

Make sure NTP is configured correctly and running (and will run at boot time). This will ensure accurate logging timestamps.

Email

Don't run a mail server; if you run a mail server you not only have to worry about compromises to your machine through that service, but you need to worry about preventing spammers and others from using your mail server just to send mail. Running a mail server securely is a huge pain, so just use the SEAS mail server. If you have users who do not have SEAS accounts, then they should use their own school's mail server, gmail, hotmail, etc.

Mail sent to every account on your machine should be forwarded to wherever that person reads mail, and mail sent to any system account (eg "root") should be forwarded to wherever you read mail. Many software services or systems send mail to "root" if something unusual happens. Often break-ins are detected because of some small side-effect which triggers a warning. For example, someone breaks in and fills up a partition with large amounts of copyrighted material. That causes some routine accounting task to fail because it can't create a temporary file. You (root) get mail that a routine accounting task failed, investigate, and discover that the machine has been compromised.

Keep informed

Consider taking a Linux systems administration course.

Subscribe to Security-SIG at www.upenn.edu/computing/group/sug/signup/ to receive announcements of local issues and training opportunities.

Go to https://forms.us-cert.gov/maillists/ and sign up for at least alerts - better to sign up for everything and then unsubscribe if you feel that you don't need a particular category.

Monitor your machine's log and processes regularly, so that you can recognize what's normal and unusual on your machine.

Prepare for the worst

Read www.upenn.edu/computing/security/root_compromise.html and make sure that you have everything that you'd need to recover from a root compromise.

Separate operating system from data partitions to ease restores and analyses in the future.

Make an archival backup of the operating system and applications in order to speed up restores in the future to a known clean state.

Help from CETS

Contact your computer support organization if you see evidence that a machine or an account has been compromised (this is required by Penn policy). If you are affiliated with SEAS, contact CETS. We can help you through the recovery process. We do not charge for providing advice to SEAS sysadmins.

CETS can provide some services for a fee, such as providing weekly, automated backups over our network. The cost of this service varies depending on the amount of data that you have and other factors. For current pricing please contact CETS. CETS can also do systems administration work on machines, such as configuring and securing the machine, setting up services, etc. There is an hourly rate charged for this sort of work. The rate depends on the amount of experience of the individual doing the work.

If you have any further questions, please feel free to contact cets@seas.