CentOS 5 server setup series: server security & reliability

This how-to will not configure any one service in particular, but rather focus on the operating system as a whole in order to improve security and reliability. This how-to
will show you how to:

  • Configure the GRUB bootloader to gracefully handle kernel panics during boots
  • Create, edit & manage custom SELinux policy modules
  • Deny access to remote users with too many failed authentication attempts over SSH (Denyhosts), POP3 or IMAP (Fail2ban)
  • Improve the password hash strength
  • Enable shell timeouts
  • ...and much more!

Gracefully recovering from kernel panics: Boot robustness & more

GRUB has two very important features that you can use to make your system more robust in the event of a kernel panic or other boot error: saved default boot entries and fallbacks.

The fallback command is extremely handy when installing and testing new kernels. Without fallbacks, testing a new kernel can be a big hassle. The new kernel may panic upon boot, so one would need to attach a KVM over IP to the server so that if the new kernel panics, it is possible to force the older [functioning] kernel to boot.

GRUB's fallback feature works around this problem by allowing you to specify a series of boot entries that will be attempted sequentially. Fallbacks become even more useful when combined with the saved default feature, which allows GRUB to store the default boot entry. Upon booting any kernel, the stored default boot target can be changed to the next fallback kernel. The workflow is like so:

  1. GRUB reads the stored default boot target
  2. GRUB sets the default boot target to the next designated fallback entry, then boots the kernel
  3. If the kernel panics, reboot the machine and go to step 1.

Notice that in each iteration the stored boot target changes and we are certain that at least one kernel version is functional (since the machine is already running at this point) so we can have GRUB attempt to boot the most recent kernel, and should it fail, work its way towards the known working version until a successful boot completes.

So, how do we get this setup? First, let's create the grub-set-default script that will be used to update the default boot target:

cat << EOF > /usr/local/sbin/grub-set-default
#!/bin/sh
echo "savedefault --default=\$1" | grub --batch
echo
EOF
chmod +x /usr/local/sbin/grub-set-default

Next, edit /etc/grub.conf and replace the default=N parameter with default saved. This will cause GRUB to boot the saved default entry which can be specified by running the grub-set-default command.

Next, GRUB needs to know which entries to boot in case of a failure. In most cases you will want this to be the second entry (n=1), so add on a new line after default saved:

fallback=1

Should you wish for GRUB to attempt a third entry in case the second also fails, you can specify instead:
fallback=1 2

In this case, just be sure that your system actually has 3 boot entries, ie 3 kernels installed simultaneously. Be warned though, GRUB will probably not like it if you reference a boot entry that has not yet been defined.

The key step to this setup is to have GRUB save the next fallback kernel as the default when booting an entry. This can be done by finding the line that looks like

kernel /path/to/vmlinuz various_boot_options
and adding directly underneath it on a new line:
savedefault fallback

On the final fallback boot entry, do not add fallback at the end and simply use savedefault instead.

The last thing is to inform GRUB of the default boot target:

grub-set-default 0

That's it! For your reference, here is a sample configuration using the saved default + fallback method with 3 kernels:
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/VolGroup00/LogVol00
#          initrd /initrd-version.img
#boot=/dev/hda
default saved
timeout=5
fallback=1 2
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-164.10.1.el5)
     root (hd0,0)
     kernel /vmlinuz-2.6.18-164.10.1.el5 ro root=/dev/VolGroup00/LogVol00
     savedefault fallback
     initrd /initrd-2.6.18-164.10.1.el5.img
title CentOS (2.6.18-164.9.1.el5)
     root (hd0,0)
     kernel /vmlinuz-2.6.18-164.9.1.el5 ro root=/dev/VolGroup00/LogVol00
     savedefault fallback
     initrd /initrd-2.6.18-164.9.1.el5.img
title CentOS (2.6.18-164.el5)
     root (hd0,0)
     kernel /vmlinuz-2.6.18-164.el5 ro root=/dev/VolGroup00/LogVol00
     savedefault
     initrd /initrd-2.6.18-164.el5.img

In this sample, the system would be currently running kernel version 2.6.18-164.9.1.el5 and the new kernel 2.6.18-164.10.1.el5 would need testing. Prior to rebooting, the system administrator would call grub-set-default 0 to make GRUB boot the first entry (2.6.18-164.10.1.el5). If it fails, savedefault fallback causes the next fallback entry (kernel 2.6.18-164.9.1.el5) to be booted. If it also fails, the second entry (2.6.18-164.el5) is booted. Because it is the last entry, no more fallback entries can be booted which is why the third entry uses savedefault and not savedefault fallback.

Note that in this sample, the fallback entries follow the order in which they are defined. However, it doesn't have to be so. If you would prefer specifying the last entry (2.6.18-164.el5) as the first fallback kernel, then all you would have to do is change fallback=1 2 to fallback=2 1 when savedefault fallback is executed, it will look to the fallback parameter and set the next boot entry to entry 2 first, then entry 1.

Now that the system can gracefully handle kernel panics while booting, an obvious question is what if a kernel panic occurs once the system is running? Fortunately, there is a sysctl parameter that can automatically reboot the system after a kernel panic:

cat << EOF >> /etc/sysctl.conf

# Reboot 10 seconds after a kernel panic
kernel.panic = 10
kernel.panic_on_oops = 10
EOF

Your system can now gracefully handle boot failures and kernel panics.

Preventing repeated local/unix login attempts

pam_tally2 can be enabled to lock out users who fail to authenticate 3 times consecutively:

touch /var/log/tallylog
cat << 'EOF' > /etc/pam.d/system-auth
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 500 quiet
auth        required      pam_deny.so
auth        required      pam_tally2.so deny=3 onerr=fail unlock_time=60

account     required      pam_unix.so
account     sufficient    pam_succeed_if.so uid < 500 quiet
account     required      pam_permit.so
account     required      pam_tally2.so per_user

password    requisite     pam_cracklib.so try_first_pass retry=3 minlen=9 lcredit=-2 ucredit=-2 dcredit=-2 ocredit=-2
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok remember=10
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
EOF

A username can be unlocked by running:
pam_tally2 --reset -u username

Preventing repeated remote login attempts

SSH: Denyhosts

On any server with SSH exposed, it is a good idea to install Denyhosts. Denyhosts will periodically audit the /var/log/secure file periodically and ban any users over a given number of authentication failures within a set amount of time. It is extremely useful in combatting the inevitable automated brute force attacks that your system will be a victim of once it is connected to the Internet.

yum install denyhosts
chkconfig denyhosts on
service denyhosts start

Although the defaults should 'just work' for more systems, if you wish to configure the number of failed authentication attempts before a ban (or the ban time, for example) you can look in /etc/denyhosts.conf.

Denyhosts has a whitelist file located at /var/lib/denyhosts/allowed-hosts to which you should consider adding your home IP address/hostname to to prevent yourself from getting accidentally locked out. Denyhosts will resolve domain names, so if you have a dynamic DNS account, you can add it to the file and rest easy knowing that your home IP is always whitelisted.

POP3/IMAP: Fail2ban

Like Denyhosts, Fail2ban also locks out malicious users once they have tried (and failed) to authenticate too many times on your system within a configurable period of time. However, unlike Denyhosts which only analyses the logs for SSH authentication failures, Fail2ban supports multiple "jail" definitions which can be configured with different log files and different regular expression patterns to match against. The code below will setup a Fail2ban jail that will lock out users with too many SASL authentication failures over POP3/IMAP, but you are also free to implement additional jails for other services if the need be.

yum install fail2ban
cat << EOF > /etc/fail2ban/filter.d/dovecot-pop3imap.conf
[Definition]
failregex = dovecot: auth-worker\(default\): sql\(.*,\): Password mismatch
            dovecot: auth-worker\(default\): sql\(.*,\): unknown user
ignoreregex =
EOF
cat << EOF >> /etc/fail2ban/jail.conf
[dovecot-pop3imap]
enabled = true
filter = dovecot-pop3imap
action = iptables-multiport[name=dovecot-pop3imap, port="pop3,imap,pop3s,imaps,smtp", protocol=tcp]
logpath = /var/log/maillog
maxretry = 20
findtime = 1200
bantime = 1200
EOF

Now, let's start all configured Fail2ban jails:
chkconfig fail2ban on
service fail2ban start

You should consider editing /etc/fail2ban/jail.conf and add your IP address to the ignoreip parameter so you do not accidentally lock yourself out while testing. Just like Denyhosts, DNS hostnames are also resolved so feel free to use your dynamic DNS hostname.

Hardening the system

Below are a list of code snippets that you can execute to help harden the server with little or no effect from a user perspective. Many of these tips were found in the NSA's guide to securing RHEL 5.

  • Update password hashing to sha512 (md5 has some well-known vulnerabilities):
    /usr/sbin/authconfig --passalgo=sha512 --update

    Note that you will need to reset any existing passwords to take advantage of the new algorithm, even choose the same password. In other words, at a minimum reset the password for root and your user account by running password username for each.
  • Nobody should be trying to plug memory stick into your server when you're not around, so allow administrators to manually insmod usb-storage but disable autoloading:
    echo "install usb-storage : " >> /etc/modprobe-nousbstorage.conf
  • Blacklist wifi modules from loading:
    rm -f /etc/modprobe.d/blacklist-wireless;for i in $(find /lib/modules/`uname -r`/kernel/drivers/net/wireless -name "*.ko" -type f) ; do echo blacklist $(basename ${i%%.ko}) >> /etc/modprobe.d/blacklist-wireless; done
  • Disable core dumps:
    sed -i'' 's/# End of file/* hard core 0\n# End of file/' /etc/security/limits.conf
    cat << EOF >> /etc/sysctl.conf

    # Disable core dumps for SUID executables
    fs.suid_dumpable = 0
    EOF
  • Force the use of ExecShield (it is enabled by default):
    cat << EOF >> /etc/sysctl.conf

    # Force ExecShield to be enabled (default)"
    kernel.exec-shield = 1
    kernel.randomize_va_space = 1
    EOF
  • Tweak sysctl network settings:
    cat << EOF >> /etc/sysctl.conf

    # Only needed for routers
    net.ipv4.conf.all.send_redirects = 0
    net.ipv4.conf.default.send_redirects = 0

    # Recommended settings for all hosts
    net.ipv4.conf.all.accept_source_route = 0
    net.ipv4.conf.all.accept_redirects = 0
    net.ipv4.conf.all.secure_redirects = 0
    net.ipv4.conf.all.log_martians = 1
    net.ipv4.conf.default.accept_redirects = 0
    net.ipv4.conf.default.secure_redirects = 0
    net.ipv4.icmp_echo_ignore_broadcasts = 1
    # This next one is recommended but apparently not valid
    #net.ipv4.icmp_ignore_bogus_error_messages = 1
    net.ipv4.conf.all.rp_filter = 1
    EOF
  • Disable use of su command unless user is in wheel group:
    allowed_users="<strong>myusernameuser</strong> <strong>other_user1</strong> <strong>other_user2</strong>"
    for user in $allowed_users;do
      usermod -a -G wheel $user
    done
    sed -i'' 's/#auth\t\trequired\tpam_wheel.so use_uid/auth\t\trequired\tpam_wheel.so use_uid/' /etc/pam.d/su
  • Disable interactive boot:
    sed -i'' 's/PROMPT=yes/PROMPT=no/' /etc/sysconfig/init
  • A shell timeout will prevent anyone from accidentally leaving idle shells open:
    echo "TMOUT=900" >> /etc/profile.d/tmout.sh
    echo "readonly TMOUT" >> /etc/profile.d/tmout.sh
    echo "export TMOUT" >> /etc/profile.d/tmout.sh
    chmod 755 /etc/profile.d/tmout.sh
  • Rewrite the default system login banner so that you do not give away the OS and release version:
    echo 'Unauthorized access OR use of this system is prohibited.' > /etc/issue
  • Disable zeroconf network configuration (169.254.* IP addresses):
    echo "NOZEROCONF=yes" >> /etc/sysconfig/network
  • Some services are sensitive to the system date, so have ntpd update the system clock every 15 minutes:
    yum install ntp
    echo "15 * * * * root /usr/sbin/ntpd -q -u ntp:ntp > /dev/null" >> /etc/cron.d/ntpd
  • Disable your Apache server signature; there's no need to advertise if you are running a version with known flaws:
    sed -i'' 's/ServerSignature On/ServerSignature Off/' /etc/httpd/conf/httpd.conf
  • Disable unneeded httpd modules. Below are just some of the LoadModule lines that you can comment out /etc/httpd/httpd.conf:
    LoadModule authn_alias_module modules/mod_authn_alias.so
    LoadModule authn_anon_module modules/mod_authn_anon.so
    LoadModule authz_owner_module modules/mod_authz_owner.so
    LoadModule authz_dbm_module modules/mod_authz_dbm.so
    LoadModule auth_digest_module modules/mod_auth_digest.so
    LoadModule ldap_module modules/mod_ldap.so
    LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
    LoadModule include_module modules/mod_include.so
    LoadModule dav_module modules/mod_dav.so
    LoadModule dav_fs_module modules/mod_dav_fs.so
    LoadModule status_module modules/mod_status.so
    LoadModule info_module modules/mod_info.so
    LoadModule proxy_module modules/mod_proxy.so
    LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
    LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
    LoadModule proxy_http_module modules/mod_proxy_http.so
    LoadModule proxy_connect_module modules/mod_proxy_connect.so

    Similarly, in /etc/httpd/conf.d/proxy_ajp.conf, you can disable:

    LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
  • Disable prelinking by changing PRELINKING=yes to PRELINKING=no in /etc/sysconfig/prelink. In order to apply the changes immediately, start the prelink cron job one last time:
    sh /etc/cron.daily/prelink

Firewall: iptables/netfilter

SELinux

TODO

Intrusion detection

Intrusion detection software is a very useful warning system to detect an intrusion in a server. Although they will require your careful attention because any minute change on the server will result in a warning, for the same reason they can provide detailed information about exactly what was compromised during a break-in.

yum install rkhunter aide

Although the defaults for AIDE are fine, rkhunter needs a bit more configuration. Near line 200, you will find:
ALLOW_SSH_ROOT_USER=yes

This setting needs to match the PermitRootLogin setting in sshd_config, so therefore the value needs to be changed to no. Near line 540, you will see:
XINETD_CONF_PATH=/etc/xinetd.conf

This line needs to be commented out, as xinetd is not installed at all. Near line 610, you will find:
APP_WHITELIST=""

rkhunter needs to be told that it is OK to be running the older versions of Apache, PHP and OpenSSL that the server will be using (Red Hat backports security patches), so change the line to:
APP_WHITELIST="httpd:2.2.3 php:5.2.9 sshd:5.4p1"

Now that rkhunter is fully configured, have rkhunter analyse the system:
rkhunter --propupd

We can now do the same for AIDE:
/usr/sbin/aide --init
cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz

If you change a configuration file, update software or add users on the server, you will need to re-run these two command snippets to recalibrate rkhunter and AIDE.

Another trick that can help detect intrusions is to have RPM verify all installed packages nightly. It is trivial to set up a cron job to do this for us:

cat << 'EOF' > /etc/cron.d/rpm-verify-all
#S file Size differs
#M Mode differs (includes permissions and file type)
#5 MD5 sum differs
#D Device major/minor number mis-match
#L readLink(2) path mis-match
#U User ownership differs
#G Group ownership differs
#T mTime differs
0 2 * * * root rpm -qVa --nomtime | awk '$2!="c" {print $0}'
EOF
chmod +x /etc/cron.d/rpm-verify-all

Resources and further reading

Rating: