CentOS 5 HTTP/HTTPS web server with PHP, database, virtual hosts, & web statistics [httpd+mpm_itk, mod_ssl, mod_php, awstats]

This how-to will show you how to configure:

  • An Apache 2 web server using virtual hosts
  • The ITK MPM allows each virtual host to serve requests as its own user/group
  • mod_ssl to serve pages over the secure HTTP (HTTPS) protocol
  • mod_security to help prevents everything from SQL injections to data leaks
  • mod_php for PHP scripts along with mod_suhosin to help protect mitigate risks from known and unknown flaws in PHP scripts

Rebuilding httpd for ITK

About privilege separation

By default, the Apache web server runs as the 'apache' user. This is good because a successful attack on the web server will only cause limited damage to the system, as they do not have root access. However on a shared servers which hosts multiple websites, an attack can still be very dangerous because the 'apache' user is used by all of the websites. Thus, an attack on one website can potentially grant access to any other websites that are being hosted on the system!

Privilege separation is a technique that can be used to mitigate the risk of an attack against a shared hosting server. By allowing webpages to be served by different users, each host can be assigned its own user and so the damage from a site hack will be limited to just that account and not all of the accounts hosted on the system. We will be using the ITK Apache MPM to achieve this.

Rebuild process

Unfortunately, the ITK MPM is not included in the stock httpd distribution. Fortunately, it is relatively easy to add it. Run this as your regular user to install the httpd source RPM and download the ITK patch sources:

yumdownloader --source httpd
rpm -i httpd*.src.rpm
rm httpd*.src.rpm
cd ~/rpmbuild/SOURCES
wget http://mpm-itk.sesse.net/apache2.2-mpm-itk-2.2.17-01/{02-rename-prefork-to-itk.patch,03-add-mpm-to-build-system.patch,04-correct-output-makefile-location.patch,05-add-copyright.patch,06-hook-just-after-merging-perdir-config.patch,07-base-functionality.patch,08-max-clients-per-vhost.patch,09-capabilities.patch,10-nice.patch,11-fix-htaccess-reads-for-persistent-connections.patch}

With that done, let's make a few quick modifications to the RPM spec file located at ~/rpmbuild/SPECS/httpd.spec.

We will first need to add the ITK patches. Find the last patch line in the RPM spec file, for example, Patch202: httpd-2.2.3-deflate2215.patch, then add after it:

# ITK MPM
Patch802: 02-rename-prefork-to-itk.patch
Patch803: 03-add-mpm-to-build-system.patch
Patch804: 04-correct-output-makefile-location.patch
Patch805: 05-add-copyright.patch
Patch806: 06-hook-just-after-merging-perdir-config.patch
Patch807: 07-base-functionality.patch
Patch808: 08-max-clients-per-vhost.patch
Patch809: 09-capabilities.patch
Patch810: 10-nice.patch

Now in the %pre section - after the Red Hat patches are applied - add the following lines:
# ITK MPM
mkdir server/mpm/experimental/itk/
cp -d --preserve=all server/mpm/prefork/* server/mpm/experimental/itk/
mv server/mpm/experimental/itk/prefork.c server/mpm/experimental/itk/itk.c
%patch802 -p1 -b .mpm02
%patch803 -p1 -b .mpm03
%patch804 -p1 -b .mpm04
%patch805 -p1 -b .mpm05
%patch806 -p1 -b .mpm06
%patch807 -p1 -b .mpm07
%patch808 -p1 -b .mpm08
%patch809 -p1 -b .mpm09
%patch810 -p1 -b .mpm10

This will copy the (fully Red Hat-patched) prefork MPM code into a new folder and then apply the ITK patches.

In the %build section of the spec file, you will see these lines:

# For the other MPMs, just build httpd and no optional modules
mpmbuild worker --enable-modules=none
mpmbuild event --enable-modules=none

Following the same format, add a line for the ITK MPM:
mpmbuild itk --enable-modules=none

Similarly, we find in the %install section:
install -m 755 worker/httpd $RPM_BUILD_ROOT%{_sbindir}/httpd.worker
install -m 755 event/httpd $RPM_BUILD_ROOT%{_sbindir}/httpd.event

Add a line for the ITK MPM:
install -m 755 itk/httpd $RPM_BUILD_ROOT%{_sbindir}/httpd.itk

Last of all, in the %check section there is another section were we need to add in the ITK MPM:

# Verify that the same modules were built into the httpd binaries
./prefork/httpd -l | grep -v prefork > prefork.mods
for mpm in worker; do

Change the middle line to read:
for mpm in worker event itk

We are now ready to rebuild our ITK-enabled httpd package.

cd ~/rpmbuild/SPECS
rpmbuild -ba httpd.spec

Install the dependencies listed and then re-run the rpmbuild again if necessary. After the build has finished, install your new RPMs:
rpm -Uhv ../RPMS/[arch]/{httpd,mod_ssl}-2*.rpm

Remember to replace [arch] in the second to last command with the appropriate value (most probably i686 for 32-bit machines or x86_64 for 64-bit machines). The last step is to set the httpd worker to the ITK binary by editing /etc/sysconfig/httpd and adding after the line #HTTPD=/usr/sbin/httpd.worker:
HTTPD=/usr/sbin/httpd.itk

Installing add-on modules and additional software

Let's install PHP, AWStats and a few other useful add-ons:

yum install mod_security php php-{suhosin,mysql,mcrypt,mhash,gd} awstats

Keep in mind that by default, mod_security's rules are very restrictive and you will almost certainly need to tweak them. Be sure to test thoroughly before enabling mod_security on your live server. You can modify the mod_security settings by editing the rule files in /etc/httpd/modsecurity.d.

Configuring the web server

You will find in a stock installation, no indexes are permitted at all and that visiting localhost displays the standard CentOS test page. This can be changed by editing /etc/httpd/conf.d/welcome.conf and comment out the entire LocationMatch section:

#<LocationMatch "^/+$">
#    Options -Indexes
#    ErrorDocument 403 /error/noindex.html
#</LocationMatch>

Next, let's add a small configuration file with some custom settings. Below we will try and prevent information leaks by restricting access to backup files (files ending with a "~"), .sql and .inc plaintext files which could potentially reveal critical information about how a site functions. We will also add a one-line include directive so that all files in the virtual hosts configuration directory get included as well.

mkdir /etc/httpd/vhosts.d
cat << EOF > /etc/httpd/conf.d/z_custom.conf
# named with a z_ prefix to ensure this is parsed last.

# Restrict access to sensitive file extensions that shouldn't be read via a
# browser: *~ for temp/backup files, *.sql, *.inc as it isn't registered as
# a php file.
<FilesMatch "\.(inc|.*sql|.*~)$">
  Order allow,deny
  Deny from all
</FilesMatch>

# Include our VirtualHost configurations
Include vhosts.d/*.conf
EOF

The following will initialize a template VirtualHost configuration that can be used by a script to automate the installation of new VirtualHosts:

cat << EOF > /etc/httpd/vhosts.d/sample.conf
#<VirtualHost *:80>
#    ServerName \$WEBDOMAIN
#    ServerAdmin webmaster@\$DOMAIN
#    ServerAlias \$DOMAIN_ALIASES
#    DocumentRoot /home/\$USER/web/public_html
#    ErrorLog /home/\$USER/web/logs/error_log
#    CustomLog /home/\$USER/web/logs/access_log combined
#    php_admin_flag log_errors on
#    php_admin_value error_log /home/\$USER/web/php_error_log
#    <IfModule itk.c>
#        php_admin_value session.save_path "/var/lib/php/session/\$USER"
#        AssignUserId \$USER \$USER
#    </IfModule>
#    <Directory />
#        Options FollowSymLinks
#        AllowOverride FileInfo AuthConfig Limit Indexes Options
#    </Directory>
#</VirtualHost>
EOF

Similarly, we will setup a awstats template that can create configuration via a script:
cat << EOF > /etc/awstats/awstats.model_custom.conf
LogFile="/home/\$USER/web/logs/access_log"
LogType=W
LogFormat=1
LogSeparator=" "

SiteDomain="\$WEBDOMAIN"
HostAliases="\$DOMAIN_ALIASES"
DNSLookup=2
DirData="/var/lib/awstats"
DirCgi="/awstats"
DirIcons="/awstatsicons"
AllowToUpdateStatsFromBrowser=0
AllowFullYearView=3
EnableLockForUpdate=1
DNSStaticCacheFile="dnscache.txt"
DNSLastUpdateCacheFile="dnscachelastupdate.txt"
SkipDNSLookupFor=""

AllowAccessFromWebToAuthenticatedUsersOnly=1
AllowAccessFromWebToFollowingAuthenticatedUsers="\$USER"
AllowAccessFromWebToFollowingIPAddresses=""

CreateDirDataIfNotExists=0
BuildHistoryFormat=text
BuildReportFormat=html
SaveDatabaseFilesWithPermissionsForEveryone=0
PurgeLogFile=0
ArchiveLogRecords=0
KeepBackupOfHistoricFiles=0
DefaultFile="index.html"
SkipHosts="127.0.0.1"
SkipUserAgents=""
SkipFiles=""
SkipReferrersBlackList=""
OnlyHosts=""
OnlyUserAgents=""
OnlyUsers=""
OnlyFiles=""
NotPageList="css js class gif jpg jpeg png bmp ico rss xml swf"
ValidHTTPCodes="200 304"
ValidSMTPCodes="1 250"
AuthenticatedUsersNotCaseSensitive=0
URLNotCaseSensitive=0
URLWithAnchor=0
URLQuerySeparators="?;"
URLWithQuery=0
URLWithQueryWithOnlyFollowingParameters=""
URLWithQueryWithoutFollowingParameters=""
URLReferrerWithQuery=0
WarningMessages=1
ErrorMessages=""
DebugMessages=0
NbOfLinesForCorruptedLog=50
WrapperScript="/awstats"
DecodeUA=0
MiscTrackerUrl="/js/awstats_misc_tracker.js"
LevelForBrowsersDetection=2
LevelForOSDetection=2
LevelForRefererAnalyze=2
LevelForRobotsDetection=2
LevelForSearchEnginesDetection=2
LevelForKeywordsDetection=2
LevelForFileTypesDetection=2
LevelForWormsDetection=0
UseFramesWhenCGI=1
DetailedReportsOnNewWindows=1
Expires=3600
MaxRowsInHTMLOutput=1000
Lang="auto"
DirLang="./lang"
ShowMenu=1
ShowSummary=UVPHB
ShowMonthStats=UVPHB
ShowDaysOfMonthStats=VPHB
ShowDaysOfWeekStats=PHB
ShowHoursStats=PHB
ShowDomainsStats=PHB
ShowHostsStats=PHBL
ShowAuthenticatedUsers=0
ShowRobotsStats=HBL
ShowWormsStats=0
ShowEMailSenders=0
ShowEMailReceivers=0
ShowSessionsStats=1
ShowPagesStats=PBEX
ShowFileTypesStats=HB
ShowFileSizesStats=0
ShowOSStats=1
ShowBrowsersStats=1
ShowScreenSizeStats=0
ShowOriginStats=PH
ShowKeyphrasesStats=1
ShowKeywordsStats=1
ShowMiscStats=a
ShowHTTPErrorsStats=1
ShowSMTPErrorsStats=0
ShowClusterStats=0
AddDataArrayMonthStats=1
AddDataArrayShowDaysOfMonthStats=1
AddDataArrayShowDaysOfWeekStats=1
AddDataArrayShowHoursStats=1
IncludeInternalLinksInOriginSection=0
MaxNbOfDomain = 10
MinHitDomain  = 1
MaxNbOfHostsShown = 10
MinHitHost    = 1
MaxNbOfLoginShown = 10
MinHitLogin   = 1
MaxNbOfRobotShown = 10
MinHitRobot   = 1
MaxNbOfPageShown = 10
MinHitFile    = 1
MaxNbOfOsShown = 10
MinHitOs      = 1
MaxNbOfBrowsersShown = 10
MinHitBrowser = 1
MaxNbOfScreenSizesShown = 5
MinHitScreenSize = 1
MaxNbOfWindowSizesShown = 5
MinHitWindowSize = 1
MaxNbOfRefererShown = 10
MinHitRefer   = 1
MaxNbOfKeyphrasesShown = 10
MinHitKeyphrase = 1
MaxNbOfKeywordsShown = 10
MinHitKeyword = 1
MaxNbOfEMailsShown = 20
MinHitEMail   = 1
FirstDayOfWeek=1
ShowFlagLinks=""
ShowLinksOnUrl=1
UseHTTPSLinkForUrl=""
MaxLengthOfShownURL=64
HTMLHeadSection=""
HTMLEndSection=""
Logo="awstats_logo6.png"
LogoLink="http://awstats.sourceforge.net"
BarWidth   = 260
BarHeight  = 90
StyleSheet=""
color_Background="FFFFFF" # Background color for main page (Default = "FFFFFF")
color_TableBGTitle="CCCCDD" # Background color for table title (Default = "CCCCDD")
color_TableTitle="000000" # Table title font color (Default = "000000")
color_TableBG="CCCCDD" # Background color for table (Default = "CCCCDD")
color_TableRowTitle="FFFFFF" # Table row title font color (Default = "FFFFFF")
color_TableBGRowTitle="ECECEC" # Background color for row title (Default = "ECECEC")
color_TableBorder="ECECEC" # Table border color (Default = "ECECEC")
color_text="000000" # Color of text (Default = "000000")
color_textpercent="606060" # Color of text for percent values (Default = "606060")
color_titletext="000000" # Color of text title within colored Title Rows (Default = "000000")
color_weekend="EAEAEA" # Color for week-end days (Default = "EAEAEA")
color_link="0011BB" # Color of HTML links (Default = "0011BB")
color_hover="605040" # Color of HTML on-mouseover links (Default = "605040")
color_u="FFAA66" # Background color for number of unique visitors (Default = "FFAA66")
color_v="F4F090" # Background color for number of visites (Default = "F4F090")
color_p="4477DD" # Background color for number of pages (Default = "4477DD")
color_h="66DDEE" # Background color for number of hits (Default = "66DDEE")
color_k="2EA495" # Background color for number of bytes (Default = "2EA495")
color_s="8888DD" # Background color for number of search (Default = "8888DD")
color_e="CEC2E8" # Background color for number of entry pages (Default = "CEC2E8")
color_x="C1B2E2" # Background color for number of exit pages (Default = "C1B2E2")
LoadPlugin="geoip GEOIP_STANDARD /var/lib/GeoIP/GeoIP.dat"
ExtraTrackedRowsLimit=500
EOF

(For those wondering, this template was derived by removing all comments from the stock awstats model configuration at /etc/awstats/awstats.model.conf and then substituting important values such as SiteDomain with variables so that can be easily changed via sed)

Because the web server will be using the ITK MPM, each site's requests will be run under its respective owner and group. This causes problems with the standard PHP session setup, which uses one directory owned by "apache" for all session information. To solve this, a session directory will need to be created for each user. Let's create the default one for the apache user now:

chown root.apache /var/lib/php/session
chmod 0771 /var/lib/php/session
mkdir -m 0770 /var/lib/php/session/apache
chown root.apache /var/lib/php/session/apache

You are now ready to add VirtualHosts for your domains. See Adding a new web hosting account section of Administering the server below for more information on how to do so.

The last step is to add the firewall port exceptions for http/https and start Apache:

chkconfig httpd on
service httpd start
iptables -I RH-Firewall-1-INPUT 4 -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
iptables -I RH-Firewall-1-INPUT 4 -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
service iptables save

Suhosin PHP extension

We will now be configuring /etc/php.d/suhosin.ini. Because suhosin can interfere with some site's functionality, it is best to enable it in simulation mode so that errors are logged and you can configure suhosin accordingly:

suhosin.simulation = On

Limit the maximum PHP memory_limit that scripts can set:
suhosin.memory_limit = 512M

Lastly, disable session encryption (breaks roundcubemail if enabled):
suhosin.session.encrypt = Off

Optional add-on: roundcubemail

Let's rebuild roundcubemail 0.3 for CentOS 5:

su - myusername
cd ~/rebuilds
fedpkg clone -a roundcubemail
cd roundcubemail
fedpkg switch-branch el6

Edit the spec file roundcubemail.specand comment the line Requires: php-pear-Mail-mimeDecode so that it reads #Requires: php-pear-Mail-mimeDecode. This is the only change required for CentOS 5, so let's build the package now:
fedpkg local
exit
yum install --nogpgcheck /home/myusername/rebuilds/roundcubemail/noarch/roundcubemail*.rpm

Now it is time to configure roundcubemail. In the file /etc/httpd/conf.d/roundcubemail.conf, add after the line <Directory /usr/share/roundcubemail/>:

        <IfModule itk.c>
            AssignUserId apache apache
        </IfModule>

As well, by default roundcubemail denies access to any non-localhost clients. To change this, comment out Deny from all and add Allow from all:
        Order Deny,Allow
        #Deny from all
        Allow from 127.0.0.1
        Allow from all

We will also need to create the roundcubemail database. If you have not setup your database server, follow the database tutorial now.

CREATE DATABASE roundcubemail;
USE roundcubemail;
source /usr/share/doc/roundcubemail-0.3.1/SQL/mysql.initial.sql
GRANT ALL PRIVILEGES ON roundcubemail.* TO 'roundcubemail'@'localhost' IDENTIFIED BY 'random-password';
FLUSH PRIVILEGES;

Then configure /etc/roundcubemail accordingly. You should now be able to access the webmail portal at http://www.yourdomain.com/roundcubemail.

But I want PHP 5.2!

PHP 5.2 is available in the CentOS 5 Testing repository. Certain PHP add-ons, such as php-suhosin, have not been rebuilt so you will need to do that manually.

rpm -e php-suhosin
yum --enablerepo=c5-testing update php\*

Now rebuild and reinstall php-suhosin:
yumdownloader --source php-suhosin
rpmbuild --rebuild php-suhosin-*.src.rpm

Now install the resulting php-suhosin binary RPM and then move the configuration file (/etc/php.d/suhosin.ini.rpmsave) back into place.

But I want PHP 5.3!

PHP 5.3 is available as of CentOS 5.6 the php53 package. Certain PHP add-ons, such as php-suhosin and php-mcrypt have not been rebuilt so you will need to do that manually. This is a bit tricky as you will need to rebuild the php-extras SRPM among others; I'll leave you to figure out the details. See the instructions above for PHP 5.2 for a rough workflow using php-suhosin as an example.

Some final words

Two 3rd party MPM modules satisfied my need privilege separation in this setup, the ITK MPM and the Peruser MPM. Each has its own advantages and disadvantages.

ITK's approach is to accept a request, determine which virtual host the request belongs to, fork, set the UID and GID of the fork, serve the request and then to terminate the fork after the request has been completed allowing it to fork again later. Because ITK is very simple in nature and functions similarly to the traditional prefork MPM, it is more compatible with modules like mod_ssl, mod_php and mod_python. That said, it is also slower than because of the fork/kill overhead associated with each request.

Peruser, on the other hand, starts a pool of processes that fork only on startup. There are multiplexers than accept requests, determine which virtual host the request belongs to, and then forwards the request to the appropriate process from the pool. This approach incurs practically no overhead as the process from the pool has already been forked and had its UID/GID changed. Peruser is therefore much faster than ITK; its performance is nearly almost on par with that of the traditional prefork MPM! The downfall of Peruser is that because of this connection passing from the multiplexer to a preforked process, it is more complex and has had a long history of bugs and incompatibilities. Experimental support for mod_ssl was only recently added, and upstream development still seems to have a number of bugs to work out. In addition, because Peruser requires at least one process per virtual host for the pool and then requires multiplexers to handle incoming requests, it is much more resource intensive that the ITK MPM, where a process can dynamically handle a request from any virtual host. For these reasons, I favoured ITK over Peruser.

Keep in mind that ITK does have one security flaw. The request headers must be processed in order to determine which virtual host that request belongs to, and therefore which UID/GID ITK would need to switch to. Because of this, all of the header processing is performed as root (albeit with restricted POSIX capabilities). Should someone be able to take advantage of a flaw in Apache or another module (e.g. mod_ssl), the server could theoretically be rooted.

There is an emphasis on theoretically because an attacker would need to be extremely clever (or lucky) to successfully root the server via a flaw in the request processing given the restricted POSIX capabilities. In most cases, that specific apache process would just crash and life goes on. All in all, I still consider the ITK MPM to be more secure than the standard prefork MPM because a much more likely attack on the server is a hacker breaking in through a flaw in one of the hosted website's scripts (e.g. an outdated CMS installation). In this scenario the ITK MPM protects you completely; the potential damage from the attacker is essentially limited to that user's home directory, since each user's scripts run as their own UID/GID. They do not get access to any other user's scripts, nor to any other user's emails.

Administering the server

Setting up the scripts

The following code will setup the "web_domain_add" script which can be used to create the initial website configurations for a hosting users on your server:

cat << EOF > /root/bin/web_domain_add
#!/bin/sh
username=\$1
shift
domain=\$1
shift

if [ -z \$domain ] || [ "\$username" == "-h" ];then
  echo "Usage: \$1 user domain [alias1] [alias2] [...]"
  exit 1
fi

aliases="\$domain"
for alias in "\$@";do
  aliases="\$aliases \$alias www.\$alias "
done

cat /etc/httpd/vhosts.d/sample.conf | \
  sed "s/\\$WEBDOMAIN/www.\${domain}/g" | \
  sed "s/\\$DOMAIN_ALIASES/\${aliases}/g" | \
  sed "s/\\$USER/\${username}/g" | \
  sed "s/\\$DOMAIN/\${domain}/g" | \
  sed "s/^#//g" \
  > "/etc/httpd/vhosts.d/\${username}.conf"
echo "*** Writing VHost configuration /etc/httpd/vhosts.d/\${username}.conf"

cat /etc/awstats/awstats.model_custom.conf | \
  sed "s/\\$WEBDOMAIN/www.\${domain}/g" | \
  sed "s/\\$DOMAIN_ALIASES/\${aliases}/g" | \
  sed "s/\\$USER/\${username}/g" | \
  sed "s/\\$DOMAIN/\${domain}/g" \
  > /etc/awstats/awstats.www.\${domain}.conf
echo "*** Writing AWStats configuration /etc/awstats/awstats.www.\${domain}.conf"

echo "*** Creating password for AWStats"
htpasswd /home/awstats-htpasswd "\$username"

echo "*** Done"
EOF
chmod +x /root/bin/web_domain_add

Adding a new web hosting account

In order to create the new website configurations, you must also create a new system user that the VirtualHost will be mapped to via ITK. See the SSH+SFTP tutorial for details on how to add a new restricted system user. Once you have done so, you can run the web_domain_add script:

/root/bin/web_domain_add system_username primarydomain.tld

This will initialize a new configuration for the domain primarydomain.tld mapped to system_username. You may optionally specify as many domain aliases as needed, for example:
/root/bin/web_domain_add exampleuser example.com example.net example.org

www.example.com will be used as the primary domain, and the script will automatically alias example.com as well as www.example.net, example.net, www.example.org and example.org as domain aliases for www.example.com.
Note: The script does not create any DNS entries for these domains! That task must be performed separately with your DNS provider.

Resources and further reading

Rating: