Brain pattern recognition

Sometimes the brain just amazes me. On the way home a few days ago I was listening to Paul van Dyk - Complicated from his album In Between. I couldn't pinpoint it, but for some reason something about the song suddenly seemed very familiar... The best way I can describe it was like audio déja-vu. I knew that the familiarness wasn't just because I had heard the song a bunch of times before (I had, after all, bought the album).

Today I was thinking about it again but I still couldn't find out what made it seem so familiar. So I forgot about it, and then suddenly David Guetta vs The Egg - Love Don't Let Me Go got stuck in my head. Almost immediately, I realized "hey, that's the song!"

I think Paul Van Dyk has used a sample from it. Check it out:

Paul van Dyk - Complicated (In Between, 2007) -- sample appears throughout

David Guetta vs The Egg - Love Don't Let Me Go (Single, 2006) -- sample appears at 0:39

(Disclaimer: I didn't upload the videos, nor do I own the copyright. Sorry if the embeds break.)

I found it incredibly cool that just by thinking about a song, the brain can recognize a pattern and match it up with similar bits from other songs without you even consciously trying to do it.

Rating: 

Installing the Darwin Calendar Server 2.4 on Fedora 13 or Fedora 14

As I mentioned in my last post, I've been playing with the Darwin Calendar Server (DCS) on Linux... Today I was able to re-test my setup notes to see if they worked properly, so below I've written a tutorial on how to get your own DCS server going on Fedora 13 or 14.

Installing Dependencies

Since we will be installing CalendarServer directly from the 2.4 branch subversion repository, the first thing to do is to install subversion and the dependencies for DCS:

su -
# Required to check out the source code from the repository
yum install subversion
# Dependencies
yum install patch memcached krb5-devel python-zope-interface PyXML pyOpenSSL python-kerberos
# Requirements for compiling xattr
yum install python-setuptools gcc gcc-c++ python-devel

Enable extended file attributes (xattrs)

DCS requires user extended file attributes so the user_xattr mount option must be enabled for the partition on which CalendarServer will be storing its documents and data (in this case, /srv). If you have not already enabled this option (it is disabled by default), edit /etc/fstab and add the user_xattr mount option after defaults, for example:

/dev/mapper/VolGroup-lv_root /                       ext4    defaults,user_xattr        1 1

Grab DCS from SVN and run auto-setup

Once these packages have been installed and extended file attributes have been enabled, we will begin setting up the CalendarServer as your regular, non-root user.

# Directory to hold CalendarServer checkout and its dependencies
mkdir CalendarServer
cd CalendarServer
# Checkout the code from the repo
svn checkout http://svn.calendarserver.org/repository/calendarserver/CalendarServer/tags/release/CalendarServer-2.4 CalendarServer-2.4
cd CalendarServer-2.4
# Start auto-setup
./run -s

Auto-setup will now attempt to grab any missing dependencies for CalendarServer an will unpack and patch them accordingly. You may find that the download for PyDirector stalls - if so, hit to abort setup and download it manually:
pushd ..
wget http://downloads.sourceforge.net/pythondirector/pydirector-1.0.0.tar.gz
tar xfz pydirector-1.0.0.tar.gz
popd
# Resume unpacking
./run -s

Prepare for installation

Since DCS bundles a modified version of Twisted as well as a few other projects (such as pydirector), we will now prepare an installation root folder to avoid conflicts with system libraries (i.e., Twisted if it has been installed from the Fedora repos). This code will be run as root.

su -
# setup data & document roots
mkdir -p /srv/CalendarServer/{Data,Documents}
chown -R daemon:daemon /srv/CalendarServer/
# setup installation root
mkdir -p /opt/CalendarServer/etc/caldavd
mkdir -p /opt/CalendarServer/var/run/caldavd
mkdir -p /opt/CalendarServer/var/log/caldavd

Install DCS and configure the server instance

The last step is to install DCS from the Subversion checkout we made earlier into the installation root. Replace /home/regularuser with the actual path to the home directory of your regular user.

# install DCS to installation root
cd /home/regularuser/CalendarServer/CalendarServer-2.4
./run -i /opt/CalendarServer
rm -rf /opt/CalendarServer/usr/caldavd/caldavd.plist
# copy sample configuration files
cp conf/servertoserver-test.xml /opt/CalendarServer/etc/caldavd/servertoserver.xml
cp conf/auth/accounts.xml /opt/CalendarServer/etc/caldavd/accounts.xml
cp conf/caldavd-test.plist /opt/CalendarServer/etc/caldavd/caldavd.plist
cp conf/sudoers.plist /opt/CalendarServer/etc/caldavd/sudoers.plist
# change permissions; passwords are stored plaintext!
chmod 600 /opt/CalendarServer/etc/caldavd/*

I have reported bugs #390 and #391 about problems with the setup script on 64-bit machines as well as a problem if a custom destination installation directory is used (which we did). This bit of code works around both of the bugs:
# 64-bit fix - see https://trac.calendarserver.org/ticket/391
sitelib="$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())')"
sitearch="$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib(1))')"
if [ "$sitelib" != "$sitearch" ];then
  mv /opt/CalendarServer"${sitelib}"/twisted/plugins/caldav.py* /opt/CalendarServer"${sitearch}"/twisted/plugins
  # PYTHONPATH fix for 64-bit - see https://trac.calendarserver.org/ticket/390
  sed -i.orig 's|PYTHONPATH="'"${sitelib}"'|DESTDIR=/opt/CalendarServer\nPYTHONPATH="${DESTDIR}'"${sitelib}"':${DESTDIR}'"${sitearch}"':|' /opt/CalendarServer/usr/bin/caldavd
else
  # PYTHONPATH fix for 32-bit - see https://trac.calendarserver.org/ticket/390
  sed -i.orig 's|PYTHONPATH="'"${sitelib}"'|DESTDIR=/opt/CalendarServer\nPYTHONPATH="${DESTDIR}'"${sitelib}"':|' /opt/CalendarServer/usr/bin/caldavd
fi

If you would like your server to use SSL (highly recommended), you will need to generate a certificate. If you have a certificate and key ready to install, place it in /opt/CalendarServer/etc/tls. If not, you can easily generate a free self-signed one:

# Generate SSL keys
mkdir /opt/CalendarServer/etc/tls
openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout www.example.com.key -out www.example.com.crt

Now, edit /opt/CalendarServer/etc/caldavd/caldavd.plist in your favorite editor and configure the server as follows:
    <!-- Network host name [empty = system host name] -->
    <key>ServerHostName</key>
    <string>example.com</string> <!-- The hostname clients use when connecting -->

# Data roots
    <!-- Data root -->
    <key>DataRoot</key>
    <string>/srv/CalendarServer/Data/</string>
        
    <!-- Document root -->
    <key>DocumentRoot</key>
    <string>/srv/CalendarServer/Documents/</string>

# Test accounts configuration
    <!-- XML File Directory Service -->
    <key>DirectoryService</key>
    <dict>
      <key>type</key>
      <string>twistedcaldav.directory.xmlfile.XMLDirectoryService</string>
         
      <key>params</key>
      <dict>
        <key>xmlFile</key>
        <string>/opt/CalendarServer/etc/caldavd/accounts.xml</string>
      </dict>
    </dict>

# Sudoers configuration
    <!-- Principals that can pose as other principals -->
    <key>SudoersFile</key>
    <string>/opt/CalendarServer/etc/caldavd/sudoers.plist</string>

# Delete this section
<!-- Wikiserver authentication (Mac OS X) -->
      <key>Wiki</key>
      <dict>
        <key>Enabled</key>
        <true/>
        <key>Cookie</key>
        <string>sessionID</string>
        <key>URL</key>
        <string>http://127.0.0.1/RPC2</string>
        <key>UserMethod</key>
        <string>userForSession</string>
        <key>WikiMethod</key>
        <string>accessLevelForUserWikiCalendar</string>
      </dict>

# logging
    <!--
        Logging
      -->

    <!-- Apache-style access log -->
    <key>AccessLogFile</key>
    <string>/opt/CalendarServer/var/log/caldavd/access.log</string>
    <key>RotateAccessLog</key>
    <false/>

    <!-- Server activity log -->
    <key>ErrorLogFile</key>
    <string>/opt/CalendarServer/var/log/caldavd/error.log</string>

    <!-- Log levels -->
    <key>DefaultLogLevel</key>
    <string>info</string> <!-- debug, info, warn, error -->
# a bit further down…
    <!-- Global server stats -->
    <key>GlobalStatsSocket</key>
    <string>/opt/CalendarServer/var/run/caldavd/caldavd-stats.sock</string>
# <snip>
    <!-- Server statistics file -->
    <key>ServerStatsFile</key>
    <string>/opt/CalendarServer/var/log/caldavd/stats.plist</string>
       
    <!-- Server process ID file -->
    <key>PIDFile</key>
    <string>/opt/CalendarServer/var/run/caldavd/caldavd.pid</string>

# SSL 
    <!-- Public key -->
    <key>SSLCertificate</key>
    <string>/opt/CalendarServer/etc/tls/www.example.com.crt</string>
           
    <!-- Private key -->
    <key>SSLPrivateKey</key> 
    <string>/opt/CalendarServer/etc/tls/www.example.com.key</string>

# Privilege drop
    <!--
        Process management
      -->
   
    <key>UserName</key>
    <string>daemon</string>
   
    <key>GroupName</key>
    <string>daemon</string>
       
    <key>ProcessType</key>
    <string>Combined</string>

# iSchedule server-to-server settings
      <!-- iSchedule protocol options -->
      <key>iSchedule</key>
      <dict>
        <key>Enabled</key>
        <false/>
        <key>AddressPatterns</key>
        <array>
        </array>
        <key>Servers</key>
        <string>/opt/CalendarServer/etc/caldavd/servertoserver.xml</string>
      </dict>

# Communication socket
    <!-- A unix socket used for communication between the child and master processes.
         An empty value tells the server to use a tcp socket instead. -->
    <key>ControlSocket</key>
    <string>/opt/CalendarServer/var/run/caldavd/caldavd.sock</string>


# Twisted
    <!--
        Twisted
      -->
        
    <key>Twisted</key>
    <dict>
      <key>twistd</key>
      <string>/opt/CalendarServer/usr/bin/twistd</string>
    </dict>

# Load balancer
    <!--
        Python Director
      -->

    <key>PythonDirector</key>
    <dict>
      <key>pydir</key>
      <string>/opt/CalendarServer/usr/bin/pydir.py</string>

      <key>ConfigFile</key>
      <string>/opt/CalendarServer/etc/pydir.xml</string>

      <key>ControlSocket</key>
      <string>/opt/CalendarServer/var/run/caldavd/caldavd-pydir.sock</string>
    </dict>

...Profit!

Try starting the server!

/opt/CalendarServer/usr/bin/caldavd -T /opt/CalendarServer/usr/bin/twistd -f /opt/CalendarServer/etc/caldavd/caldavd.plist -X

If all goes well, press to kill the process and then daemonize it:
/opt/CalendarServer/usr/bin/caldavd -T /opt/CalendarServer/usr/bin/twistd -f /opt/CalendarServer/etc/caldavd/caldavd.plist

Rating: 

Playing with the Darwin Calendar Server

Sorry for the lack of posts lately, I've been caught up with school and updating my FOSS on the side whenever I have some spare time.

The server that runs this site as well as diffingo.com and a few others is almost always idle - the load averages rarely exceed 0.5, and are most often sitting somewhere around 0.2 (it's a 1U/Core 2 Quad Q9550@2.83GHz/8GB RAM/2x1TB RAID 1). I have bigger plans for it in the future, but at the moment there just isn't a great deal for it to do since it handles everything so quickly.

I have been wanting to see how KVM VMs perform on it as well as try out the Darwin Calendar Server (DCS)... Seeing as DCS requires Python 2.5 and I didn't want to mess around with the live server's configuration (CentOS 5.x ships with Python 2.4), I installed Fedora 13 in a virtual machine so I could test the calendar server safely.

To my surprise, KVM works really, really well... I wasn't expecting that seeing as the versions I had used in Fedora were so much more recent. The performance is good - I haven't performed any stress-testing yet (which obviously will show a gap between the VM and running natively) but the DCS is running very smoothly and feels very responsive, so I'm confident that the difference in performance is not so large.

How I got the DCS running on Linux is a whole other story... I'll save that another post (F13/F14 DCS installation guides coming soon) but is it ever handy to have a CalDAV server! Previously, I was only able to sync calendars manually (with the USB cable) which made checking for homework assignments extremely annoying, as half of the information was always my iPod and the other half on iCal and I had to sync all the time. Now it's all over-the-air, so as long as there's Internet connectivity the calendar events can be pulled in or pushed out.

Rating: 

A fun little script

#!/bin/bash
if [ "$1" == "-h" ] || [ "$1" == "--help" ] || [ "$1" == "-v" ] || [ "$1" == "--verbose" ];then
  echo -e "http://xkcd.com/149\n"
fi

if [ $UID -eq 0 ];then
  echo "Okay."
else
  echo "What? Make it yourself."
fi

Rating: 
Tags: 

Today...

...I decided to tackle problem I recently found out about in fwbackups. If there is a large amount of stdout or stderr output in a short amount of time from the backup subprocesses, fwbackups hangs. That level of output should never normally happen while backing up user files, but if /proc or /dev is included in a backup for example then it could trigger the problem.

After 8 hours [1] of backup benchmarks, I've concluded that not only is python's tarfile module horrendously slow (with gzip compression enabled, it is consistently two times slower than calling "tar" from the command line as a subprocess) but that I unfortunately have to reverse the feature I added in 1.43.3rc3 that adds the display of the file currently being backed up in the GUI. Sorry in advance, but it was the only way to keep both performance and prevent fwbackups from hanging mid-backup in certain situations.

[1] which as it so happens, is also how many hours there are in the first season of How I Met your Mother. It's a hilarious show!

Rating: 
Tags: 

Fixing the iTunes 10 badness

I think many will agree that iTunes 10 was more of a marketing ploy than anything else... I found that iTunes 10.0 was worse than 9.x, primarily because of the "traffic light" style buttons in the window corner and graystyle icons. That was bearable though, and then the update to 10.0.1 came along made things even worse. No more Genius sidebar, and obnoxious "Ping" buttons every time a song is selected.

So I went on a quest to fix it, and found these commands on various forum threads and blogs:

(If you are unsure how to run these commands, quit iTunes and open Application > Utilities > Terminal. Copy/paste these commands in, hitting <Enter> after each to execute it.)

defaults write com.apple.iTunes hide-ping-dropdown 1
Disables the obnoxious "Ping" drop-down menu I mentioned earlier.

defaults write com.apple.iTunes show-store-link-arrows -bool TRUE
Restores the store arrow links present when a song is selected in 10.0 and earlier.

defaults write com.apple.iTunes disablePingSidebar 1
Disables the ping sidebar. Sadly, I haven't found a way to re-enable the Genius sidebar.

defaults write com.apple.iTunes full-window -boolean YES
Puts iTunes in "full window" mode, removing the traffic light window controls.

Now, the last thing is to restore color in iTunes. To do this, download the iTunes.rsrc file linked to in comment #7 on this thread. Next, right-click on iTunes in the Applications folder and select Show Folder Contents. Inside the Contents/Resources folder, copy the downloaded iTunes.rsrc file and opt to replace existing files when prompted. Restart iTunes and you should have coloured icons!

Note: If you're reading this but you're running iTunes on Windows, a user on Apple Discussions has posted how to do the same on Windows.

Rating: 

Cryptic MySQL error

Today as I was attempting to test one of my PHP applications, I received this error after attempting to connect to a MySQL database:

Warning:  mysql_connect() [function.mysql-connect]: OK packet 6 bytes shorter
than expected in index.php on line 29

Warning:  mysql_connect() [function.mysql-connect]: mysqlnd cannot connect to
MySQL 4.1+ using old authentication in index.php on line 29

The script giving the error was running on OS X 10.6.4 with the stock PHP 5.3.1. After doing a bit of searching and reading the MySQL documentation on the old password format, I was a bit confused because I ran this on the server:
[user@host ~]# rpm -q mysql mysql-server
mysql-5.0.77-4.el5_5.3
mysql-server-5.0.77-4.el5_5.3

Both the server and client should support the new authentication version, which was introduced all the way back in MySQL 4.1. So why wouldn't it connect?

It turns out that CentOS 5 disables the new password hashes by default in favour of remaining compatible with 3.x (and earlier) MySQL clients. All you have to do is edit /etc/my.cnf and comment the old_passwords=1 line. After restarting the server, you should notice that running SELECT PASSWORD('foobar'); in a MySQL prompt will return 41-character hashes, not the old-style 16 character hashes. Reset the user passwords to start using the new hashes and you'll be good to go.

Rating: 

Heads up!

I'm going to be testing a module that emails commenters when there has been a reply.

Rating: 

Some quick PHP performance stats: suPHP vs prefork+mod_php vs itk+mod_php

I have been doing lots of research on how to properly secure PHP on a shared server, especially with regards to finding the best way to sandbox users. On stock apache installations, the apache user must have access to web content in order to serve it which has the unfortunate side effect that every user on the shared hosting server can read the files of every other user.

The solution to them is "sandboxing" them, or in other words having Apache serve each user's web files as that user. I will post a tutorial relatively soon detailing how to do so (along with configuring many other services) but in the mean time here are some benchmarks:

prefork: 2.720166 seconds
suphp: 13.621006 seconds
itk: 4.263002 seconds

These benchmarks were generated using the "ab" benchmark included with the httpd server. They represent the time it took to load the front page of my blog 200 times:
ab -c 1 -n 200 http://www.firewing1.com/
prefork is the standard apache MPM working with mod_php. It's the fastest, but for the reasons outlined above also the most insecure. suPHP tackles the problem by using a SUID executable and running PHP under CGI, but it is extremely slow - even for this modest drupal site, it is just over 5x slower than stock. I compiled the ITK MPM for Apache which also offers the feature of running files under different users but it is based on Prefork and uses mod_php. The performance is still worse (2x slower) than stock, but much better than suPHP.

Rating: 
Tags: 

The sequel to my Ubercart i18n adventures

It has been a while since I last wrote about Ubercart, but I'm still working on some multi-lingual stores for clients. I have opted for disabling the stock Catalog module and using Views instead since Views is so much more flexible and easier to theme. I have a very simple setup; some terms in a vocabulary that is localized per-term, and then a custom View that takes a term name as an argument and returns nodes belonging to that term and displays them in a nicely themed grid.

Recently, I ran into an irritating issue where the View would return results from the wrong language if two languages had the same term name. After hours of investigating (and learning all about how to implement View handlers and plugins), it seems that the stock taxonomy term argument validator for Views cannot differentiate between terms of the same name in different languages. So if multiple languages contain the term "Stewart Adam" for example, the view will just returns nodes for whichever term (and therefore language) comes first in the database query. To be fair, the i18n module adds the "language" column to the term_data table so it's not really View's fault... Nonetheless, I was surprised that the i18n module had not already corrected this issue.

I've just reported Drupal issue #832100, Taxonomy term argument validator should not validate terms defined in other languages that includes a fix to the problem by limiting query for term names to terms within the active language. It's not the greatest way to go about solving it since it essentially just copies the original validator and makes two tiny modifications in the SQL query, but it's better then modifying the View module directly.

Rating: