linux

Building a home media server with ZFS and a gaming virtual machine

Work has kept me busy lately so it's been a while since my last post... I have been doing lots of research and collecting lots of information over the holiday break and I'm happy to say that in the coming days I will be posting a new server setup guide, this time for a server that is capable of running redundant storage (ZFS RAIDZ2), sharing home media (Plex Media Server, SMB, AFP) as well as a full Windows 7 gaming rig simultaneously!

Windows runs in a virtual machine and is assigned it's own real graphics card from the host's hardware using the using the brand-new VFIO PCI passthrough technique with the VGA quirks enabled. This does require a motherboard and CPU with support for IOMMU, more commonly known as VT-d or AMD-Vi.

Rating: 

How to Convert a GPT disk layout to a MS-DOS/MBR layout without data loss (and Gigabyte Hybrid EFI)

If you're coming here from Google searching for how to convert a GPT disk layout to MS-DOS/MBR and don't want to read through my (probably boring) story, click here ;)

Adventures with Hybrid EFI

My gaming PC has been long overdue due for a reformat. I naively allocated only 30GB to the Windows partition (and the other 120GB to 3 flavours of Linux) thinking I wouldn't use Windows for much other than Starcraft 2, but a few months back I had the urge to play Battlefield 2 again. Ever since installing and fully patching it disk space has been running pretty tight. I had to disable sleep, hibernation as well as system restore and still only had 4GB of free space, so my filesystem became fragmented easily. With the release of Windows 8 Customer Preview (download it free here), I figured it was a good time to reformat my disk and reinstall all my OSs from scratch.

I figured while I'm at it, I would make all of the big changes at once and enabled EFI booting on my Gigabyte GA-Z68A-D3H board. Little did I know that when the BIOS says "EFI," it really means Gigabyte's "Hybrid EFI" implementation and not UEFI (although in retrospect, the fact that I made the change in the BIOS should have been enough of a hint, right?). With Hybrid EFI enabled, Windows 7 and Windows 8CP installed perfectly and even created a nice GPT disk layout so reinstalled my games and activated Windows 7. Then I rebooted to play around in Windows 8CP for a bit (I do not like it, btw).

I then tried installing Fedora 16. To my surprise EFI booting failed every time, despite the all of the Fedora 16 installation media being EFI-capable. When attempting to boot from my Fedora 16 Live (x86_64) USB key I just would get a black screen with "........." printed one dot a time and then it would proceed to fall back to the next boot device (Windows boot manager on the hard disk). Upon re-examining my BIOS settings, I was disappointed to find that the setting was actually called "CD/DVD EFI Boot Option" indicating that perhaps USB EFI booting was not supported. Fair enough, I burnt the same F16 image I was using on the USB key to a CD and tried again. The same "........." text appeared.

It was then as I went back to boot Windows 7 that I discovered my attempts to set it as the default OS from Windows 8CP removed my capability of booting Windows 7 somehow. At this point it was 2AM and I was fed up with this stupid Hybrid EFI. I looked for a way to revert to a good old MS-DOS/MBR partition layout. After some Googling I stumbled across Rod Smith's website. He has extensive documentation on EFI booting, including with Gigabyte's implementation of Hybrid EFI. He says that it shares a large amount of code with EFI DUET (tianocore) and although it does work natively with Windows 7, it is not a full UEFI implementation. That would explain the problems I was having with Fedora, then.

The actual GPT to MBR conversion

Through the Rod Smith's guidance and a few dirty tricks, I was successfully able to convert my GPT partition - without data loss or deleting any partitions - and then boot Windows 7 in legacy/MBR mode. In order to do this you'll need your Windows installation media at hand as well as a copy of the Fedora 16 Live media. If you don't have a copy of Fedora 16 Live handy, you can download the Live media ISO (64-bit) from a local mirror here. See the Fedora 16 Installation Guide for details on burning this image to a CD or on creating a bootable USB key.

Keep in mind that at this point I only had 3 partitions and a bunch of unpartitioned space on the disk, so conversion was a rather straightforward process (all GPT partitions mapped directly to primary partitions). Although it is theoretically possible to convert GPT partitions with >4 partitions by defining which ones are to be logical partitions after conversion, I have not tested this.

  1. Boot your Fedora 16 Live media and wait for your session to start. If you're having troubles booting, press Tab at the boot loader screen and try booting with the nomodeset parameter added.
  2. Depending on your graphics card, you'll either be presented with the new Gnome 3 Shell or with the traditional interface. Start a terminal session by putting your mouse in the top right corner of the screen and typing "terminal" in the search (Gnome Shell) or by selecting Applications > System Tools > Terminal (traditional interface)
  3. Install gdisk:
    su -
    yum -y install gdisk
    This may take a few moments.
  4. Make a backup of your current GPT scheme:
    gdisk -b sda-preconvert.gpt /dev/sda
  5. Now we will attempt to convert your GPT disk layout to MS-DOS/MBR. Start gdisk:
    gdisk /dev/sda

    You should be prompted with:
    Command (? for help):
  6. Press r to start recovery/transformation.
  7. Press g to convert GPT to MBR.
  8. Press p to preview the converted MBR partition table.
  9. Make any modification necessary to the partition layout. See Rod Smith's Converting to or from GPT page for more details on this.
  10. When you're happy with the MS-DOS/MBR layout, press w to write changes to the disk.
  11. Shutdown Fedora 16 and boot from the Windows 7 installation media
  12. Enter your language & keyboard layout and then select the option to repair your computer in the bottom left corner.
  13. From the available options, select Startup Repair. Windows will ask for a reboot.
  14. Follow the previous three steps again to boot the Windows 7 installation and run startup repair
  15. Once again, boot the Windows 7 installation media but this time opt to open a command prompt instead of choosing startup repair. Type:
    bootrec /scanos
    bootrec /rebuildbcd
    bootrec /fixmbr
    bootrec /fixboot
  16. Close the command prompt and run Startup Repair one last time.

That's it! You should now have a bootable installation of Windows 7 on a MBR partition layout.

References

Rating: 

Installing the Darwin Calendar Server 2.4 on Fedora 13 or Fedora 14

As I mentioned in my last post, I've been playing with the Darwin Calendar Server (DCS) on Linux... Today I was able to re-test my setup notes to see if they worked properly, so below I've written a tutorial on how to get your own DCS server going on Fedora 13 or 14.

Installing Dependencies

Since we will be installing CalendarServer directly from the 2.4 branch subversion repository, the first thing to do is to install subversion and the dependencies for DCS:

su -
# Required to check out the source code from the repository
yum install subversion
# Dependencies
yum install patch memcached krb5-devel python-zope-interface PyXML pyOpenSSL python-kerberos
# Requirements for compiling xattr
yum install python-setuptools gcc gcc-c++ python-devel

Enable extended file attributes (xattrs)

DCS requires user extended file attributes so the user_xattr mount option must be enabled for the partition on which CalendarServer will be storing its documents and data (in this case, /srv). If you have not already enabled this option (it is disabled by default), edit /etc/fstab and add the user_xattr mount option after defaults, for example:

/dev/mapper/VolGroup-lv_root /                       ext4    defaults,user_xattr        1 1

Grab DCS from SVN and run auto-setup

Once these packages have been installed and extended file attributes have been enabled, we will begin setting up the CalendarServer as your regular, non-root user.

# Directory to hold CalendarServer checkout and its dependencies
mkdir CalendarServer
cd CalendarServer
# Checkout the code from the repo
svn checkout http://svn.calendarserver.org/repository/calendarserver/CalendarServer/tags/release/CalendarServer-2.4 CalendarServer-2.4
cd CalendarServer-2.4
# Start auto-setup
./run -s

Auto-setup will now attempt to grab any missing dependencies for CalendarServer an will unpack and patch them accordingly. You may find that the download for PyDirector stalls - if so, hit to abort setup and download it manually:
pushd ..
wget http://downloads.sourceforge.net/pythondirector/pydirector-1.0.0.tar.gz
tar xfz pydirector-1.0.0.tar.gz
popd
# Resume unpacking
./run -s

Prepare for installation

Since DCS bundles a modified version of Twisted as well as a few other projects (such as pydirector), we will now prepare an installation root folder to avoid conflicts with system libraries (i.e., Twisted if it has been installed from the Fedora repos). This code will be run as root.

su -
# setup data & document roots
mkdir -p /srv/CalendarServer/{Data,Documents}
chown -R daemon:daemon /srv/CalendarServer/
# setup installation root
mkdir -p /opt/CalendarServer/etc/caldavd
mkdir -p /opt/CalendarServer/var/run/caldavd
mkdir -p /opt/CalendarServer/var/log/caldavd

Install DCS and configure the server instance

The last step is to install DCS from the Subversion checkout we made earlier into the installation root. Replace /home/regularuser with the actual path to the home directory of your regular user.

# install DCS to installation root
cd /home/regularuser/CalendarServer/CalendarServer-2.4
./run -i /opt/CalendarServer
rm -rf /opt/CalendarServer/usr/caldavd/caldavd.plist
# copy sample configuration files
cp conf/servertoserver-test.xml /opt/CalendarServer/etc/caldavd/servertoserver.xml
cp conf/auth/accounts.xml /opt/CalendarServer/etc/caldavd/accounts.xml
cp conf/caldavd-test.plist /opt/CalendarServer/etc/caldavd/caldavd.plist
cp conf/sudoers.plist /opt/CalendarServer/etc/caldavd/sudoers.plist
# change permissions; passwords are stored plaintext!
chmod 600 /opt/CalendarServer/etc/caldavd/*

I have reported bugs #390 and #391 about problems with the setup script on 64-bit machines as well as a problem if a custom destination installation directory is used (which we did). This bit of code works around both of the bugs:
# 64-bit fix - see https://trac.calendarserver.org/ticket/391
sitelib="$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib())')"
sitearch="$(python -c 'from distutils.sysconfig import get_python_lib; print(get_python_lib(1))')"
if [ "$sitelib" != "$sitearch" ];then
  mv /opt/CalendarServer"${sitelib}"/twisted/plugins/caldav.py* /opt/CalendarServer"${sitearch}"/twisted/plugins
  # PYTHONPATH fix for 64-bit - see https://trac.calendarserver.org/ticket/390
  sed -i.orig 's|PYTHONPATH="'"${sitelib}"'|DESTDIR=/opt/CalendarServer\nPYTHONPATH="${DESTDIR}'"${sitelib}"':${DESTDIR}'"${sitearch}"':|' /opt/CalendarServer/usr/bin/caldavd
else
  # PYTHONPATH fix for 32-bit - see https://trac.calendarserver.org/ticket/390
  sed -i.orig 's|PYTHONPATH="'"${sitelib}"'|DESTDIR=/opt/CalendarServer\nPYTHONPATH="${DESTDIR}'"${sitelib}"':|' /opt/CalendarServer/usr/bin/caldavd
fi

If you would like your server to use SSL (highly recommended), you will need to generate a certificate. If you have a certificate and key ready to install, place it in /opt/CalendarServer/etc/tls. If not, you can easily generate a free self-signed one:

# Generate SSL keys
mkdir /opt/CalendarServer/etc/tls
openssl req -new -newkey rsa:1024 -days 365 -nodes -x509 -keyout www.example.com.key -out www.example.com.crt

Now, edit /opt/CalendarServer/etc/caldavd/caldavd.plist in your favorite editor and configure the server as follows:
    <!-- Network host name [empty = system host name] -->
    <key>ServerHostName</key>
    <string>example.com</string> <!-- The hostname clients use when connecting -->

# Data roots
    <!-- Data root -->
    <key>DataRoot</key>
    <string>/srv/CalendarServer/Data/</string>
        
    <!-- Document root -->
    <key>DocumentRoot</key>
    <string>/srv/CalendarServer/Documents/</string>

# Test accounts configuration
    <!-- XML File Directory Service -->
    <key>DirectoryService</key>
    <dict>
      <key>type</key>
      <string>twistedcaldav.directory.xmlfile.XMLDirectoryService</string>
         
      <key>params</key>
      <dict>
        <key>xmlFile</key>
        <string>/opt/CalendarServer/etc/caldavd/accounts.xml</string>
      </dict>
    </dict>

# Sudoers configuration
    <!-- Principals that can pose as other principals -->
    <key>SudoersFile</key>
    <string>/opt/CalendarServer/etc/caldavd/sudoers.plist</string>

# Delete this section
<!-- Wikiserver authentication (Mac OS X) -->
      <key>Wiki</key>
      <dict>
        <key>Enabled</key>
        <true/>
        <key>Cookie</key>
        <string>sessionID</string>
        <key>URL</key>
        <string>http://127.0.0.1/RPC2</string>
        <key>UserMethod</key>
        <string>userForSession</string>
        <key>WikiMethod</key>
        <string>accessLevelForUserWikiCalendar</string>
      </dict>

# logging
    <!--
        Logging
      -->

    <!-- Apache-style access log -->
    <key>AccessLogFile</key>
    <string>/opt/CalendarServer/var/log/caldavd/access.log</string>
    <key>RotateAccessLog</key>
    <false/>

    <!-- Server activity log -->
    <key>ErrorLogFile</key>
    <string>/opt/CalendarServer/var/log/caldavd/error.log</string>

    <!-- Log levels -->
    <key>DefaultLogLevel</key>
    <string>info</string> <!-- debug, info, warn, error -->
# a bit further down…
    <!-- Global server stats -->
    <key>GlobalStatsSocket</key>
    <string>/opt/CalendarServer/var/run/caldavd/caldavd-stats.sock</string>
# <snip>
    <!-- Server statistics file -->
    <key>ServerStatsFile</key>
    <string>/opt/CalendarServer/var/log/caldavd/stats.plist</string>
       
    <!-- Server process ID file -->
    <key>PIDFile</key>
    <string>/opt/CalendarServer/var/run/caldavd/caldavd.pid</string>

# SSL 
    <!-- Public key -->
    <key>SSLCertificate</key>
    <string>/opt/CalendarServer/etc/tls/www.example.com.crt</string>
           
    <!-- Private key -->
    <key>SSLPrivateKey</key> 
    <string>/opt/CalendarServer/etc/tls/www.example.com.key</string>

# Privilege drop
    <!--
        Process management
      -->
   
    <key>UserName</key>
    <string>daemon</string>
   
    <key>GroupName</key>
    <string>daemon</string>
       
    <key>ProcessType</key>
    <string>Combined</string>

# iSchedule server-to-server settings
      <!-- iSchedule protocol options -->
      <key>iSchedule</key>
      <dict>
        <key>Enabled</key>
        <false/>
        <key>AddressPatterns</key>
        <array>
        </array>
        <key>Servers</key>
        <string>/opt/CalendarServer/etc/caldavd/servertoserver.xml</string>
      </dict>

# Communication socket
    <!-- A unix socket used for communication between the child and master processes.
         An empty value tells the server to use a tcp socket instead. -->
    <key>ControlSocket</key>
    <string>/opt/CalendarServer/var/run/caldavd/caldavd.sock</string>


# Twisted
    <!--
        Twisted
      -->
        
    <key>Twisted</key>
    <dict>
      <key>twistd</key>
      <string>/opt/CalendarServer/usr/bin/twistd</string>
    </dict>

# Load balancer
    <!--
        Python Director
      -->

    <key>PythonDirector</key>
    <dict>
      <key>pydir</key>
      <string>/opt/CalendarServer/usr/bin/pydir.py</string>

      <key>ConfigFile</key>
      <string>/opt/CalendarServer/etc/pydir.xml</string>

      <key>ControlSocket</key>
      <string>/opt/CalendarServer/var/run/caldavd/caldavd-pydir.sock</string>
    </dict>

...Profit!

Try starting the server!

/opt/CalendarServer/usr/bin/caldavd -T /opt/CalendarServer/usr/bin/twistd -f /opt/CalendarServer/etc/caldavd/caldavd.plist -X

If all goes well, press to kill the process and then daemonize it:
/opt/CalendarServer/usr/bin/caldavd -T /opt/CalendarServer/usr/bin/twistd -f /opt/CalendarServer/etc/caldavd/caldavd.plist

Rating: 

Playing with the Darwin Calendar Server

Sorry for the lack of posts lately, I've been caught up with school and updating my FOSS on the side whenever I have some spare time.

The server that runs this site as well as diffingo.com and a few others is almost always idle - the load averages rarely exceed 0.5, and are most often sitting somewhere around 0.2 (it's a 1U/Core 2 Quad Q9550@2.83GHz/8GB RAM/2x1TB RAID 1). I have bigger plans for it in the future, but at the moment there just isn't a great deal for it to do since it handles everything so quickly.

I have been wanting to see how KVM VMs perform on it as well as try out the Darwin Calendar Server (DCS)... Seeing as DCS requires Python 2.5 and I didn't want to mess around with the live server's configuration (CentOS 5.x ships with Python 2.4), I installed Fedora 13 in a virtual machine so I could test the calendar server safely.

To my surprise, KVM works really, really well... I wasn't expecting that seeing as the versions I had used in Fedora were so much more recent. The performance is good - I haven't performed any stress-testing yet (which obviously will show a gap between the VM and running natively) but the DCS is running very smoothly and feels very responsive, so I'm confident that the difference in performance is not so large.

How I got the DCS running on Linux is a whole other story... I'll save that another post (F13/F14 DCS installation guides coming soon) but is it ever handy to have a CalDAV server! Previously, I was only able to sync calendars manually (with the USB cable) which made checking for homework assignments extremely annoying, as half of the information was always my iPod and the other half on iCal and I had to sync all the time. Now it's all over-the-air, so as long as there's Internet connectivity the calendar events can be pulled in or pushed out.

Rating: 

Random thought

In any of my future classes related to operating systems, I want to try saying "sudo" instead of "please" and see how many people get it :P

I'll keep you updated.

Rating: 
Tags: 

Fedora 11 is out!

You can grab it here. I will post some updated howtos soon.

Rating: 

I'm tired of this (rev3)

Apple has been in my good books for a long time because as their computers have no problems. Nearly no viruses, popups or adware at all. No additional software needed. Their computers 'just worked', right out of the box. I'm seeing more and more things in Apple that I don't like, things that I stopped using Microsoft's products for. Apple's products work wonderfully, but in many cases only with other Apple-based products. iPhone. iPod. iLife. iMac. And even then I find they don't work that well all the time.

A perfect example is recently when I was creating a slideshow using music purchased from iTunes. The iTunes Plus tracks worked flawlessly - Drag & drop, that was it. I'm happy Steve Jobs supports it and I really hope the industry moves DRM free... But I'm getting off track. I try the regular iTunes tracks (DRM encumbered) and turns out they refused to be added to the iMovie slideshow claiming the computer wasn't authorized. I entered my password, authorizing from iMovie which didn't work so then from iTunes too. I even deauthorized and reauthorized the Macbook in iTunes to make sure. Then I tried playing the tracks in iTunes - It worked. I switched to iMovie and what d'ya you know, same results. In the end I burnt all the songs to two CDs and then ripped them. Another two hours of my time wasted. Apple's FairPlay doesn't seem too fair at all - I couldn't use it on the very same computer I had purchased the songs from, and forget even trying to play them on another computer. I don't even know if you can put songs purchased from iTunes onto non-iPod players without having to break the DRM first (which is illegal in the US).

This time, Apple has added encrypted firmware and hashes in the database which makes it near impossible to use a new iPod with 3rd party tools (see the article I posted at the beginning of this entry). To make it worse, the encrypted firmware makes you unable to run Linux (aka Rockbox) on it to workaround the database issue. One could say otherwise, but I don't see the advantage of encrypted firmware or hashes in the database to users... What do the 3rd party tools change from Apple point of view? Users have still purchased their iPods, and whether people update iPods from iTunes or GtkPod doesn't make a difference to Apple whatsoever.

Considering one can't use an iPod with Linux anymore, I'll have to use iTunes from Windows or Mac OS X. And considering what happened the last time I used iTunes, I won't be buying the new iPod everyone's talking about either.

Rating: 

My ramblings on Marvell 88E8056, 2.6.21, sky2

The Marvell LAN on my GA-965P-S3 board stopped working all of a sudden after a reboot the other day, and I franticly tried flashing the BIOS & firmware, I undid my overclock, even rewired everything inside. No luck.

The problem was due to the semi-broken sky2 driver... From what I've heard it's been broken for a while (and developers know it). Sky2 seems to work pretty well from my experience.. Apart from this.

Anyways the cause was the new 2.6.21 kernels, which is why it only started happening after a reboot, as I started booting new kernel. Reverting to 2.6.20 (in my case, the FC6 kernel) worked perfectly.

$ /sbin/lspci | grep Ethernet
04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 14)
Rating: