Advanced Server Monitoring with Riemann and Graphite

Choosing the right tool for the job

My current server monitoring setup is documented in my CentOS 5 server tutorials. It consists of Nagios for service monitoring and Cacti for graphing of metrics including system load, network and disk space.

Both tools are very commonly used and lots of resources are available on their setup & configuration, but I never kicked the feeling that they were plain clunky. Over the past several months, I have performed several research and evaluated a variety of tools and thankfully came across the monitoring sucks effort which aims to document a bunch of blog posts on monitoring tools and their different merits and weaknesses. The collection of all documentation the is now kept in the monitoring sucks GitHub repo.

Long story short, each tool seems to only do part of the job. I hate redundancy, and I believe that a good monitoring system would:

  1. provide an overview of the current service status;
  2. notify you appropriately and timely when things go wrong; and
  3. provide a historical overview of data to establish some sort of baseline / normal level for collected metrics (i.e graphs and 99-percentiles)
  4. ideally, be able to react proactively when things go wrong

You'll find that most tools will do two of four above well, which is just enough to be annoyingly useful. You'll need to implement 2-3 overlapping tools that do one thing well and the other just okay. Well, I don't like to live with workarounds.

I did a bit of research and solicited some advice on r/sysadmin, but sadly it did not get enough upvotes to be very noticed. Collectd looked like a wonderful utility. It is simple, high-performance and focused on doing one thing well. It was trivial to get it writing tons of system metrics to RRD files, at which point Visage provided a smooth user interface. Although it was a step in the right direction as far as what I was looking for, it still only did two of the four items above.

Introducing Riemann

Then, I stumbled across Riemann through his Monitorama 2013 presentation. Although not the easiest to configure and its notification support is a bit lacking, it has several features that immediately piqued my interest:

  • Its architecture forgoes the traditional polling and instead processes arbitrary event streams.
    • Events can contain data (the metric) as well as other information (hostname, service, state, timestamp, tags, ttl)
    • Events can be filtered by their attributes and transformed (percentiles, rolling averages, etc)
    • Monitoring up new machines is as easy as pushing to your Riemann server from the new host
    • Embed a Riemann client into your application or web service and easily add application level metrics
    • Let collectd do what it does best and have it shove the machine's health metrics to Riemann as an event stream
  • It is built for scale, and can handle thousands of events per second
  • Bindings (clients) are available in multitudes of languages
  • Has (somewhat primitive) support for notifications and reacting to service failures, but Riemann is extensible so you can add what you need
  • An awesome, configurable dashboard

All of this is described more adequately and in greater detail on its homepage. So how do you get it?

Installing Riemann

This assumes you are running CentOS 6 or more better (e.g. recent version of Fedora). In the case of CentOS, it also assumes that you have installed the EPEL repository.

yum install ruby rubygems jre-1.6.0
gem install riemann-tools daemonize
rpm -Uhv http://aphyr.com/riemann/riemann-0.2.4-1.noarch.rpm
chkconfig riemann on
service riemann start

Be sure to open ports 5555 (both TCP and UDP), 5556 (TCP) and in your firewall. Riemann will uses 5555 for event submission, 5556 for a WebSockets connection to the server.

Riemann is now ready to go and accept events. You can modify your configuration at /etc/riemann/riemann.config as required - here is a sample from my test installation:

; -*- mode: clojure; -*-
; vim: filetype=clojure

(logging/init :file "/var/log/riemann/riemann.log")

; Listen on the local interface over TCP (5555), UDP (5555), and websockets (5556)
(let [host "my.hostname.tld"]
  (tcp-server :host host)
  (udp-server :host host)
  (ws-server  :host host))

; Expire old events from the index.
(periodically-expire 5)

; Custom stuffs

; Graphite server - connection pool
(def graph (graphite {:host "localhost"}))
; Email handler
(def email (mailer {:from "riemann@my.hostname.tld"}))

; Keep events in the index for 5 minutes by default.
(let [index (default :ttl 300 (update-index (index)))]

  ; Inbound events will be passed to these streams:
  (streams

    (where (tagged "rollingavg")
      (rate 5
        (percentiles 15 [0.5 0.95 0.99] index)
        index graph
      )
      (else
        index graph
      )
    )

    ; Calculate an overall rate of events.
    (with {:metric 1 :host nil :state "ok" :service "events/sec" :ttl 5}
      (rate 5 index))

    ; Log expired events.
    (expired
      (fn [event] (info "expired" event)))
))

The default configuration was modified here to do a few things differently:
  • Expire old events after only 5 seconds
  • Automatically calculate percentiles for events tagged with rollingavg
  • Send all event data to Graphite for graphing and archival
  • Set an email handler that, with some minor changes, could be used to send service state change notifications

Installing Graphite

Graphite can take data processed by Riemann and store it long-term, while also giving you tons of neat graphs.

yum --enablerepo=epel-testing install python-carbon python-whisper graphite-web httpd

We now need to edit /etc/carbon/storage-schemas.conf to tweak the time density of retained metrics. Since Riemann supports processing events quickly, I like to retain events at a higher precision than the default settings:
# Schema definitions for Whisper files. Entries are scanned in order,
# and first match wins. This file is scanned for changes every 60 seconds.
#
#  [name]
#  pattern = regex
#  retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ...

# Carbon's internal metrics. This entry should match what is specified in
# CARBON_METRIC_PREFIX and CARBON_METRIC_INTERVAL settings
[carbon]
pattern = ^carbon\.
retentions = 60:90d

#[default_1min_for_1day]
#pattern = .*
#retentions = 60s:1d

[primary]
pattern = .*
retentions = 10s:1h, 1m:7d, 15m:30d, 1h:2y

After making your changes, start the carbon-cache service:
service carbon-cache start
chkconfig carbon-cache on
touch /etc/carbon/storage-aggregation.conf

Now that Graphite's storage backend, Carbon, is running, we need to start Graphite:

python /usr/lib/python2.6/site-packages/graphite/manage.py syncdb
chown apache:apache /var/lib/graphite-web/graphite.db
service httpd graceful

Graphite should now be available on http://localhost - if this is undesirable, edit /etc/httpd/conf.d/graphite-web.conf and map it to a different hostname / URL according to your needs.

Note: as of writing, there's a bug in the version of python-carbon shipped with EL6 that complains incessantly to your logs if the storage-aggregation.conf configuration file doesn't exist. Let's create it to avoid a hundred-megabyte log file:

touch /etc/carbon/storage-aggregation.conf

But what about EL5

I am not going to detail how to install the full Riemann server on EL5, as the dependencies are far behind and it would require quite a bit of work. However, it is possible to install riemann-tools on RHEL/CentOS 5 for monitoring the machine with minimal work.

The rieman-health initscript requires the 'daemonize' command, install it via yum (EL6) or obtain it for EL5 here: http://pkgs.repoforge.org/daemonize/

The riemann-tools ruby gem and its dependencies will require a few development packages in order to build, as well as Karan's repo providing an updated ruby-1.8.7:

cat << EOF >> /etc/yum.repos.d/karan-ruby.repo
[kbs-el5-rb187]
name=kbs-el5-rb187
enabled=1
baseurl=http://centos.karan.org/el\$releasever/ruby187/\$basearch/
gpgcheck=1
gpgkey=http://centos.karan.org/RPM-GPG-KEY-karan.org.txt
EOF
yum update ruby\*
yum install ruby-devel libxml2-devel libxslt-devel libgcrypt-devel libgpg-error-devel
gem install riemann-tools --no-ri --no-rdoc

Rating: 

Installing ZFS and setting up a pool

This part of the Fedora 20 home server setup howtos will show you how to install ZFS and administer a pool to create one or more filesystems backed by redundant storage.

Rating: 

Building a home media server with ZFS and a gaming virtual machine

Work has kept me busy lately so it's been a while since my last post... I have been doing lots of research and collecting lots of information over the holiday break and I'm happy to say that in the coming days I will be posting a new server setup guide, this time for a server that is capable of running redundant storage (ZFS RAIDZ2), sharing home media (Plex Media Server, SMB, AFP) as well as a full Windows 7 gaming rig simultaneously!

Windows runs in a virtual machine and is assigned it's own real graphics card from the host's hardware using the using the brand-new VFIO PCI passthrough technique with the VGA quirks enabled. This does require a motherboard and CPU with support for IOMMU, more commonly known as VT-d or AMD-Vi.

Rating: 

Setting the OS X kernel asynchronous I/O limits to avoid VirtualBox crashing OS X

I had the need to setup a new VM for software testing today, and I kept running into intermittent problems where VirtualBox would freeze and then an OS X kernel panic, freezing/crashing the entire machine.

Luckily, I had made a snapshot in the OS moments earlier to the crash so I had a safe place to revert to, but the crashes kept happing at seemingly random times.

I setup a looped execution of 'dmesg' to see what was going on just before the crash and saw this at the next freeze:

VBoxDrv: host_vmxon  -> vmx_use_count=1 rc=0
VBoxDrv: host_vmxoff -> vmx_use_count=0
VBoxDrv: host_vmxon  -> vmx_use_count=1 rc=0
aio_queue_async_request(): too many in flight for proc: 16.
aio_queue_async_request(): too many in flight for proc: 16.
aio_queue_async_request(): too many in flight for proc: 16.
aio_queue_async_request(): too many in flight for proc: 16.
aio_queue_async_request(): too many in flight for proc: 16.
aio_queue_async_request(): too many in flight for proc: 16.

The first VBoxDrv messages didn't pull anything interesting in Google, but the other messages did: Virtual Box ticket #11219 and this blog post.

It would appear that the default limits for the OS X kernel's asynchronous I/O are very, very low. VirtualBox likely exceeds them when your VM(s) are performing heavy disk I/O, hence the 'too many in flight' message in the logs.

Luckily for us, there's a quick and easy solution:

sudo sysctl -w  kern.aiomax=512 kern.aioprocmax=128 kern.aiothreads=16

then restart VirtualBox. These settings will apply until you reboot. To make the changes permanent, add/update the following lines in /etc/sysctl.conf:
kern.aiomax=512
kern.aioprocmax=128
kern.aiothreads=16

Note: you can probably set those limits even higher, as documentation for Sybase (by SAP) recommends values 2048 / 1024 / 16 when using its software.

Rating: 

Compiling the IMAP extension for PHP 5.3.15 with OS X 10.8.3

I have been testing the Drupal support module locally which features the ability to create tickets from email messages to an IMAP inbox. It requires the imap_open() PHP function provided by the imap PHP extension, which unfortunately is not included in the OS X builds of PHP.

ivucica has published a wonderful script to his blog that compiles the IMAP extension without having to recompile PHP entirely, but unfortunately it was not working for me and nobody else seemed to have my problem either. Compiling the imap library and PCRE went very smoothly, but when it came time to build the PHP extension this error appeared during ./configure:

checking whether build with IMAP works... no
configure: error: build test failed. Please check the config.log for details.

Well, crap. I check config.log and determine it's a linking failure:

configure: program exited with status 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME ""
| #define PACKAGE_TARNAME ""
| #define PACKAGE_VERSION ""
| #define PACKAGE_STRING ""
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL ""
| #define COMPILE_DL_IMAP 1
| #define HAVE_IMAP 1
| #define HAVE_IMAP2000 1
| #define HAVE_IMAP2004 1
| #define HAVE_NEW_MIME2TEXT 1
| #define HAVE_LIBPAM 1
| #define HAVE_IMAP_KRB 1
| #define HAVE_IMAP_SSL 1
| /* end confdefs.h.  */
|
|
| #if defined(__GNUC__) && __GNUC__ >= 4
| # define PHP_IMAP_EXPORT __attribute__ ((visibility("default")))
| #else
| # define PHP_IMAP_EXPORT
| #endif
|
|       PHP_IMAP_EXPORT void mm_log(void){}
|       PHP_IMAP_EXPORT void mm_dlog(void){}
|       PHP_IMAP_EXPORT void mm_flags(void){}
|       PHP_IMAP_EXPORT void mm_fatal(void){}
|       PHP_IMAP_EXPORT void mm_critical(void){}
|       PHP_IMAP_EXPORT void mm_nocritical(void){}
|       PHP_IMAP_EXPORT void mm_notify(void){}
|       PHP_IMAP_EXPORT void mm_login(void){}
|       PHP_IMAP_EXPORT void mm_diskerror(void){}
|       PHP_IMAP_EXPORT void mm_status(void){}
|       PHP_IMAP_EXPORT void mm_lsub(void){}
|       PHP_IMAP_EXPORT void mm_list(void){}
|       PHP_IMAP_EXPORT void mm_exists(void){}
|       PHP_IMAP_EXPORT void mm_searched(void){}
|       PHP_IMAP_EXPORT void mm_expunged(void){}
|       void rfc822_output_address_list(void);
|       void (*f)(void);
|       char foobar () {f = rfc822_output_address_list;}
|
|     char foobar();
|     int main() {
|       foobar();
|       return 0;
|     }
|
configure:6808: result: no
configure:6819: checking whether build with IMAP works
configure:6863: cc -o conftest -g -O2   conftest.c  -Wl,-rpath,/usr/local/imap-2007f/lib -L/usr/local/imap-2007f/lib -lc-client -lpam  -lkrb5  >&5
Undefined symbols for architecture x86_64:
  "_BIO_free", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_BIO_new_mem_buf", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_BIO_new_socket", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_ERR_error_string", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
      _ssl_genkey in libc-client.a(osdep.o)
  "_ERR_get_error", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
      _ssl_genkey in libc-client.a(osdep.o)
  "_ERR_load_crypto_strings", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_EVP_PKEY_free", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_PEM_read_bio_PrivateKey", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_PEM_read_bio_X509", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_RAND_seed", referenced from:
      _ssl_onceonlyinit in libc-client.a(osdep.o)
  "_RSA_generate_key", referenced from:
      _ssl_genkey in libc-client.a(osdep.o)
  "_SSL_CTX_ctrl", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_CTX_free", referenced from:
      _ssl_abort in libc-client.a(osdep.o)
  "_SSL_CTX_load_verify_locations", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_CTX_new", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_CTX_set_cipher_list", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_SSL_CTX_set_default_verify_paths", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_CTX_set_tmp_rsa_callback", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_SSL_CTX_set_verify", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_CTX_use_PrivateKey", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_CTX_use_RSAPrivateKey_file", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_SSL_CTX_use_certificate", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_CTX_use_certificate_chain_file", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_SSL_accept", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_SSL_ctrl", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_free", referenced from:
      _ssl_abort in libc-client.a(osdep.o)
  "_SSL_get_error", referenced from:
      _ssl_getdata in libc-client.a(osdep.o)
      _ssl_sout in libc-client.a(osdep.o)
  "_SSL_get_fd", referenced from:
      _ssl_getdata in libc-client.a(osdep.o)
      _ssl_server_input_wait in libc-client.a(osdep.o)
  "_SSL_get_peer_certificate", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_library_init", referenced from:
      _ssl_onceonlyinit in libc-client.a(osdep.o)
  "_SSL_load_error_strings", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_SSL_new", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_pending", referenced from:
      _ssl_getdata in libc-client.a(osdep.o)
      _ssl_server_input_wait in libc-client.a(osdep.o)
  "_SSL_read", referenced from:
      _ssl_getdata in libc-client.a(osdep.o)
      _ssl_server_input_wait in libc-client.a(osdep.o)
  "_SSL_set_bio", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_set_connect_state", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_set_fd", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_SSL_shutdown", referenced from:
      _ssl_abort in libc-client.a(osdep.o)
  "_SSL_state", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSL_write", referenced from:
      _ssl_start in libc-client.a(osdep.o)
      _ssl_sout in libc-client.a(osdep.o)
  "_SSLv23_client_method", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_SSLv23_server_method", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_TLSv1_client_method", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_TLSv1_server_method", referenced from:
      _ssl_server_init in libc-client.a(osdep.o)
  "_X509_NAME_oneline", referenced from:
      _ssl_open_verify in libc-client.a(osdep.o)
  "_X509_STORE_CTX_get_current_cert", referenced from:
      _ssl_open_verify in libc-client.a(osdep.o)
  "_X509_STORE_CTX_get_error", referenced from:
      _ssl_open_verify in libc-client.a(osdep.o)
  "_X509_free", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_X509_get_ext_d2i", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_X509_get_subject_name", referenced from:
      _ssl_open_verify in libc-client.a(osdep.o)
  "_X509_verify_cert_error_string", referenced from:
      _ssl_open_verify in libc-client.a(osdep.o)
  "_sk_num", referenced from:
      _ssl_start in libc-client.a(osdep.o)
  "_sk_value", referenced from:
      _ssl_start in libc-client.a(osdep.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I couldn't figure out why it wasn't picking up the symbols from libssl, even when manually trying to compile said file and adding a -lssl flag.

After an hour of struggling with it and my debugging efforts going nowhere, I try adding -lcrypto for the hell of it and it works!

tl;dr, if you get this error then simply replace the following line of the aformentioned script:

./configure --with-imap=/usr/local/imap-2007f --with-kerberos --with-imap-ssl

With the following line that adds the required linker flags:
LDFLAGS="-lssl -lcrypto" ./configure --with-imap=/usr/local/imap-2007f --with-kerberos --with-imap-ssl

That's it!

Rating: 

Attaching MBR or GPT disk partitions to a VirtualBox VM

On one of my development machines I triple-boot so that I can test in a variety of operating environments. My default is OS X and the other day I wanted to write documentation using DocBook. I knew that my Linux installation already had publican installed and ready to go (publican is a wonderful documentation tool written by Red Hat to facilitate the publishing of DocBook-based documentation) and I really did not want to setup a new Fedora VM that duplicated a setup I already had.

In reality, a new VM would not be all that much effort, but I though it would be really handy to know how to boot a VM up from a physical partition and this way I could kill two birds with one stone. After some research, it seemed using VMWare's VMDK (Virtual Machine Disk Format) images was the most appropriate, as the metadata could be tricked into pointing to physical partitions instead of files. Unfortunately, the tools VirtualBox provide to create them are not compatible with GPT disk tables out of the box. This blog post does mention a trick to convert the VMDK metadata to match the GPT disk layout, but it failed consistently when I tried.

I then, by fluke, found Dave Vasilevsky's vmdk-raw-parts script on GitHub which worked flawlessly! Its usage is well documented so I will not bore you with the details, but with that said after using it to create my custom VMDK, the only change required was to run chmod 666 /dev/diskXsY to grant my user write permissions to the disk (prevents you from having to start VirtualBox as root).

Some Extra Troubleshooting

These details are completely irrelevant to the above, but I wanted to mention them here for others wanting to run the same setup:

  • If you need USB pass-through, remember to set your VCPUs to 1; set it to anything higher and USB stops functioning.
  • You'll need to redo the 'chmod' when ever you reboot your machine, as the permissions on the dev node get reset upon boot.
  • During the trial-and-error process of getting this to work, I often needed to recreate my VMDK and reload it. VirtualBox will moan about mismatched UUIDs... The simple fix is to open File > Virtual Media Manager... to delete then re-add your VMDK file.
Rating: 

Syncing your iOS device (either iPod, iPhone or iPad) with multiple iTunes libraries

With Apple's changes surrounding iCloud, managing media on iOS devices has become much easier and it is no longer absolutely bound to a single computer. However, I have still found it tricky to sync a single iOS device with multiple computers (different iTunes libraries) without having to entirely wipe the device first.

I synchronize my iTunes library between two computers regularly, so both machines have up-to-date copy of all media files. Furthermore, I set them up with the same username (but different hostname), so the iTunes media file paths are identical between machines.

Searching Apple forums revealed this solution by turingtest2: apparently, each iTunes library is assigned a randomly-generated "library ID". This identifier is stored on the iOS device, so if it doens't match when connecting to a new computer, it will request to wipe the device before proceeding with the sync. Simply copying my entire ~/Music folder from one machine to the other was enough copy all the iTunes metadata (including the library ID) and my device now happily synchronizes on either machine. For those wanting to save time and not copy their entire ~/Music folder, see the forum post for the exact files you'll need to copy.

Rating: 

Microsoft's Office 2013 licensing changes

I recently gave a few price quotes for a client and in that process, I did some background research regarding the new Office 2013.

The licensing for Microsoft Office 2013 states that copies are installable on a single PC only, non-transferable. If your disk or motherboard fails, that Office installation was tied to the PC and you must purchase another.

I just don't know what Microsoft was thinking. As of last week, Newegg.ca was selling copies of Office 2010 Home & Student (3 user) DVD media for $144.99 and Office 2013 Home & Student (single user, non-transferable) for $139.99. If you ignore the non-transferability of Office 2013 and just consider the raw cost per user-license, then Office 2013 is 2.89 times more expensive! Then consider in the fact that should your PC ever fail, you would have to get another brand new copy for $139.99... The cost per user-license of Office 2013 could therefore rise to nearly 6 times as expensive per user-license compared to Office 2010 after your first PC failure.

Many (myself included) saw this as a money-grab and an attempt to force users to their Office 365 subscription-based service, a $99.99 per year subscription grants access to the latest Microsoft Office software for up to 5 users in a household. The pricing scheme for this is a little more reasonable per-user, but a static fee each year means that Microsoft inevitably makes more money... Say a PC lasts 5 years, then they're making 5 x $99.99 instead of selling a single copy of Office for ~$150.

Last week it look like Microsoft gave in to the massive outcry from users and announced an updated clause for the Office 2013 license agreement:

You may transfer the software to another computer that belongs to you, but not more than one time every 90 days (except due to hardware failure, in which case you may transfer sooner).

Well, at least that's a little more reasonable.

Nonetheless, given the recent release of LibreOffice 4.0, I recommend it to everyone looking for an excellent Office suite and a way out from Microsoft's vendor lock-in. LibreOffice is free and always will be.

Rating: 
Tags: 

Spam comments with Mollom

I have used the free Mollom plan to protect the forms on this site with CAPTCHA or text analysis. For the past few months, it has been working very poorly to be honest. Comment posts that were very obviously spam were getting though consistently and I have been removing 20-30 spam posts en masse about once a week.

I submitted a ticket when I first noticed this happening with Mollom support and got a quick reply, but after my initial replies the request was waiting a follow-up from them for quite some time (and twice, at that).

Today I got news that it was a service-wide issue and it has been resolved. Hopefully you will notice a significant reduction in spam posts! Honestly I am not very happy with Mollom at the moment and may look to other alternatives if this doesn't pan out, but then again it's a free service and it worked wonderfully in the past so I can't complain. You get what you pay for.

The issue was more that I was looking at taking 5-10 clients to their basic paid plan, but after my experience with their customer service I am hesitant to do so. As always, I guess time will tell.

Rating: 

Bootstrap responsive theme

I have been reading a lot of buzz around Twitter's Bootstrap responsive theme and although I am a huge fan of the Omega theme, I think bootstrap has a lot of potential and wanted to experiment with it.

My blog is long overdue for a facelift, so I now present to you my Bootstrap-ized blog! At the moment it is simply using the base theme, but I will be implementing a customized sub-theme shortly. I am interested to hear what you think (and if it does or doesn't work on mobile devices) in the comments!

Update 2013-02-08: I have performed minimal customization to the theme and after working with it for a bit I can start to see Bootstrap's strengths and weaknesses. Although Omega is very flexible and has a bunch of responsive options, I feel like they could use a little more polish; Bootstrap's responsive design implementation feels so quick and smooth and it also scales well at all viewport sizes. I also like Bootstrap's barebone styles (font, text layout, etc) much better than Omega's defaults which I believe use serif fonts (*ugh*). That said, such a thing is very easy to change in CSS and so it's not a huge disadvantage for Omega.

Where Omega also beats Bootstrap (and by a large margin) is when it comes to theme customizability. The Bootstrap theme settings page offers few options and in several instances I found that several common Drupal-specific CSS rules (such .indent) were missing entirely. Omega has been around for quite some time and as a result has excellent integration and provides a plethora options to modify the zone, region and block grid right the theme's settings page.

If I had to do it over I would stick to Omega, but Bootstrap will be giving it a run for its money once it picks up on some of the grid customization tricks Omega has.

Rating: