Talk Like A Duck

Keeping it up (Your Rails App that is)

Posted by Rick DeNatale Thu, 02 Aug 2007 16:23:00 GMT

I live in the fastest growing corner of one of the fastest growing counties in the US. Lately that has seemed to come with more than my share of power interruptions.

This blog is hosted on a ‘server’ in my home office, which also hosts several other things, some for internal use, and others, like talklikeaduck are public facing. Besides this blog I have a wikimedia based site on the technical history of Project Mercury, a personal interest of mine.

One of my frustrations of late has been the fact that the blog doesn’t suffer those power interruptions well. For deployment, I’m using a combination of Apache 2.0, with pen for load balancing, and mongrel_cluster. That last has been the main source of that minor annoyance. The version of mongrel_cluster I’ve been using doesn’t properly deal with stale pid files, so that when the system reboots after a power outage, the blog doesn’t come back without manual intervention.

My frustration level finally rose to the point where I started to look for a solution.

Stable vs. Backwater

The sharp eyed might notice that I’m using Apache 2.0 instead of the later versions which support mod_proxy_load_balancer. The reason is that the server is running Ubuntu Dapper which doesn’t have the latest Apache, and since I have so much other stuff running, I haven’t wanted to attack that problem. Don’t ask me about my php4 to php5 platform migration plans, either. Wikimedia moved on to PHP5 some time back, and I’ve stopped moving with it.

I’m running Dapper since that’s the latest Ubuntu LTS release.

It’s the usual dilemma of wanting a stable server vs. being left behind.

For things like Ruby, where I’ve got the time and interest to spend, I’ve gone to installing from source rather than debian packages. This allows me to keep up, and gets around things like the issues between the views of the debian maintainers and the rubygems folks about how software should be installed.

Searching For Religion

A few weeks ago, I started hearing about god, a new ruby server monitoring framework.

So I installed god, only to find that it wouldn’t install in Ubuntu Dapper. I asked about this on the ruby-talk forum, and the developers seemed to agree with me that it might be nice if, as a server oriented offering, god supported Dapper.

So I’ve been waiting for god to come to Dapper, but with after another power outage this morning, I decided to look at alternatives

God is inspiring, but who inspired god?

God is actually patterned after another non-ruby offering called monit.

Monit has been around awhile, has been packaged for debian, and even is available in packaged form for dapper. It’s also showed up recently in several of the rss feeds I monitor.

I think it’s fair to say that monit:god :: make:rake.

Monit and Mongrel_Cluster

So this morning I installed monit, and started to configure it.

While in the midst of doing this I started wondering how best to configure it to monitor a mongrel cluster. Monit is a daemon which periodically examines things like pid files and sees if the processes which they reference are still running. It can be configured to automatically restart services when they die, restart them under certain load conditions, etc.

The problem is that mongrel_cluster is really responsible for multiple processes, and I was scratching my head over how to map this to monit for a few seconds before resorting to google. I noticed that, as I predicted, others had been there or in similar places before.

Aha! There’s a new Mongrel_cluster

While perusing the words of wisdom from the pioneers, I noticed that those who had setup monit to monitor clusters of mongrels were mentioning the need for version 1.0.1 So, as directed I did a

sudo gem install mongrel_cluster --source http://mongrel.rubyforge.org/releases/

But when I tried it out, with

mongrel_rails cluster::status

At this point I couldn’t restart my blog, so I uninstalled the 1.0.1 version of mongrel_cluster, tried to start it again and, relieved when it did start, looked for a solution.

My first thought was to see if maybe there was an even later version of mongrel_cluster. Doing a gem list—remote, showed that there as a 1.0.2 in the normal source. So I installed that, and, same problem.

Then, with a little help from google, I uninstalled the earlier version of mongrel_cluster, and lo and behold, the new cluster commands worked.

Permissions problem

But then I ran into another problem. Starting the blog was thowing an exception because of a permissions problem. I use an init script to start mongrel_rails, and mongrel_rails runs as a particular user. The mongrel_cluster init file looks for existing pid files, and it uses a sequence which changes to the working directory specified in the mongrel cluster configuration, looks for the file, then changes back to the old working directory. But the user running it didn’t have permission to my home directory which was I started, so it failed when on that change back.

The solution was to change myinit script to change to the rails root of my typo installation before invoking mongrel_rails. Problem solved.

Still waiting for god(ot)

So what of monit?

Now that I’ve got the new version of mongrel_rails, which properly handles stale pid-files, I’m less motivated to setup monit. I do like the monitoring functions it provides, so I might still pursue it, but on a less urgent priority.

On the other hand I just might wait for god to bless Dapper, or for the next Ubuntu LTS release, whichever comes first.