Installing Passenger on Ubuntu 8.10

A bit of help from my friend google, led me to this how-to which I followed. I'm not entirely happy with the approach, since it installs Ruby using the debian package, and I generally prefer to install it from source. I also thought that it installed Rubygems from a package, but looking at it now it seems to have installed it from source, but installed in in /usr/bin which may not be the wisest place on a debian system, it really should be in /usr/local/bin or somewhere else that dpkg/apt-get doesn't muck with. I might need to straighten that out.

I had to hack his scripts a bit as well. They assume that the current directory is on the path, which is something I don't like to do, so I went through and added './' where necessary.

Finally, his script for upgrading to enterprise Ruby for improved performance didn't work, so that's a job for later.

I posted a few comments on the article about this, but as of now they are waiting for moderation.

Typo Upgrade

For those who haven't used it typo comes a a gem which provides a typo command, you use it to either install typo

typo install typo_dir

Either installs the latest version of typo, which is really just a rails app, in typo_dir, or upgrade an existng version.

One of the first things the upgrade does is to backup the database to a yml file. For a blog like mine with a lot of content, this took forever. As I sat there waiting, it occurred to me that I already had a mysql backup, which I'd already used to recreate the database, so I killed the upgrade and hacked the rails-app-installer gem (which typo uses) to skip the backup.

The next hurdle was a gem version problem. I got to the point where the upgrade script was trying to migrate the database, but Rake was failing

	Migrating Typo's database to newest release
	rake aborted!
	RubyGem version error: actionpack(1.13.3 not = 1.13.6)

This was particularly strange since Typo 5.2 runs on Rails 2.2. Somebody was loading the wrong version of rails, and then complaining about the wrong version of actionpack. And the 1.13.6 version of the actionpack gem doesn't seem to be around anymore, probably left behind when Rails moved to github.

So I asked on the typo mailing list, and my former co-worker Ben Burdick gave me the solution:

	sudo gem install datanoise-actionwebservice --source http://gems.github.com

This got me up to the point where I could run the blog on the MacBook. I used the very nice Passenger pane, you just drag the folder containing your rails project into the pane, and restart, and your app is running! Since I wanted to urls to match up. I told passenger pane to use a url of talklikeaduck.denhaven2.com instead of something.local, and put an entry into /etc/hosts to point that name to local host.

Once I was there, it was a matter of recreating the articles which had been lost since the last backup. Mark Imbriacco emailed me what he had cached for each article in his RSS reader. It turned out to be a pretty simple job of cutting and pasting, and telling Typo to publish the articles on their original dates which could be determined by the permalink urls. I didn't worry about the exact time.

I then did a mysqldump of the database, shipped it over to the server with scp, and recreated the database on the server.

Capistrano Deployment

Now the task was setting up Capistrano and having it get the code from git. For the time being I set up a git repository in my home directory on the server, and pushed the code to it from the MacBook over ssh.

The next step was to allow the apache user (which is www-data on debian based systems) to access the repo. This was a matter of generating a public/private key pair in the www-data users home directory, adding the public key part in the .ssh/authorized_keys file in MY home directory, and a

	sudo su - www-data
	ssh rick@localhost
	# answer yes to the prompt to add the host being connected to
	exit

I then upgraded my capistrano gem and capified my typo project. Then I edited the config/deploy.rb based on a couple of blog posts and some trial and error.

  set :application, "talklikeaduck" 
  set :repository, "ssh://rick@aaa.bbb.ccc.ddd/home/rick/git/tlad.git"
  set :scm, :git
  set :branch, "master"
  set :deploy_via, :remote_cache
  set :user, "www-data"
  set :runner, "www-data"
  set :deploy_to, "/var/www/rails/#{application}"

  role :app, "aaa.bbb.ccc.ddd"
  role :web, "aaa.bbb.ccc.ddd"
  role :db,  "aaa.bbb.ccc.ddd", :primary => true

  namespace :deploy do
    desc "Restarting mod_rails with restart.txt"
    task :restart, :roles => :app, :except => { :no_release => true } do
      run "touch #{current_path}/tmp/restart.txt"
    end

    [:start, :stop].each do |t|
      desc "#{t} task is a no-op with mod_rails"
      task t, :roles => :app do ; end
    end
  end

  desc "Link in the production database.yml" 
  task :after_update_code do
    run "ln -nfs #{deploy_to}/#{shared_dir}/config/database.yml #{release_path}/config/database.yml" 
  end

Note that I've replaced the lan address of the server with aaa.bbb.ccc.ddd, it probably wouldn't be a security exposure to show the real address behind the nat, but paranoia is paranoia. Normally I'd use a local dns server name, but I haven't gotten around to setting that back up.

Getting this working was a matter of trying cap deploy:check until I ironed out permissions issues, usually by sshing into the server creating files using sudo, then chowning them to www-data.www-data, since www-data doesn't have sudo privileges.

Then I worked on getting cap deploy:cold to work. More sshing, mkdiring, and chowning to set up the current, releases and shared subdirectories. I'd scratched my head a bit about what to do with config/database.yml which I'd gitignored as usual. Ben pointed me to a blog article which added a post setup task to upload it to the shared directory, but it turned out to be easier to just do it via scp and ssh.

If I recall correctly, the next hurdle was familiar, cap:deploy cold was failing for the same reasons as on the Mac, the gem version problem killing Rake. The solution here was to rake gems:push, git a

	rake gems:unpack
	git add vendor/gems
	git commit
	git push origin
	cap deploy:cold

Now I ran into a problem because I'd missed a bit of code in the deploy.rb from the article I was using a model. It was this piece:

[:start, :stop].each do |t|
  desc "#{t} task is a no-op with mod_rails"
  task t, :roles => :app do ; end
end

Without this capistrano was trying to get the www-data user to sudo to run script/spinner. Before I noticed the omission I was considering writing scripts to stop the server by disabling the virtual host, and reloading apache, and starting in in a similar manner, and giving www-data limited sudo privileges to run those scripts. It turns out that this is unnecessary using Passenger, so it was just a matter of overriding the default capistrano tasks to do nothing.

So now there was joy in Mudville. The cap deploy:cold worked. At this point I created a vhost configuration in /etc/apache2/sites-available, used a2ensite to enable it, and reloaded apache. Now to check it out.

One Last Wafer Thin Mint

Back on the MacBook, I edited /etc/hosts to now point to the servers lan address. I still wasn't ready to expose port 80 of the server to the internets.

I cranked up my browser and reloaded the url. Everything was working! But being the suspicious type, I wanted to make sure. Tailing the log on the server showed no acivity, hmmmm?

I pinged talklikeaduck.denhaven2.com on the MacBook, and it was pinging 127.0.0.1. I knew that I'd changed /etc/hosts. Ben told me over ichat to try dscacheutil -flushcache, which seemed to have no effect. So I rebooted the MacBook, and it was still resolving talklikeaduck.denhaven2.com to localhost.

At that point I tried:

$ dscacheutil -q host 127.0.0.1

Which produced:

name: talklikeaduck.denhaven2.com
ip_address: 127.0.0.1
name: localhost
ip_address: 127.0.0.1
name: talklikeaduck.denhaven2.com
ip_address: aaa.bbb.ccc.ddd

So there were two cache entries, and the local host one was winning.

I finally got a clue when I grepped /etc for references to talklikeaduck, and found one in /etc/hosts, but also one in /etc/apache2/sites-available/tlad which was the vhost configuration generated by Passenger pane. When I removed the blog project directory from the Passenger pane, and hit restart, then did the dscacheutil -flushcache, the name was now resolving to the server.

Now I was hitting the server, and I did a little more prodding to make sure things were working, including writing my resurrection announcement.

Now I was finally ready to unblock port 80 on the router.

Hopefully someone will find something useful in this post, at least I've captured the lessons I learned from the exercise.

Stable vs. Backwater

The sharp eyed might notice that I’m using Apache 2.0 instead of the later versions which support
mod_proxy_load_balancer. The reason is that the server is running Ubuntu Dapper which doesn’t have the latest Apache, and
since I have so much other stuff running, I haven’t wanted to attack that problem. Don’t ask me about my php4 to php5
platform migration plans, either. Wikimedia moved on to PHP5 some time back, and I’ve stopped moving with it.

I’m running Dapper since that’s the latest Ubuntu LTS release.

It’s the usual dilemma of wanting a stable server vs. being left behind.

For things like Ruby, where I’ve got the time and interest to spend, I’ve gone to installing from source rather than
debian packages. This allows me to keep up, and gets around things like the issues between the views of the debian
maintainers and the rubygems folks about how software should be installed.

Searching For Religion

A few weeks ago, I started
hearing about god, a new ruby server monitoring framework.

So I installed god, only to find that it wouldn’t install in Ubuntu Dapper. I asked about this on the ruby-talk forum,
and the developers seemed to agree with me that it might be nice if, as a server oriented offering, god supported Dapper.

So I’ve been waiting for god to come to Dapper, but with after another power outage this morning, I decided to look at
alternatives

God is inspiring, but who inspired god?

God is actually patterned after another non-ruby offering called monit.

Monit has been around awhile, has been packaged for debian, and even is available in packaged form for dapper.
It’s also showed up recently in several
of the rss feeds I monitor.

I think it’s fair to say that monit:god :: make:rake.

Monit and Mongrel_Cluster

So this morning I installed monit, and started to configure it.

While in the midst of doing this I started wondering how best to configure it to monitor a mongrel cluster. Monit
is a daemon which periodically examines things like pid files and sees if the processes which they reference are still
running. It can be configured to automatically restart services when they die, restart them under certain load conditions,
etc.

The problem is that mongrel_cluster is really responsible for multiple processes, and I was scratching my head over
how to map this to monit for a few seconds before resorting to google. I noticed that, as I predicted,
others had been
there or in
similar places
before.

Aha! There’s a new Mongrel_cluster

While perusing the words of wisdom from the pioneers, I noticed that those who had setup monit to monitor clusters of
mongrels were mentioning the need for
version 1.0.1
So, as directed I did a

sudo gem install mongrel_cluster --source http://mongrel.rubyforge.org/releases/

But when I tried it out, with

mongrel_rails cluster::status

mongrel_rails couldn’t find the
cluster::status command.

At this point I couldn’t restart my blog, so I uninstalled the 1.0.1 version of mongrel_cluster, tried to start it again
and, relieved when it did start, looked for a solution.

My first thought was to see if maybe there was an even later version of mongrel_cluster. Doing a gem list —remote,
showed that there as a 1.0.2 in the normal source. So I installed that, and, same problem.

Then, with a little help from google, I uninstalled the earlier version of mongrel_cluster, and lo and behold, the new
cluster commands worked.

Permissions problem

But then I ran into another problem. Starting the blog was thowing an exception because of a permissions problem.
I use an init script to start mongrel_rails, and mongrel_rails runs as a particular user. The mongrel_cluster init file looks
for existing pid files, and it uses a sequence which changes to the working directory specified in the mongrel cluster
configuration, looks for the file, then changes back to the old working directory. But the user running it didn’t have
permission to my home directory which was I started, so it failed when on that change back.

The solution was to change myinit script to change to the rails root of my typo installation before invoking mongrel_rails. Problem solved.

Still waiting for god(ot)

So what of monit?

Now that I’ve got the new version of mongrel_rails, which properly handles stale pid-files, I’m less motivated to setup
monit. I do like the monitoring functions it provides, so I might still pursue it, but on a less urgent priority.

On the other hand I just might wait for god to bless Dapper, or for the next
Ubuntu LTS release, whichever comes first.