Installing Passenger on Ubuntu 8.10

A bit of help from my friend google, led me to this how-to which I followed. I'm not entirely happy with the approach, since it installs Ruby using the debian package, and I generally prefer to install it from source. I also thought that it installed Rubygems from a package, but looking at it now it seems to have installed it from source, but installed in in /usr/bin which may not be the wisest place on a debian system, it really should be in /usr/local/bin or somewhere else that dpkg/apt-get doesn't muck with. I might need to straighten that out.

I had to hack his scripts a bit as well. They assume that the current directory is on the path, which is something I don't like to do, so I went through and added './' where necessary.

Finally, his script for upgrading to enterprise Ruby for improved performance didn't work, so that's a job for later.

I posted a few comments on the article about this, but as of now they are waiting for moderation.

Typo Upgrade

For those who haven't used it typo comes a a gem which provides a typo command, you use it to either install typo

typo install typo_dir

Either installs the latest version of typo, which is really just a rails app, in typo_dir, or upgrade an existng version.

One of the first things the upgrade does is to backup the database to a yml file. For a blog like mine with a lot of content, this took forever. As I sat there waiting, it occurred to me that I already had a mysql backup, which I'd already used to recreate the database, so I killed the upgrade and hacked the rails-app-installer gem (which typo uses) to skip the backup.

The next hurdle was a gem version problem. I got to the point where the upgrade script was trying to migrate the database, but Rake was failing

	Migrating Typo's database to newest release
	rake aborted!
	RubyGem version error: actionpack(1.13.3 not = 1.13.6)

This was particularly strange since Typo 5.2 runs on Rails 2.2. Somebody was loading the wrong version of rails, and then complaining about the wrong version of actionpack. And the 1.13.6 version of the actionpack gem doesn't seem to be around anymore, probably left behind when Rails moved to github.

So I asked on the typo mailing list, and my former co-worker Ben Burdick gave me the solution:

	sudo gem install datanoise-actionwebservice --source http://gems.github.com

This got me up to the point where I could run the blog on the MacBook. I used the very nice Passenger pane, you just drag the folder containing your rails project into the pane, and restart, and your app is running! Since I wanted to urls to match up. I told passenger pane to use a url of talklikeaduck.denhaven2.com instead of something.local, and put an entry into /etc/hosts to point that name to local host.

Once I was there, it was a matter of recreating the articles which had been lost since the last backup. Mark Imbriacco emailed me what he had cached for each article in his RSS reader. It turned out to be a pretty simple job of cutting and pasting, and telling Typo to publish the articles on their original dates which could be determined by the permalink urls. I didn't worry about the exact time.

I then did a mysqldump of the database, shipped it over to the server with scp, and recreated the database on the server.

Capistrano Deployment

Now the task was setting up Capistrano and having it get the code from git. For the time being I set up a git repository in my home directory on the server, and pushed the code to it from the MacBook over ssh.

The next step was to allow the apache user (which is www-data on debian based systems) to access the repo. This was a matter of generating a public/private key pair in the www-data users home directory, adding the public key part in the .ssh/authorized_keys file in MY home directory, and a

	sudo su - www-data
	ssh rick@localhost
	# answer yes to the prompt to add the host being connected to
	exit

I then upgraded my capistrano gem and capified my typo project. Then I edited the config/deploy.rb based on a couple of blog posts and some trial and error.

  set :application, "talklikeaduck" 
  set :repository, "ssh://rick@aaa.bbb.ccc.ddd/home/rick/git/tlad.git"
  set :scm, :git
  set :branch, "master"
  set :deploy_via, :remote_cache
  set :user, "www-data"
  set :runner, "www-data"
  set :deploy_to, "/var/www/rails/#{application}"

  role :app, "aaa.bbb.ccc.ddd"
  role :web, "aaa.bbb.ccc.ddd"
  role :db,  "aaa.bbb.ccc.ddd", :primary => true

  namespace :deploy do
    desc "Restarting mod_rails with restart.txt"
    task :restart, :roles => :app, :except => { :no_release => true } do
      run "touch #{current_path}/tmp/restart.txt"
    end

    [:start, :stop].each do |t|
      desc "#{t} task is a no-op with mod_rails"
      task t, :roles => :app do ; end
    end
  end

  desc "Link in the production database.yml" 
  task :after_update_code do
    run "ln -nfs #{deploy_to}/#{shared_dir}/config/database.yml #{release_path}/config/database.yml" 
  end

Note that I've replaced the lan address of the server with aaa.bbb.ccc.ddd, it probably wouldn't be a security exposure to show the real address behind the nat, but paranoia is paranoia. Normally I'd use a local dns server name, but I haven't gotten around to setting that back up.

Getting this working was a matter of trying cap deploy:check until I ironed out permissions issues, usually by sshing into the server creating files using sudo, then chowning them to www-data.www-data, since www-data doesn't have sudo privileges.

Then I worked on getting cap deploy:cold to work. More sshing, mkdiring, and chowning to set up the current, releases and shared subdirectories. I'd scratched my head a bit about what to do with config/database.yml which I'd gitignored as usual. Ben pointed me to a blog article which added a post setup task to upload it to the shared directory, but it turned out to be easier to just do it via scp and ssh.

If I recall correctly, the next hurdle was familiar, cap:deploy cold was failing for the same reasons as on the Mac, the gem version problem killing Rake. The solution here was to rake gems:push, git a

	rake gems:unpack
	git add vendor/gems
	git commit
	git push origin
	cap deploy:cold

Now I ran into a problem because I'd missed a bit of code in the deploy.rb from the article I was using a model. It was this piece:

[:start, :stop].each do |t|
  desc "#{t} task is a no-op with mod_rails"
  task t, :roles => :app do ; end
end

Without this capistrano was trying to get the www-data user to sudo to run script/spinner. Before I noticed the omission I was considering writing scripts to stop the server by disabling the virtual host, and reloading apache, and starting in in a similar manner, and giving www-data limited sudo privileges to run those scripts. It turns out that this is unnecessary using Passenger, so it was just a matter of overriding the default capistrano tasks to do nothing.

So now there was joy in Mudville. The cap deploy:cold worked. At this point I created a vhost configuration in /etc/apache2/sites-available, used a2ensite to enable it, and reloaded apache. Now to check it out.

One Last Wafer Thin Mint

Back on the MacBook, I edited /etc/hosts to now point to the servers lan address. I still wasn't ready to expose port 80 of the server to the internets.

I cranked up my browser and reloaded the url. Everything was working! But being the suspicious type, I wanted to make sure. Tailing the log on the server showed no acivity, hmmmm?

I pinged talklikeaduck.denhaven2.com on the MacBook, and it was pinging 127.0.0.1. I knew that I'd changed /etc/hosts. Ben told me over ichat to try dscacheutil -flushcache, which seemed to have no effect. So I rebooted the MacBook, and it was still resolving talklikeaduck.denhaven2.com to localhost.

At that point I tried:

$ dscacheutil -q host 127.0.0.1

Which produced:

name: talklikeaduck.denhaven2.com
ip_address: 127.0.0.1
name: localhost
ip_address: 127.0.0.1
name: talklikeaduck.denhaven2.com
ip_address: aaa.bbb.ccc.ddd

So there were two cache entries, and the local host one was winning.

I finally got a clue when I grepped /etc for references to talklikeaduck, and found one in /etc/hosts, but also one in /etc/apache2/sites-available/tlad which was the vhost configuration generated by Passenger pane. When I removed the blog project directory from the Passenger pane, and hit restart, then did the dscacheutil -flushcache, the name was now resolving to the server.

Now I was hitting the server, and I did a little more prodding to make sure things were working, including writing my resurrection announcement.

Now I was finally ready to unblock port 80 on the router.

Hopefully someone will find something useful in this post, at least I've captured the lessons I learned from the exercise.

Mark, bless his heart, offered up a machine he was no longer using, and the process of rebuilding began.

Luckily, I had put a graphics card in the old system, did I tell you that it wasn't just a server? I used to use it as a development workstation as well. This was lucky because Mark had used the new system as a gaming machine, and it had a nice gamer's graphics card installed, whose on board fan was hanging limply from it's attachment. I invested in a 500GM SATA drive for the install, I wanted to bring the system up before touching the drives from the old server.

I'd been holding off updating the system, which was running on Ubuntu 6.06 "Dapper Drake" for nearly 3 years. I'd fallen for the "Long Term Support" pitch, which had the effect of leaving me behind on things like Apache. So this seemed like the right time to take the step up. I decided to just use the latest Ubuntu 8.10 release.

Once the system was installed, it was time to examine the wreckage. It turned out that one of the two 160GB drives had gone bad, or had perhaps been corrupted by the controller. One of the two drives held the home directories, and various other directories, the other was used to hold a nice rsync backup set, kind of a home-built Time Machine using techniques I'd learned at the local Linux user's group. Right after the crash, I thought that it was actually the backup drive which had gone bad, since the first time I booted, I did seem to have access to the data on the 'live drive.' But now linux couldn't find the superblocks, and all I had was the backup.

Two minor problems. The first was that I discovered that the last good backup I had was from sometime in January, so I was going to lose some data. The second was that my nice backups was full of files whose permissions were based on the /etc/passwd and /etc/groups files on the OLD system. I'd faced this problem back when I'd moved the system from Redhat/Fedora to Ubuntu. Redhat distros tend to assign fixed uid's and gid's for the users associated with server processes, while Debian based systems assign them when packages are installed. Reconciling the differences is a major pain, and most of the data is still on the backup drive waiting for me to slog through it.

What I did have were good copies of my database backups (at least until sometime in January, and my local code, like the instance of Typo running this blog. I decided to reconstruct the blog on my MacBook, upgrade it from Typo 4.1 to the latest version of typo, and make it deployable on the server using capistrano, with git for the scm, and passenger, now that I had a version of apache that was modern enough to handle it. No more of the old rube-goldberg setup of Apache talking to pen for load balancing, talking to a mongrel cluster.

I haven't done so much concentrated system admin in quite a while. I've got to thank Mark, and my other friend Ben Burdick, who used to be the sysadmin and QA guru at Near-Time and now works for RackSpace/Slicehost, who have been subject to repeated iChat sessions, when my googling skills flagged.

Mark also had my blog posts for quite a while cached in his RSS reader, and provided me with the contents of the articles I'd written since the last backup. The few image files I needed were still on my MacBook. So I think I've been able to recover everything from the blog, except for any comments made on those missing posts.

Some of the things I've learned from this episode are:

• How to get the apache user to ssh back to the same server to access git.
• How to configure capistrano using git and passenger instead of subversion and mongrel/mongrel-cluster
• That running apache/passenger using the passenger pane on os x, is cool. It let me easily check out the blog. But, I had a final head-scratching moment. I'd changed /etc/hosts on the Mac so that talklikeaduck.denhaven2.com would resolve to local host, which worked great. But when I changed it to point to the lan address of the server, the Mac stubbornly refused to resolve it to anything but 127.0.0.1. I was chatting with Ben during this episode, and he told me to use

      dscacheutil --flushcache
    

  to clear out the Mac's dns cache, but this didn't work, even after a reboot . Ben just kept saying "That's wierd." I was thinking that some other configuration file must be causing this, so I tried:

    grep -R talklikeaduck /etc

  Which revealed that the only other place the word occurred was in the VHost configuration file which the Passenger pane had generated. I tried removing the blog entry from the Passenger pane and restarting apache, and lo and behold, after clearing the cache, the host was now resolving to the server!

You'll notice that the blog has a new style. The old theme wasn't compatible with Typo 5.2, and getting the blog back up was a higher priority than porting it. Actually I like the new theme, and although I kind of miss old Ruby the Duck Dog. I suspect that he'll she will be coming back in a new form when I get the time to do it.

So having written this first article after the great crash of 2009, I think I'm ready to once again expose the server to the inter-tubes.