Talk Like A Duck

Lately, most of the web development community, at least the parts I see regularly seems to be swept up in the NoSQL wave. Some of use seem to be completely eschewing relational databases in favor of newer technology like mongoDB, couchDB, Riak, the project which must not be named, or one of the many such NoSql databases which are au courant.

Of late, I had been working on a project for a client which had me swimming against that trend. My task was to convert a sizable Rails application from mongoDB to SQL.

It is tempting to see a new shiny technology and wield it as if it were Maslow's Hammer.We see it, find it attractive, and attempt to use it as a panacea. We might convince ourselves that because an application has a "Document" model, that a "document" oriented database like couchDB or mongoDB, is more appropriate than SQL. Maybe it is, and maybe it isn't.

A lot of developers seem to look for "best practices," or "best practice patterns," a set of templates for code or processes which will ensure success. The world just ain't that simple. In reality we face a series of choices each with different characteristics and consequences. Rather than a single Maslovian hammer, we need to have a fully stocked toolbox, and consider when we should apply each tool. The job of design, whether it is designing software, houses, or bridges, is to solve a system of conflicting, and reinforcing forces.

Recently, my friend David Chelimsky remarked "For me, there is only one best practice: think."

The 1980s and early 1990s saw a lot of discussion about the relative merits of relational vs. object-oriented databases. There is a natural tension between traditional databases which strive to separate the data representation from the application code, and object-oriented programming which strongly couples data representation and object methods, and loosely couples objects. This was often called an "impedance mismatch". In the Smalltalk community we saw the genesis of object-oriented databases like Gemstone/S which majored on making the graph of objects persistent, with transactional semantics.

We also saw the advent of object relational mappers. My first encounter with an ORM was TOPLink. This work provided the seeds for today's implementations like ActiveRecord. An ORM acts as a kind of "impedance matching transformer" if you like.

"Big" Dave Thomas used to make the analogy back then that some data was like a corn field, where a combine harvester(SQL) was appropriate, while sometimes you really wanted a Japanese garden, and therefore different tools.

When today's NoSQL movement came up on my radar, my history led me to think of it as the revival of object-oriented databases. The more I look at it, it is becoming clearer that it is not.

Most of the NoSQl movement is trying to solve a different problem than an impedance mismatch between the database and objects. The seminal events which led to today's NoSQL movement seem to be:

1. The announcement in 2000 of Eric Brewer's conjecture which led to...
2. A proof of the conjecture in 2002 by Seth Gilbert and Nancy Lynch, after which the conjecture became known as Brewer's CAP theorem.
3. The 2007 publication of an academic paper on Amazon's Dynamo key value store, which applied a combination of known technologies, to provide a distributed data store which allows for high availability in the face of failing network components.

The key problem is how to create large scale distributed data stores. Brewer's CAP theorem proves that you can not have a combination of Consistency, Availability, and tolerance of network Partitioning, due to component failures. Dynamo trades off consistency for availability in the face of an unreliable network. The use-case presented in the paper is managing Amazon shopping carts.

Relational database systems tend to strive for systems which prioritize consistency over availability and partition tolerance. Features such as two-phase commit allow the implementation of ACID transactions which are atomic (they either succeed totally or fail totally) by ensuring that a transaction either fails when a network failure is detected, or wait for communications to complete before making the results of a transaction globally visible.

More recently it has been pointed out that there is an asymmetry between the three legs of the CAP stool. There seems to be another conjecture here. CAP implies you can build a system which is consistent, and available, but does not tolerate partitioning(CA); a system which is consistent and tolerates partitioning with a sacrifice to availability(CP), or one which provides availability in the face of partitioning by sacrificing consistency(AP). Not tolerating partitioning does not outlaw it though, and the way a CP system reacts to partitioning is to suspend operation until the partitioning is healed, and in practice a CA system does the same thing, it can not tolerate a partition, so it makes the system unavailable to maintain consistency.

Also the realization has come that there is another leg, which is latency. Actually I think that partitioning and latency are related. Network partitioning is really just an drastic increase in latency isn't it? The NoSQL guys talk about "eventual" consistency, which means that the distributed datastore will reach consistency eventually after a partitioning (or after an abnormally long latency message).

I think that the reason for this difference between practice and theory is that whether or not the data store provides consistency makes a big difference to the programming model. An application which is written to deal with the possibility of inconsistency, or eventual consistency, must of necessity be designed differently than one which can rely on the data store to provide consistency .

There is another pitfall for developers used to relational data modeling when they start to use NoSQL approaches, which is what to do about 'relations' between objects. For the most part, and I might be wrong here, NoSQL systems leave you to roll your own here, usually by either putting explicit hash-key links between value blobs in the key-value store, even though those blobs are probably mapped by JSON or the like, or to "de-normalize" the data by copying it, which leads to the additional problem of how to deal with updates. It is not that these are necessarily insurmountable problems, just that solving them requires thinking them through with fresh eyes.

Which brings me to the title of this little essay. To some it appears that some are taking the No in NoSQL to mean just that, "no SQL." In reality, the proponents seem to saying that it stands for "not only SQL." Whether to use relational or NoSQL data technology for all or part of an applications data is not an all or nothing proposition.

It seems to me that the strength of NoSQL is in meeting non-functional requirements, it makes it easier to scale out systems or parts of systems which do not need cross-user ACID characteristics but need to have lots of distributed parallel processing to achieve high throughput and high availability. There are certainly parts of many systems which have those characteristics. Things like shopping carts and other user session data (the use case for Dynamo) do not really need to be consistent globally. A weaker form of consistency, consistency viewed by an individual user is often enough, and is where NoSQL systems tend to focus their efforts, by providing features like "read what you wrote."

The application needs to deal with what happens when consistency is lost or delayed. This is not unlike what we do with optimistic locking in a traditional database, but I would posit that it is not exactly the same thing.

For other parts of the system, like the transactions needed to turn a shopping cart into an order, or credit/debit transactions (e.g for payment or inventory control), an ACID transaction semantic is needed. Building such a semantic on top of a NoSQL base is problematic. Building these parts using a more traditional relational approach seems to make sense.

So the lesson I take from this is that although I want to have NoSQL in my toolbox for use when appropriate, I don't want to see it, or any other tool, as a Maslovian hammer.

The Time Machine!
© Chaotic Good01

I've been contemplating improving my backup strategies lately.

Last year, I signed up with Backblaze to back up my MacBook to the cloud. There are a few players in this arena, another is Carbonite.

But off-site backup is only one leg of the backup stool. The wisest course for backup is a 3, 2, 1 strategy. That is to have three copies of your data, on two media (i.e. at least one of the copies should be on a different medium than the others), and one of the three should be offsite.

I consider on-line backup (a la Backblaze) to be a different medium, where rather than the medium just being a hard disk it's the whole off-line backup company's infrastructure.

Offline/offsite backup is really targeted for disaster recovery, if the building housing my on-site backups was destroyed by fire, or some other calamity, along with those backups I could recover the data from backblaze, but it would take a while to get it, either by downloading or getting them to ship me an external hard disk.

And for certain important data, I also use DropBox which can not only save data to the cloud, but sync that data between multiple machines. To make sure I've got a copy of the contents of this blog, I have a cron job on the Linux server where it runs which does a sql dump of the database to a dropbox folder, along with any new files (images, documents, etc) used by the blog. This runs in the wee-dark hours, and every morning, my MacBook informs me that DropBox has copied that data down to my MacBook.

But I've been feeling a gnawing need for better on-site backup, with faster access for either full recovery, or to get back a mistakenly deleted, or corrupted file.

One requirement is that the backups be done automatically. I've got too many other things on my mind than to remember to backup frequently, that's what I have computers for. Also I want the backups of my Macbook to work anywhere it can access the LAN. I tend to use my MBP in various areas of the house, upstairs in my office, where it's next to the server, and can be jacked into the same ethernet switch as the server, but also in the family room and even the bedroom, where it's connected by wi-fi.

TimeMachine

I've been wanting to use Apple's TimeMachine for a while, but haven't had the necessary hardware. I'd been thinking about buying a TimeCapsule, but I've been just a little leery since several of my friends have had their TimeCapsules die within a year.

I did a bit of research, and came up with the idea of looking at using my Linux server to hold a TimeMachine backup. The box has several available drive bays, and I could get a 2TB WD SATA drive for $120. With the cable and tax it came to just around $150.

But first I wanted to make sure that it would really work.

Existing How-Tos

I asked my friend Google about TimeMachine and Ubuntu and the first hit was a detailed how-to on kremaliscious.. Unfortunately that how-to is about two years old. Fortunately it still pretty much applies, but things have been simplified a bit.

That how-to, which was last updated when the latest Ubuntu was 8.04, "Hardy Heron" and the latest OS X was still 10.5.x, "Leopard", describes building a custom debian package for netatalk to add support for ssh connections. Netatalk is an open source implementation of Apple's AppleTalk Protocol Suite. I also encountered a much more succinct article by my old OTI friend Andrew "Roo" Low, which did the same.

For context, I'm running Ubuntu 10.04, "Lucid Lynx", on the Linux box, and OS X 10.6.4, "Snow Leopard" on the MBP.

A comment near the end of a long string of comments in that first article indicated that a custom netatalk package was no longer necessary, since the standard debian/ubuntu netatalk package now has ssh support. But following the rest of the instructions got me to the state where my Mac could see the shared volumes on the server, but when I tried to connect to them, I couldn't authenticate.

Figuring that I really did have to build a custom netatalk package, I followe Roo's article and built one. But it ended up in the same place.

It turned out that the crucial bit was a change to the configuration of netatalk's afp configuration.

The article gives the following configuration line in /etc/netatalk/afpd.conf

  - - transall -uamlist uams_randnum.so,uams_dhx.so -nosavepassword -advertise_ssh

But with the current netatalk this needs to be changed to:

  - - transall -uamlist uams_randnum.so,uams_dhx2.so -nosavepassword -advertise_ssh

This is actually mentioned in comment 330 (of 670!) but I missed it.

After making that change, I could connect to the shares. So I removed my custom netatalk package and installed the standard one, and it still works.

Avahi setup

Netatalk is one half of what's needed to have a Linux box participate in an Apple friendly network. The other half is Avahi which is an open source implementation of the Zeroconf standard or what Apple calls Bonjour. Bonjour/Zeroconf is how Macs find services on the local network.

Most current linux distributions include Avahi, as does Ubuntu 10.04, so it was just a matter of configuring it. The instructions in the kremaliscious article are fine. However, while I was at it, I also configured Avahi to publish a VNC server which allows me to easily start a graphic login to the server from the Mac finder. Bonjour uses the service type rfb (remote frame buffer) for the VNC server. To get this working I followed this article. I combined the configuration of both the afpd and rfb services into a single configuration file in /etc/avahi/services/multi.service

Here's that file:

  <?xml version="1.0" standalone='no'?><!--*-nxml-*-->
  <!DOCTYPE service-group SYSTEM "avahi-service.dtd">
  <service-group>
    <name replace-wildcards="yes">%h</name>
    <service>
      <type>_rfb._tcp</type>
      <port>5901</port>
    </service>
    <service>
      <type>_afpovertcp._tcp</type>
      <port>548</port>
   </service>
    <service>
      <type>_device-info._tcp</type>
      <port>0</port>
      <txt-record>model=Xserve</txt-record>
    </service> 
  </service-group>

File Server Setup

First I configured netatalk to give me a test 'drive' by publishing a directory in my own home directory on the linux box. Once I'd proved to myself that TimeMachine could in fact connect to this, it was time to buy that drive and hook it up.

I make it a standard procedure to install disks on Linux under control of LVM. Here's a pretty good and succinct guide on getting started with LVM on Ubuntu.

So I installed the drive, and set it up as an LVM Physical Volume called FileServer, I then added a couple of logical volumes to the group:

  sudo lvdisplay
    --- Logical volume ---
    LV Name                /dev/FileServer/TimeMachine
    VG Name                FileServer
    LV UUID                VIdN7y-0gDa-1Azr-0oYB-MRj3-Woxt-Utyef8
    LV Write Access        read/write
    LV Status              available
    # open                 1
    LV Size                1.50 TiB
    Current LE             393216
    Segments               1
    Allocation             inherit
    Read ahead sectors     auto
    - currently set to     256
    Block device           251:3

    --- Logical volume ---
    LV Name                /dev/FileServer/MunimulaClone
    VG Name                FileServer
    LV UUID                Y36agh-Tgnk-1dcf-34tw-AJLm-1Mc2-jcqsuk
    LV Write Access        read/write
    LV Status              available
    # open                 0
    LV Size                320.00 GiB
    Current LE             81920
    Segments               1
    Allocation             inherit
    Read ahead sectors     auto
    - currently set to     256
    Block device           251:4

I gave TimeMachine 1.5 terabytes to play with, and reserved enough to hold a snapshot of my laptop drive in MunimulaClone. Munimula is the name I gave my MacBook pro, inspired by it's construction and a childhood memory. I then installed an ext3 filesystem in both of these logical volumes.

I set up /etc/fstab to mount these in to mount points in the /var directory

  /dev/FileServer/TimeMachine /var/TimeMachine ext3
  /dev/FileServer/MunimulaClone /var/MunimulaClone ext3

Following Roo's suggestion I created an empty file “/var/TimeMachine/.com.apple.timemachine.supported” with the touch command, and on the Mac I executed the command:

  defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Then I added the following two lines at the end of /etc/netatalk/AppleVolumes.default to publish the two network volumes:

  /var/TimeMachine TimeMachine allow:rick cnidscheme:cdb options:usedots,upriv
  /var/MunimulaClone MunimulaClone allow:rick cnidscheme:cdb options:usedots,upriv

I did run into the problem encountered by others of TimeMachine giving an error that it couldn't create the backup disk image, but IIRC this only happened once, it created a file called munimula.tmp.sparsebundle, and removing the .tmp from the file name seemed to allow TimeMachine to proceed.

Results

With the MBP connected via an ethernet cable through a switch to the linux box, the first backup (about 290GB) took around 11-12 hours. It had just finished when I checked this morning. The second 'hourly' backup took just a few minutes. I then disconnected the hard-wired connection and brought the laptop downstairs.

It didn't seem to automatically start a new backup cycle over wifi when it should have, but getting it started by holding the mouse down over the TimeMachine icon in the dock until the menu appeared and then selecting "Back up now" got things kicked off, and it's been taking hourly backups wirelessly all day so far. The hourly backups take a bit longer, may 10-15 minutes. I'm not sure if that's because of lower bandwidth over wifi, or just because I've been using the laptop, and there's more delta data than there was first thing in the morning.

So far I'm pretty happy with the setup, I'll update this article should any surprises occur, good or bad.

I'm working a project which uses RSpec, Bundler, and Rails 2.3.4.

The team has been struggling with the changes in gem management with Bundler. The "Old School" rails way to configure gems for different rails environments is to conditionally execute gem.configure based on the rails environment. Then use the rake:gems:install task to gem install the gems, one for each environment.

Bundler does things a bit differently. It uses groups to segregate gems by environment, so you name the gems needed for the test environment in a :test group. When you run bundle install, it installs all of the gems for all of the groups (unless you opt out of some groups. It uses the groups at run time, to selectively expose only gems applicable to the current environment.

This cause problems for things like the spec rake task. If you run

without explicitly setting the Rails environment.

The way the spec rake task works is that it depends on a rails provided rake task called :environment, which 'boots' the rails environment, then the spec_helper file which each spec whould require, sets ENV['RAILS_ENV'] to 'test' if it's not already set, then it requires needed gem code, like 'spec/rails'.

In the old school approach this works, since those gems are installed and visible. With bundler, it fails since they won't be exposed, since the bundler environment got set up using the development environment, so gems in the :test group aren't available.

In an attempt to fix this, other members on the team were mixing things up, doing things like putting conditional tests of the rails environment in the Gemfile, and then running the bundle command with different overrides of the RAILS_ENV environment variable.

But that way lies madness.

This morning I worked out a solution which involves deferring setting up the bundle environment.

1. I needed to keep the spec task from running the environment task. The project was already using a variation of override_rake_task, so I added an override in a task within the lib directory:

         override_task :spec do
           Rake::Task["spec:original"].execute
         end
       

2. The next step was to bootstrap the rails environment in spec_helper. To do this I started it with:

         ENV["RAILS_ENV"] ||= 'test'
   
         RAILS_ROOT = "#{File.dirname(__FILE__)}/.." unless defined?(RAILS_ROOT)
         environment_path = File.expand_path(File.join(RAILS_ROOT, 'config', 'environment'))
         require(environment_path)
           require 'spec/rails'
           #... any other needed requires
       

   It's crucial to expand the path which ensures that the same string is used whenever the environment file is required. Before I did that I was having problems with the environment being required a second time, presumably for the second spec. If you have any code which patches anything using alias_method_chain, or a similar technique, loading that code a second time can cause infinite loops which can be mystifying.

So this seems to be working. I'll try to update if I find anything else, but in the meantime I hope some folks find it useful.

Angel Dance
© Ina Centaur

Recently, Yehuda Katz wrote an article in reaction to a Pythonista's criticism of Ruby and in particular to the complaint that you can't call a proc with parentheses:

  my_method = Proc.new {1 + 2}
  my_method()

Yehuda did a great job of defending why this is consistent with the rest of Ruby, and talked about how blocks in Ruby are really used. As part of the article, he conflated blocks and Procs. The difference is rather subtle, and in most cases can be ignored. But Giles pointed out that the difference is there, and is documented both in the Pickaxe and in David Flanigan and Matz's "The Ruby Programming language" and I posted a comment in support of Giles after Yehuda pushed back maintaining his view that there was no difference.

Yehuda followed up with another article, Which prompted this article.

Let me start by saying that I'm not interested in starting a war here. First, I've nothing but respect for Yehuda, and I'm very impressed by his body of work, particularly the refactoring of Rails from Rails 2 to Rails 3. Second, I agree that most of this stuff really doesn't matter all that much, and only when discussing things at a rather deep level. It can be like a debate about how many angels can dance on the head of a pin! But such discussions can be fun and in rare instances even enlightening.

Proc Identity

Here's Yehuda's example from the second article:

    def foo
      yield
    end

    def bar(&block)
      puts block.object_id
      baz(&block)
    end

    def baz(&block)
      puts block.object_id
      yield
    end

    foo { puts "HELLO" } #=> "HELLO"
    bar { puts "HELLO" } #=> "2148083200\n2148083200\nHELLO"
  

He points out that the identity of the proc object bound to the two arguments named block in the bar and baz methods doesn't change. He then gives a slightly different example:

  def foo(&block)
    puts block.object_id
    yield
  end

  b = Proc.new { puts "OMG" }
  puts b.object_id
  foo(&b) #=> 2148084040\n2148084040\nOMG

and then proposes two "mental models" of what's going on:

1. The &b unwraps the Proc object, and the &block recasts it into a Proc. However, it somehow also wraps it back into the same wrapper that it came from into the first place. or...
2. The &b puts the b Proc into the block slot in foo’s argument list, and the &block gives the implicit Proc a name. There is no need to explain why the Proc has the same object_id; it is the same Object!

He then says that the first actually represents the MRI implementation. Based both on a reading of Flanigan and Matz and the MRI code, I respectfully disagree, the truth lies in the middle

The first thing to realize is that the use of & before the last argument of a method definition is not the same thing as the use of & before the last parameter value in a method invocation. The

An analogy with Splat

Ruby has a similar prefix *, a.k.a splat, or what David Black likes to call "the unary unarray" operator. Actually depending on where it occurs * is either an "unarray" or a "make array" operator.

Consider this code:

  array = [1, 2, 3]

  def three_args(a, b, c)
    "a is #{a}, b is #{b}, c is #{c}"
  end

  a, b, c = *array

  a # => 1
  b # => 2
  c # => 3

  three_args(*array) # => "a is 1, b is 2, c is 3"

  def glob_args(*args)
    args
  end

  x, *array2 = 4, 5, 6

  x      # => 4
  array2 # => [5, 6]

  glob_args(:now, :is, "the", Time) # => [:now, :is, "the", Time]

As I said above, * is an unarray operator when it appears either in the right hand side of an assignment, or in the argument list of a method invocation. It acts as an operator which coalesces multiple values into an array if it appears on the left hand side of an assignment or in front of a formal parameter in a method

One important consideration in the use of both splat and proc arguments, is that the method definition doesn't "know" whether the method will be invoked like this:

  foo  # with no block
  
  # or this:
  
  foo { 1 }
  
  # or this:

  foo(&b)
  
  or even this:
  
  proc = Proc.new {1}
  foo(proc)

All of these are valid, and need to work.

Method definition with a & Formal Parameter

In Yehuda's last quoted example this is the def foo case.

Method Invocation with a & arg

Two things happen when a method is called with a & argument. The cases of calling the method with either an implicit block, an explicit value for the argument, or no argument for the block at all all need to be handled. And the cases of explicitly calling the argument e.g via block.call, and yielding to the block both have to work.

So the &block argument in the definition of foo means that the method prelude ensures that the block argument will either be nil or will refer an object which is a Proc, or at least appears to be a proc. It does this at the entry to the invocation of the method.

It also must ensure that yield semantics will work whether the method was called with an implicit block, or an explicit value for the parameter. Note that in MRI, block yield works by reference to a field called iter in the current stack frame, which in Ruby 1.8 is used to find the node in the abstract syntax tree of the method which defined the block which corresponds to that block. And yield is implemented by evaluating the subtree of the AST rooted at that node. Of course the representation in YARV in Ruby 1.9 represents executable code differently but the effect is the same. The yield keyword deals with the internal "VM" representation of the executable ruby code directly without surfacing it as a Ruby object. This is the real difference between a block and an instance of Proc.

The draft Ruby standard abstracts this iter field a bit using the notation [block] to refer to a logical stack of blocks in the execution context. The draft distinguishes between block and procs. Since I started writing this article, Avdi Grimm wrote another reaction to Yehuda's second article, which looks at the same issues I'm talking about here from the perspective of the draft standard.

With all that said, here's how that formal block argument is handled when the foo method is invoked

1. If block is not nil, then it sees if a block was given in the method call, and if so, it creates a Proc object which will cause the block to be executed when the proc is called.
2. If block is NOT nil
   1. If the value of block is not already an instance of Proc send :to_proc to the value (with a guard to see if it responds, but that's a minor implementation decision to avoid having the overhead of catching a MethodMissing exception). This is why defining Symbol#to_proc allows writing things like (1..10).map(&:succ).
   2. Set up the VM so that yield will work should that happen. In Ruby 1.8 this involves correctly pushing an iterator onto the stack frame, I haven't read through the YARV implementation but I'm sure that it has the same effect.
3. If the object referenced by block isn't a proc or convertible using #to_proc, to a 'proc-like' duck which can quack to the tune of #call, then a TypeError is raised.

Note that this isn't wrapping the argument with a proc, it's ensuring that we have an object which acts as a proc, if it needs to be 'cast' into a proc it will be, but if it's already a real Proc, or it responds to to_proc by returning self, then it will be the same object.

Method invocation with a & prefix on the last argument value

This part is a bit simpler. The argument itself is just passed through. The trick is that in the process of invoking the method, the sending code must do the same thing as step 2.2 above in case the method does a yield.

Does it Matter?

Barring any embarrassing mistakes on my part, this is what happens in MRI ruby, as described in section 6.4.5 Block Arguments of Flanigan and Matsumoto, as well as my reading of the MRI code.

Yehuda makes the point that this is all pretty invisible to the Ruby programmer, and he's right. It would seem that a Ruby implementation could ALWAYS turn blocks into procs and not have a separate hidden iter structure in the VM. One reason for not doing so is performance. Since Procs are closures and capture the bindings of any variables in their scope, there is some overhead to their creation and destruction, if a block is only accessed via yield, then it's guaranteed not to have a lifetime past the return of the called method. So this is an optimization. And such optimizations are known in other dynamic language implementations. Smalltalk gives the illusion of uniformly using closures to represent blocks, but most implementations cheat and recognize cases where the overhead of creating a closure can be avoided. In some cases this is invisible to the Smalltalk programmer, but not always.

So although we might know exactly angels are dancing on the head of the pin, or what steps they are doing, the ruby language books and the draft standard let it slip that they are there.

Yesterday I found myself trying to figure how which version of Ruby was the latest on a particular date. I had trouble finding a resource with this information using google, so I posted a query to the ruby-talk forum, and Urabi Shyouhei posted a list derived from the ftp site.

In the interest of posterity, here's the list as of now: