When I Say No...

Posted by Rick DeNatale Wed, 04 Aug 2010 20:13:00 GMT
No sql

Lately, most of the web development community, at least the parts I see regularly seems to be swept up in the NoSQL wave. Some of use seem to be completely eschewing relational databases in favor of newer technology like mongoDB, couchDB, Riak, the project which must not be named, or one of the many such NoSql databases which are au courant.

Of late, I had been working on a project for a client which had me swimming against that trend. My task was to convert a sizable Rails application from mongoDB to SQL.

It is tempting to see a new shiny technology and wield it as if it were Maslow's Hammer.We see it, find it attractive, and attempt to use it as a panacea. We might convince ourselves that because an application has a "Document" model, that a "document" oriented database like couchDB or mongoDB, is more appropriate than SQL. Maybe it is, and maybe it isn't.

A lot of developers seem to look for "best practices," or "best practice patterns," a set of templates for code or processes which will ensure success. The world just ain't that simple. In reality we face a series of choices each with different characteristics and consequences. Rather than a single Maslovian hammer, we need to have a fully stocked toolbox, and consider when we should apply each tool. The job of design, whether it is designing software, houses, or bridges, is to solve a system of conflicting, and reinforcing forces.

Recently, my friend David Chelimsky remarked "For me, there is only one best practice: think."

The 1980s and early 1990s saw a lot of discussion about the relative merits of relational vs. object-oriented databases. There is a natural tension between traditional databases which strive to separate the data representation from the application code, and object-oriented programming which strongly couples data representation and object methods, and loosely couples objects. This was often called an "impedance mismatch". In the Smalltalk community we saw the genesis of object-oriented databases like Gemstone/S which majored on making the graph of objects persistent, with transactional semantics.

We also saw the advent of object relational mappers. My first encounter with an ORM was TOPLink. This work provided the seeds for today's implementations like ActiveRecord. An ORM acts as a kind of "impedance matching transformer" if you like.

"Big" Dave Thomas used to make the analogy back then that some data was like a corn field, where a combine harvester(SQL) was appropriate, while sometimes you really wanted a Japanese garden, and therefore different tools.

When today's NoSQL movement came up on my radar, my history led me to think of it as the revival of object-oriented databases. The more I look at it, it is becoming clearer that it is not.

Most of the NoSQl movement is trying to solve a different problem than an impedance mismatch between the database and objects. The seminal events which led to today's NoSQL movement seem to be:

  1. The announcement in 2000 of Eric Brewer's conjecture which led to...
  2. A proof of the conjecture in 2002 by Seth Gilbert and Nancy Lynch, after which the conjecture became known as Brewer's CAP theorem.
  3. The 2007 publication of an academic paper on Amazon's Dynamo key value store, which applied a combination of known technologies, to provide a distributed data store which allows for high availability in the face of failing network components.

The key problem is how to create large scale distributed data stores. Brewer's CAP theorem proves that you can not have a combination of Consistency, Availability, and tolerance of network Partitioning, due to component failures. Dynamo trades off consistency for availability in the face of an unreliable network. The use-case presented in the paper is managing Amazon shopping carts.

Relational database systems tend to strive for systems which prioritize consistency over availability and partition tolerance. Features such as two-phase commit allow the implementation of ACID transactions which are atomic (they either succeed totally or fail totally) by ensuring that a transaction either fails when a network failure is detected, or wait for communications to complete before making the results of a transaction globally visible.

More recently it has been pointed out that there is an asymmetry between the three legs of the CAP stool. There seems to be another conjecture here. CAP implies you can build a system which is consistent, and available, but does not tolerate partitioning(CA); a system which is consistent and tolerates partitioning with a sacrifice to availability(CP), or one which provides availability in the face of partitioning by sacrificing consistency(AP). Not tolerating partitioning does not outlaw it though, and the way a CP system reacts to partitioning is to suspend operation until the partitioning is healed, and in practice a CA system does the same thing, it can not tolerate a partition, so it makes the system unavailable to maintain consistency.

Also the realization has come that there is another leg, which is latency. Actually I think that partitioning and latency are related. Network partitioning is really just an drastic increase in latency isn't it? The NoSQL guys talk about "eventual" consistency, which means that the distributed datastore will reach consistency eventually after a partitioning (or after an abnormally long latency message).

I think that the reason for this difference between practice and theory is that whether or not the data store provides consistency makes a big difference to the programming model. An application which is written to deal with the possibility of inconsistency, or eventual consistency, must of necessity be designed differently than one which can rely on the data store to provide consistency .

There is another pitfall for developers used to relational data modeling when they start to use NoSQL approaches, which is what to do about 'relations' between objects. For the most part, and I might be wrong here, NoSQL systems leave you to roll your own here, usually by either putting explicit hash-key links between value blobs in the key-value store, even though those blobs are probably mapped by JSON or the like, or to "de-normalize" the data by copying it, which leads to the additional problem of how to deal with updates. It is not that these are necessarily insurmountable problems, just that solving them requires thinking them through with fresh eyes.

Not Only sql

Which brings me to the title of this little essay. To some it appears that some are taking the No in NoSQL to mean just that, "no SQL." In reality, the proponents seem to saying that it stands for "not only SQL." Whether to use relational or NoSQL data technology for all or part of an applications data is not an all or nothing proposition.

It seems to me that the strength of NoSQL is in meeting non-functional requirements, it makes it easier to scale out systems or parts of systems which do not need cross-user ACID characteristics but need to have lots of distributed parallel processing to achieve high throughput and high availability. There are certainly parts of many systems which have those characteristics. Things like shopping carts and other user session data (the use case for Dynamo) do not really need to be consistent globally. A weaker form of consistency, consistency viewed by an individual user is often enough, and is where NoSQL systems tend to focus their efforts, by providing features like "read what you wrote."

The application needs to deal with what happens when consistency is lost or delayed. This is not unlike what we do with optimistic locking in a traditional database, but I would posit that it is not exactly the same thing.

For other parts of the system, like the transactions needed to turn a shopping cart into an order, or credit/debit transactions (e.g for payment or inventory control), an ACID transaction semantic is needed. Building such a semantic on top of a NoSQL base is problematic. Building these parts using a more traditional relational approach seems to make sense.

So the lesson I take from this is that although I want to have NoSQL in my toolbox for use when appropriate, I don't want to see it, or any other tool, as a Maslovian hammer.


Total World Domination through Storyboards, Wire-Framing, and Balsamiq Mockup

Posted by Rick DeNatale Sun, 04 Oct 2009 22:25:00 GMT

I admit it, I'm a much better programmer than graphics designer.

But getting the overall UI structure and flow of an application "right" requires a certain amount of "right brain" activity. And one of the best approaches is to create a "wire-frame" story board.

I've worked in situations where the "chief designer" would hand out web pages carefully crafted in PhotoShop, and ask for "pixel-perfect" renditions. UI discussions would be held using these images.

The problem is that this can lead to quite a bit of yak shaving, particularly of the bikeshedding variety.

I'm a big movie fan, in my dreams I've wanted to be a movie director if I couldn't be a programmer, or a Rock and Roll star, and among my heroes I count Hitchcock, Scorsese, Peter Jackson, and many others. I think that designing user interfaces is quite similar to designing movies. As someone who has watched a lot of that supplemental material on DVDs of recent an not so recent movies, I know how often directors use story boards to work out the flow and pace of a movie.

Storyboards are a rapid prototyping technique. It's a lot cheaper to work with easily replaceable "low-fidelity" artists drawings than to shoot the actual movie. Ideas can be discussed with little regard to the cost of the story-boarding.

Henchmen Small

User Interface design can, and should make use of a similar technique often called wire-framing, in which the sequence of presentations, the "scenes" if you will, which will take the form of things like web pages and dialog boxes, are sketched out and annotated as to what actions cause which transitions.

A few weeks ago I became aware of a product called Balsamiq Mockups which does a nice job of wire-framing.

Mockups exists both as an online web application and as an Adobe Air desktop version.

One way to think of Mockups is as a specialized version of PowerPoint or Keynote, although that's missing the point a bit. The similarity is that you can create storyboard pages, which are similar to pages in a presentation. There are some differences that make it a specialized wire-framing tool. First Mockups comes with lots of built-in user-interface components for web applications, desktop applications, and even iPhone apps. Second, rather than presenting the pages sequentially, you set up transitions between the pages based on user interactions, like clicking on buttons.

I find the cartoonish feeling of the pages quite nice. They get the idea across without inviting unnecessary quibbling about pixel-perfection. This is a tool for getting the big ideas down, after that a competent web-designer can polish things.

I've been looking for something like this for a while. I know quite a few Rails developers and designers who are pretty pumped up about this tool. It's well worth having a look.


RSpec/Textmate Pro-Tip

Posted by Rick DeNatale Wed, 23 Sep 2009 12:34:00 GMT

I often need to output debugging info in specs when something just isn't working. No sweat, just put in a puts or two, and run the spec.

Of course I'm usually doing this in Textmate using the RSpec bundle, which displays the results in a nice window formatted with html.

RSpecFormattedResults

But that causes problems with my debugging output

Bad Puts Format

The problem is that the puts output is just raw text so the HTML formatter ignores line breaks.

At one of the Radiant hack fests, either John Long or Adam Williams taught this old dog a simple new trick. The suggestion was to define a new Kernel#rputs method as a wrapper for puts which 'wraps' the arguments in an html <pre> tag. Then just call rputs (the r is for rspec of course) in the spec and or code under test.

Good Rputs Format

This works great as long as you run the specs under RSpec. But if you run them in the shell you see something like this:

$ spec spec/ri_cal/bugreports_spec.rb <pre> Booga booga </pre> <pre> this should be another line </pre> ....... Finished in 0.525324 seconds 7 examples, 0 failures

I've been putting up with this minor annoyance for a while, but this morning it struck me that I could define rputs conditionally depending on whether the spec was being run in Textmate or not. The trick is that Textmate defines one or more environment variables whose name starts with 'TM_'. So, without further ado, here's the new fragment from my spec_helper.rb:

  module Kernel
    if ENV.keys.find {|env_var| env_var.match(/^TM_/)}
      def rputs(*args)
        puts( *["<pre>", args.collect {|a| CGI.escapeHTML(a.to_s)}, "</pre>"])
      end
    else
      alias_method :rputs, :puts
    end
  end

With this change, running in the shell now looks like this:

$ spec spec/ri_cal/bugreports_spec.rb Booga booga this should be another line ....... Finished in 0.348086 seconds 7 examples, 0 failures

A nice refinement to my bag of tricks!

Update 27 Sept 2009

One slight change. I notice that String#start_with? only exists in Ruby 1.8.7, 1.9 and is apparently also defined the activesupport gem for <1.8.7. I changed the spec helper to use a regular expression match instead which works in all cases.


Red Light, Green Light

Posted by Rick DeNatale Thu, 03 Sep 2009 12:15:00 GMT

I've been in a lot of programming shops in my days, some fostered an agile mindset, some less so.

One feature which aids agile development is having something which gives the team visible feedback to the overall state of the project. One major real-time status item is the state of the continuous integration testing.

You are doing continuous integration testing aren't you? There are good lightweight tools and services which make this easy, I use run>run>run to test my open source projects like RiCal every time I commit to its publicly visible repository on github. Run>run>run provides free service for public open source projects, and range of paid plans to suit just about any budget.

And that's just one option, it just happens to be the one I use.

The continuous "integration" testing mindset can also be seen fractally. In between commits, something like autotest out of the ZenTest gem, or the autospec variation which is part of RSpec, can keep a continuous eye on the state of your code as you change it.

Thinking back on some of those programming shops, a couple of them had something I really liked, they had adapted real traffic lights to show build test status. Google for traffic light and you'll find that you can get them for fairly reasonable prices.

Or you could build your own. For a smaller scale version, consider this miniature traffic light built by a maker for his kids using typical maker supplies like, an Altoids tin, a cigar box, some L.E.D.s and a minimal microcontroller. When I saw this yesterday, I immediately thought how cool it would be to hook this up to a computer for C.I. feedback.

A small "confession." Until recently, I had a green-red status light for RiCal in the sidebar of this blog, using Glenn Vanderburg's runcoderun-badges but had to turn it off temporarily when someone told me that the UI for commenting to the blog was no longer working.

It turns out that the badge (which uses JQuery), was fighting with the Prototype flavored Javascript. I'm sure it wouldn't be that hard to fix using JQuery's noConflict function, I just need to find the time to try it and test it. For the time being it was more pragmatic just to remove the badge.

Of course the downside is that I once again have blogspam comments showing up for moderation, I'd been wondering where they went!


Exploratory Testing and "Dumb" Users

Posted by Rick DeNatale Mon, 03 Aug 2009 14:15:00 GMT

A tweet from Ward Cunningham yesterday led me to watch a very interesting video of a lunch time demo of exploratory testing by Elisabeth Hendrickson , while visiting Pivotal Labs. Ms Hendrickson shows how she forms a model of how the software works in order to find issues more productively. During the session someone says that they always heard that testers should be like typical "dumb" users and treat the software as a black box, but Elisabeth says that getting information about the actual implementation from the developers is fine. She also talks about her process of forming hypotheses about how the system works, and doing experiments which confirm or refute those hypotheses.

Does this sound familiar? We're talking about the scientific method here. In an interesting coincidence, I also stumbled upon another web video this weekend, an interview in which Richard Feynman describes science with an analogy of learning the rules of chess, simply by observing games being played, with no explanation.

I found myself thinking about the roles of the exploratory test. It seems to me that one of these roles is to act as a mock for the end user.

And despite the common epigram, users are usually far from dumb. A typical user forms a mental model of how a system operates, and use that to inform their use of the system. The model starts from a base of expectations about how similar systems operate, and the user adapts the model as he/she encounters behavior which doesn't fit. Very much like Feynman's hypothetical chess student, who first observes that Bishops only move to squares of the same color, and Kings move one square, and then has to adapt the model when they encounter a pawn being promoted to a Bishop, or castling.

When the user's model and the implementation model don't align, interesting things can happen, often unpleasant. It's usually desirable to eliminate such misalignments, and the conversations between an exploratory tester and a developer about misapprehensions about how either the system or the user works are valuable in achieving this.

The scientific method is also useful for exploratory development, such as when writing code which interfaces with or implements a complex API or standard, but that's fodder for a future article.


The Value of Co-Location in Agility

Posted by Rick DeNatale Tue, 30 Jun 2009 11:48:00 GMT

Last night there was a very interesting extra meeting of the local AgileRTP group. Johanna Rothman was in town and Jared Richardson arranged an extra meeting so that she could present some of the issues which arise in adopting agile methods, and how to deal with them. Johanna is very knowledgeable and gave an interesting and thought provoking thought.

One of the issues she talked about was "distributed" agility. This is a hot button as many companies are outsourcing some development to "offshore" locations. Johanna talked about the issues in following agile practices, and I'm vastly oversimplifying here, pointed at time zones as the major issue.

Someone in the audience, who works in a distributed team with members across the US, questioned her on this. As the discussion unfolded, other issues came up, such as dealing with a team in China which had one person fluent enough in English who acted as the "email-translator", as well as cultural differences affecting communication.

Language and cultural differences certainly can inhibit the free exchange of information which is the coin of the realm in Agile practices, but I had to observe that these are not really geographical or time zone issues. I've seen similar difficulties with co-located teams.

Distributed development using agile methods can indeed work, I've seen it. OTI was quite good at it. We did timeboxed projects with development spread across the globe. A given project might comprise team members in Ottawa, Zurich, Sydney, Raleigh, and Vancouver. Large projects might use a "team of teams" but smaller ones had individual members who were geographically dispersed. There was lots of communication by phone and email, and it worked quite well.

OTI had one "stand-up" meeting every year, usually in February in the balmy clime of Quebec, an annual Tech Conference where everyone in the company got together at a sort of internal "OOPSLA" to talk about what they were doing and to exchange interesting ideas, and just get to know each other better.

Now the main thing that made this work was that we shared a common culture when it came to development. Although not everyone at OTI was a native English speaker, all were fluent.

"Big Dave" had a tendency to hire good people wherever he found them. Often such a person would seed a new multi-person laboratory, but sometimes a "lab" would be a single person.

So distributed development can work, and work well, given the right people.

Some agility advocates stress the need for co-location, a notable exception is Kent Beck, the founder of eXtreme Programming, who now spends most of his time based a his farm in southern Oregon, and does remote pair-programming using VNC and Skype, with programmers around the world.

And Kent is currently auctioning off a two-hour pairing session this is a good opportunity to sample distributed agility at a high level.

I wish I could afford to bid, the current bid is at $200, which is cheap. I've learned a lot every time I interacted with Kent personally in the past. Someone is going to get a deal, whatever the winning bid turns out to be.


Ward Cunningham on Wikis and Agility

Posted by Rick DeNatale Thu, 21 May 2009 13:24:00 GMT

Here's another video from Ward in which he shows the working environment at AboutUs and talks a bit about the connection between his conception of wikis and agile programming.

In an evolution of the recommended extreme programming workspace setup. AboutUs gives everyone a desk on wheels rather than undedicated pairing stations, so that developers can rearrange the configuration to match who they want/need to work with at the time.


Brother Can You Spare the Time?

Posted by Rick DeNatale Thu, 09 Apr 2009 12:09:00 GMT
"Once I built a railroad, now it's done
 Brother, can you spare a dime?"

A couple of evenings ago at the monthly meeting of Agile RTP Jason Tanner from Enthiosys led a discussion of experience in introducing agile development methods to companies.

He led off by encouraging the audience to make index cards inscribed with impediments to the adoption of agile methods which the attendee had encountered, and tape them to a whiteboard. He then took the cards off and discussed them.

One of the first was something like, “people who can’t work without a detailed long-range plan.”

For some, reason, the old depression-era song “Brother can you spare a dime?” popped into my head as soon as that discussion started.

Planes, Trains, and Automobiles

Kent Beck tells the story in Extreme Programming Explained: Embrace Change (2nd Edition) (XP Series) about his first lesson on driving a car, given by his mother. She started by giving him a feel for the steering wheel, by having him reach across from the passenger seat and take the wheel and see how it affected the direction of the car. She then told him to look ahead and point the car down the road. After a while, is attention drifted a bit and the car started towards the gravel at the side of the road. Kent’s mom gently took control back and gave him his first real driving lesson:

"Driving is not about getting the car going in the right direction. Driving is about constantly paying attention,
 making a little correction this way, a little correction that way."

To me that’s the real difference between agile and waterfall approaches.

The waterfall “tradition” attempts to avoid getting off the road by starting out with a plan. If it works the plan is like a set of railroad tracks which forces the train to always go in the right direction. The engineer doesn’t need to pay as much attention.

Agile methods are much more like driving, setting a general short term direction, with the current long term goal in mind, and reacting to feedback so that constant correction keeps the project from going into the ditch. At a more mature level, it becomes more like flying a plane, which is like driving a car, but with more complex feedback-control relationships and in three dimensions rather than two.

The Problem with Tracks

A railroad is a great thing as long as it goes where you really need to go. But getting on the wrong train can be a costly mistake. You can end up traveling a long way away from your goal, and waste a lot of time in the error and in recovering from the error.

You might enjoy the scenery along the way, thinking you are progressing towards your goal, until you realize your mistake. This is a great metaphor for some of the waterfall projects which I’ve seen go astray. Of course those projects suffer the expense not just of riding the wrong train, but of designing and building the railroad through a process of design based on incompletely understood requirements.

As Yogi Berra said, “If you don’t know where you are going, you’ll end up somewhere else.” And all too often in developing software, we don’t really know where we are going except in a general or vague sense. Requirements based on this level of understanding are fragile in the face of knowledge gained about the problem context and about the solution as development proceeds. If you are riding the train, you might start to get a sense that you might not be where you need to be gradually, an increasing unease perhaps rising to panic.

If you are on the wrong track, you can end up wasting lots of time and lots of dimes!

I’ve found that it’s much better to file a flight plan, then react to changing weather, input from flight controllers, and radio instructions from “the home office” when business plans change and I now need to get to Chicago instead of St. Louis.

And that’s why that song popped into my head, I was thinking:

Once I built a railroad, but it went to the wrong place.

You Get What You Pay For

Posted by Rick DeNatale Wed, 08 Apr 2009 21:02:00 GMT

Well, that’s the saying. One thing is for sure. You can’t expect to get what you don’t pay for.

I got an IM from a buddy this afternoon looking for a second opinion on a Rails application. Someone had paid for 100 hours of work (I don’t know at what rate), for a Rails project which wasn’t progressing to his liking. He’d asked my buddy to give an opinion of what had been done, and I was asked to have a look as well.

I only spent about a half hour or so looking at the code, since I was doing this as a courtesy. I spend quite a bit of pro-bono time helping others by writing here and contributing on public fora, but I’ve got my limits.

The executive summary: I wasn’t impressed.

This was a completely blind sampling. I have no idea who the consultant was who wrote this app, nor even the identity of the client who had funded it.

I noticed that there was both a spec and a stories directory in the project a good sign. Looking closer though the specs all seemed to have been generated automatically by the restful_authentication plugin, same thing for the stories.

And the tests were the standard useless stub tests generated by the Rails model/controller generators.

There was some very suspect code in some of the controllers I looked at, one was creating a model in the new action. Normally the new action instantiates an unsaved model instance for use by an edit/creation form, leaving the actual creation of the model in the database to the create action after the form is posted.

The Application controller had some strange looking code, which looked like it might be boilerplate from somewhere. When I googled for a snippet from the code, it looks like it came from somebody’s rails 2.3 template for generating a new rails app using shoulda.

So we have an app which can’t seem to decide which testing framework to use and then not really use any of them to any effect.

Now I know that shoulda, rspec and test::unit can be used in combination by a sophisticated developer, but I don’t think that’s what was going on here.

I poked around in a few models. I found one which really flummoxed me. The model purported to be representing a currency exchange rate for a particular currency on a particular date. The model had a class method average which took a currency and a date range as arguments. Now from the name, I’d expect that this would maybe calculate the average rate for instances for that currency between the first and last dates of the range. Lacking any tests/specs I’ll assume that’s what it would do.

What did it do?

def self.average(currency, date_range)
     first = find(:first, :conditions => ["currency = ? AND issued_on = ?", currency, date_range[0].to_date])
     first ||= find(:first, :conditions => ["currency = ?", currency] , :order => "issued_on"
     last = find(:first, :conditions => ["currency = ? AND issued_on = ?", currency, date_range[1].to_date])
     first ||= find(:first, :conditions => ["currency = ?", currency] , :order => "issued_on ASC"

     median(first.rate, last.rate)
end

protected
def median(a, b)
    (a + b) / 2
end

See anything strange here? Actually I see several.

  1. To find the first it looks for an instance for the currency which happened exactly on the starting date. If it doesn’t find one then it finds the first instance for that currency in the database. I would have expected to find the first instance on or after the starting date, which could have been done with a single find call with the right condition.
  1. Then to find the last it does what looks like a simple variant of the code to find the first, but is it a variant, instead of finding the last instance for the currency in the database rather than the first it’s going to again find the first. He wanted :order => “issued_on DESC”.
  1. What kind of average is being computed here anyway? Average is a generic term, normally we take it to denote the mean, which is the summation of a series of values divided by the number of elements in the series. We’re not doing that here, since we look at only two elements. Now the method being called is median, which is another type of average. You get the medium by sorting the series and picking either the middle element if the series has an odd number of elements, or the mean of the two middle elements. But that’s certainly not what this median method is doing.

Having specs or tests would have helped this guy both think through what he was trying to do, and to realize that it wasn’t working.

So to me it looks like the client had paid for some guy generating a rails app from some templates and plugins and then floundering around with no real idea of how to understand, let alone achieve, the goals of the application.

Like I said in the beginning, it’s not really that you get what you pay for, more like you sometimes get what you pay for. Since I don’t know the rate the client was charged, I don’t know whether he got what he paid for by hiring someone deservedly cheap, or just didn’t get what he paid for.

It’s always a shame when customers don’t get what they pay for, but in the current economy, it’s a real shame when a client burns money hiring someone on the cheap or without real credentials instead of paying someone who might at first look more expensive but would be a bargain in comparison in terms of value for money.


Ward Cunningham on the Debt Metaphor

Posted by Rick DeNatale Mon, 02 Mar 2009 16:57:00 GMT

I just discovered that my old friend, Ward Cunningham has started putting some short interviews, and “video essays” up on YouTube

In the embedded video above, Ward relates how he came up with the metaphor of debt to explain the need for refactoring in interative development. The key insight is that you can proceed faster with an incomplete knowledge of the problem you are solving, and learn as you go. In many cases this is essential because for many problems the problem itself changes as implementation proceeds, because of effects like new feedback from users once they see the system starting to emerge. Ward compares implementation with incomplete knowledge to working with borrowed money. As the system implementation proceeds, the assumptions made on incomplete or out of date knowledge, which drove the design, or “architecture” if you prefer, of the system, start to make implementation of the more complete or evolved requirements progressively harder. In the debt metaphor, this increased cost of making progress corresponds to paying interest on the loan. Refactoring is a process of redesigning the software to reflect this new problem knowledge, without breaking it, thus “re-financing” the system.

Before I’d heard Ward express this metaphor, I’d come up with another for the same idea. One of my hobbies is metalworking. I saw an evolving system as a piece of metal which starts out as flexible and malleable. But like metal, as you start to form it by hammering, and bending, and other operations, it starts to “work-harden.” This means that the metal becomes more and more resistant to further shaping, because of local changes to it’s crystal structure. Eventually it resists all further efforts, often by failing with a stress fracture. This is why you can take a piece of, say copper, and bend it back and forth several times until it finally breaks at the bend.

In metalworking, you can use a process called annealing which returns the metal to a ductile, maleable state. Of course the analogy here is to refactoring.

I still like both metaphors, I tend to use my work hardening/annealing metaphor when talking to programmers, and Ward’s debt metaphor when talking to business people (the “gold donors.”)