A tweet from Ward Cunningham yesterday led me to watch a very interesting video of a lunch time demo of exploratory testing by Elisabeth Hendrickson , while visiting Pivotal Labs. Ms Hendrickson shows how she forms a model of how the software works in order to find issues more productively. During the session someone says that they always heard that testers should be like typical "dumb" users and treat the software as a black box, but Elisabeth says that getting information about the actual implementation from the developers is fine. She also talks about her process of forming hypotheses about how the system works, and doing experiments which confirm or refute those hypotheses.
Does this sound familiar? We're talking about the scientific method here. In an interesting coincidence, I also stumbled upon another web video this weekend, an interview in which Richard Feynman describes science with an analogy of learning the rules of chess, simply by observing games being played, with no explanation.
I found myself thinking about the roles of the exploratory test. It seems to me that one of these roles is to act as a mock for the end user.
And despite the common epigram, users are usually far from dumb. A typical user forms a mental model of how a system operates, and use that to inform their use of the system. The model starts from a base of expectations about how similar systems operate, and the user adapts the model as he/she encounters behavior which doesn't fit. Very much like Feynman's hypothetical chess student, who first observes that Bishops only move to squares of the same color, and Kings move one square, and then has to adapt the model when they encounter a pawn being promoted to a Bishop, or castling.
When the user's model and the implementation model don't align, interesting things can happen, often unpleasant. It's usually desirable to eliminate such misalignments, and the conversations between an exploratory tester and a developer about misapprehensions about how either the system or the user works are valuable in achieving this.
The scientific method is also useful for exploratory development, such as when writing code which interfaces with or implements a complex API or standard, but that's fodder for a future article.
One resource has been James Edward Gray II's book on TextMate published by the Pragmatic Programmers. I finally sat down and got serious about writing some automations of my own.
I've got to say that I'm pretty impressed by TextMate. At the last Raleigh.rb hack night I was talking to another vim user. I'd mentioned to him that you can extend TextMate easily in Ruby, without really having experienced it. Today, I wrote a neat little TextMate command to help in building Rails database test fixtures.
It acts like a tab triggered snippet, but it's a smart little critter. If I'm editing a fixture file, say 'test/fixtures/users.yml', I can type item then tab and it will produce the skeleton yaml for a new record, with:
- A dummy name selected for overtyping.
- The id set to the next available primary key Each column name as a yaml key ...
- ... A tabstop on a value which shows the column type
For example:
Suppose I've got a fixture file for a model called Item:
# == Schema Information # Schema version: 14 # # Table name: items # # id :integer(11) not null, primary key # description :string(255) # name :string(255) # price :integer(11) # # Read about fixtures at http://ar.rubyonrails.org/classes/Fixtures.html lotr1: id: 1 name: LOTR1 description: The Road Goes Ever On lotr2: id: 2 name: LOTR2 description: Introduction to Elvish
If I type a new line at the end of this file with the snippet command item and hit tab I see this:
lotr2:
id: 2
name: LOTR2
description: Introduction to Elvish
NameMe:
id: 3
description: string
name: string
price: integer
With NameMe selected so that I can easily replace it by typing. Note that the id was set to 3 since rows with ids 1 and 2 already exist.
Hitting tab again selects the first instance of string, and subsequent tabs step through the columns
Another feature which I won't show is that it looks at the table and if it doesn't have a primary key (e.g. for an association table in a has_and_belongs_to_many association), it won't generate the id: or a key.
You might be impressed, but I'm pretty happy with the results of a couple of hours of playing with TextMata and Ruby.
How it works.
The whole thing is done with a TextMate command implemented in Ruby. A few notes. First, it's just a coincidence that the snippet trigger is item and my example table is called items. Second, I'm not using the comments at the top of the fixture file which were generated when the model was created. The command works off of db/schema.db.
Here's the code:
#!/usr/bin/env ruby -w # # Created by Rick DeNatale on 2007-10-09. # Copyright (c) 2007. You may use this under the ruby license. $LOAD_PATH << "#{ENV['TM_SUPPORT_PATH']}/lib" require 'exit_codes.rb' class FixtureSnipGen attr_reader :rails_root, :table_name def initialize(fixture_file) match = %r{(.*)test/fixtures/(.*).yml}.match(ENV["TM_FILEPATH"]) @rails_root = match[1] @table_name = match[2] @no_id = false end def next_id max_id = 0 while line = gets claimed = %r{\s+id:\s+(\d+)}.match(line) max_id = [max_id, claimed[1].to_i].max if claimed end max_id + 1 end def gen_snippet cols = gen_columns if cols puts "${1:NameMe}:" puts " id: #{next_id}" unless @no_id puts cols.join("\n") else puts "Couldn't find #{table_name} in schema.rb" TextMate.exit_show_tool_tip end puts "$0" end def schema_file_name "#{rails_root}db/schema.rb" end def value_tab(text) @tab_stop += 1 (@tab_stop < 10) ? "${#{@tab_stop}:#{text}}" : " # #{text}" end def gen_columns # raise Exception.new("No schema file") unless File.exist?(schema_file_name) started = false @tab_stop = @no_id ? 0 : 1 result = [] File.new(schema_file_name).each do | line | if started break if %r{^(\s*)create_table\s+}.match(line) col_match = %r{\.column\s+\"(.*)\"\s*,\s*:(\w+)}.match(line) result << " #{col_match[1]}: #{value_tab(col_match[2])}" if col_match else started = %r{^(\s*)create_table\s+["']#{table_name}['"]}.match(line) @no_id = %r{:id\s+=>\s+false}.match(line) if started end end return result.empty? ? nil : result end end begin FixtureSnipGen.new(ENV["TM_FILEPATH"]).gen_snippet rescue Exception => ex puts ex TextMate.exit_show_tool_tip end
In the TextMate bundle editor, I just added a new command with this code, and set the following options:
- Save:
- Nothing
- Input:
- Entire Document - which pipes the entire textmate buffer for the file into stdin.
- Output:
- Insert as Snippet - which triggers TextMate to interpret the output as a snippet once the script has finished.
- Activation:
- Tab Trigger = item
- Scope Selector source.yaml - this only works within yml files.>
Finally note that although my example uses a table called items, it's just a coincidence that the tab trigger is item. It would be item in any fixture file.

Yesterday I wrote about some code I wrote to control what Time.now and Date.today return to support time-dependent testing.
This afternoon I discovered a critical, but easy to fix bug. I’ve updated the original article.
The good news is that the testcase for the time machine now has another test!

I’ve been working on adding support for localization of the user’s time zone on an existing Rails app for a client. In order to test this, I found myself building a time machine.
At first, I did a fairly simple hack which monkey patched the system methods Time.now, and Date.today. This worked until I got into some testcases of code which was triggered off of updated_at fields in various ActiveRecord models. Since I had to deal with these implicit dates, I found that I needed to have finer control than my simple patch gave.
The code has now evolved to the point where I think that it shows some interesting aspects of basic Ruby metaprogramming.
The Initial Approach
In order to control the time, I decided to add a module in Test::Unit::Testcase called TimeMachine, so that a testcase could just include TimeMachine when it needed the function.
The way it’s used is to write something like:
def testSomethingYesterday
now_as(1.days.ago) do
#code to be run yesterday here
end
endThe now_as method takes a time as a parameter. Within the block, Time.now will return that time, and Date.today will return the date corresponding to that time.
Doing this was fairly straightforward. The method aliased the two methods, redefined them, then, within a begin block with an ensure clause to restore the methods afterwards, yielded to the block.
Non-stop Time Trip Only
While this simple approach worked well, it had one drawback. It couldn’t be stacked. If you called now_as again within the block, the inner call would remove the monkeypatched methods when it returned. This first showed up when I had a bug in one of my testcase methods. That was fixed easily enough by rewriting that test.
But when I ran into the code which was using implicit times, I figured it would be easier to make my test helper a bit more sophisticated. I needed a time machine which could make side-trips along it’s round, trip in time.
The TimeMachine as of Now
So here’s my current implementation of Test::Unit:Testcase::TimeMachine:
class Test::Unit::TestCase # 1
module TimeMachine
def now_as(time)
time_class = class << Time; self; end # 5
date_class = class << Date; self; end
begin
Time.class_eval do
@now_stack ||= []
if @now_stack.empty? # 10
time_class.class_eval do
alias_method :old_now, :now
def now
@now_stack.last.dup
end # 15
end
date_class.class_eval do
alias_method :old_today, :today
def today
Time.now.to_date # 20
end
end
end
@now_stack.push(time.dup)
end # 25
yield
ensure
Time.class_eval do
@now_stack.pop
if @now_stack.empty? # 30
date_class.class_eval {alias_method :today, :old_today}
time_class.class_eval {alias_method :now, :old_now}
end
end
end # 35
end
endThe basic idea is to maintain a stack of times in a class instance variable of Time. We define the pseudo now and today methods when the first time is placed on this stack, and restore them when the last time is removed.
The tricky part of this code is knowing when to talk to the Time and Date classes and when to talk to their respective metaclasses. In lines 5 and 6 I grab the two metaclasses so that I can refer to them in the code below (DRY). In line 7 I start the begin block which ensures that things will be restored when now_as has finished.
On line 9, running in the context of the Time class, I ensure that it has an instance variable to contain the stack of now times. Then if the stack is empty, I define the methods. Lines 12-15 run in the context of Time’s metaclass to define now as a class method of Time. Lines 18-21 handle the today method in a similar fashion.
After we’ve ensured that our newly patched methods are there, we push the time on line 24.
The yield, on line 26 runs back in the context of the testcase, which proceeds to do it’s thing.
Once that’s done, succeed or not, the ensure block cleans things up. Lines 29-33 go back to the context of the Time class to manipulate the stack and, if it’s empty again, restore the original methods.
What’s in the Future?
So that’s where the time machine sits right now. It does the simplest thing that could possbily work, right now. It might be nice if it could, perhaps optionally, have the current time change rather than being fixed, in other words, Time.now would return the originally stated time plus whatever time increment had elapsed since the as_now call. But for now, I haven’t needed it, and I’m not planning to use the time machine to artifically skip ahead to find future requirements before I discover them in real time.
Postscript
After posting this yesterday, I discovered that the nesting didn’t really work. I was only pushing the time on to the stack the first time. I’ve just corrected the code above.
The other day I wrote about problems with undeclared fixtures in Rails ActiveRecord tests.
Here’s a little code snippet which might be useful in tracking down such problems.
class MyTest < Test::Unit::TestCase
def self.been_here
result = @been_here
@been_here = true
result
end
class RowCounter < ActiveRecord::Base
end
def setup
unless self.class.been_here
File.open("#{RAILS_ROOT}/data_dump_#{Time.now.strftime("%m%d%H%M%S")}", "w") do |file|
Account.connection.tables.each do |table_name|
RowCounter.table_name = table_name
file.puts "#{table_name}: #{RowCounter.count}"
end
end
end
# The rest of your setup code goes here
end
# And the rest of the TestCase code here
endThis will dump a list of each table with a count of its rows the first time setup is run for the testcase. The file name is generated with a time stamp
Okay, so the example does go along with this weeks Harry Potter fever.
rails_developer@hogwarts.edu.magic.uk/cauldron_project/trunk$ cat data_dump_0717105533 students: 17 professors: 2 owls: 2 potions: 2 magical_creatures: 2 muggles: 0
Now you can run the test individually and then along with others. Then use a diff tool to see which tables have different numbers of rows before your test case starts for the first time.
A test helper
If you find this useful, why not add it to a test helper. If you don’t already have one, edit your test/test_helper.rb file, and add the following code inside of the Test::Unit::Testcase class:
def self.been_here
result = @been_here
@been_here = true
result
end
class RowCounter < ActiveRecord::Base
end
def dump_table_counts(file_name)
File.open(file_name, "w") do |file|
Account.connection.tables.each do |table_name|
RowCounter.table_name = table_name
file.puts "#{table_name}: #{RowCounter.count}"
end
end
endI’ve moved the definition of the been_here method and the class Rowcounter from the individual testcase to the superclass. The only code in the testcase needed to use this is in the setup method, which now becomes:
def setup
dump_table_counts("#{RAILS_ROOT}/data_dump_#{Time.now.strftime("%m%d%H%M%S")}") unless self.class.been_here
# rest of setup code here
endThe other day I was working on adding support for user selected time-zones to an existing rails app for a client. As usual I was doing test-first development. One of the things that makes rails such a pleasure is that a good set of tests give confidence that you aren’t breaking “legacy” code.
I also use, and really like, the rails plugin for vim which does lots of nice things like making navigation between the files of rails apps much easier. It also has a nice feature which adds a :Rake command to vim which “does the right” thing contextually. For example if you are editing a migration and enter :Rake, it runs rake db:migrate. If you are in a test file it runs just the single test selected by the cursor, or just that test file if you aren’t positioned to a particular test.
I was doing the latter, and my test was failing, and I was having a hard time debugging it. I tried executing just that single test from the bash command line with:
$ruby test/unit/my_test.rb -n"test_mytest"
and it still failed, no surprise. The same thing happened if I ran the entire test file, in fact, other tests which had worked before were now failing, and I was really mystified now because I didn’t see how I had done anything which had a remote chance of breaking those.
So I figured I probably should test everthing so:
$rake test
And, surprise of surprises, but every test worked. Not only my failing unit tests, but the functional and integration tests as well.
To make a long story short, the problem turned out to be that the tests were failing because, now that the particular model was sensitive to the user’s timezone, it needed access to it’s associated user model, and I hadn’t told the testcase that the users fixture was needed. Running the testcase in isolation, the users table wasn’t being populated for the test case. But running rake test, or rake test:units meant that other tests run before had left the data I needed behind.
Just a little thing that makes fixtures less than ideal.




