Exploratory Testing and "Dumb" Users

Posted by Rick DeNatale Mon, 03 Aug 2009 14:15:00 GMT

A tweet from Ward Cunningham yesterday led me to watch a very interesting video of a lunch time demo of exploratory testing by Elisabeth Hendrickson , while visiting Pivotal Labs. Ms Hendrickson shows how she forms a model of how the software works in order to find issues more productively. During the session someone says that they always heard that testers should be like typical "dumb" users and treat the software as a black box, but Elisabeth says that getting information about the actual implementation from the developers is fine. She also talks about her process of forming hypotheses about how the system works, and doing experiments which confirm or refute those hypotheses.

Does this sound familiar? We're talking about the scientific method here. In an interesting coincidence, I also stumbled upon another web video this weekend, an interview in which Richard Feynman describes science with an analogy of learning the rules of chess, simply by observing games being played, with no explanation.

I found myself thinking about the roles of the exploratory test. It seems to me that one of these roles is to act as a mock for the end user.

And despite the common epigram, users are usually far from dumb. A typical user forms a mental model of how a system operates, and use that to inform their use of the system. The model starts from a base of expectations about how similar systems operate, and the user adapts the model as he/she encounters behavior which doesn't fit. Very much like Feynman's hypothetical chess student, who first observes that Bishops only move to squares of the same color, and Kings move one square, and then has to adapt the model when they encounter a pawn being promoted to a Bishop, or castling.

When the user's model and the implementation model don't align, interesting things can happen, often unpleasant. It's usually desirable to eliminate such misalignments, and the conversations between an exploratory tester and a developer about misapprehensions about how either the system or the user works are valuable in achieving this.

The scientific method is also useful for exploratory development, such as when writing code which interfaces with or implements a complex API or standard, but that's fodder for a future article.


My First Serious TextMate Automation

Posted by Rick DeNatale Tue, 09 Oct 2007 12:15:00 GMT
I recently got assimilated by the Mac/TextMate borg. I'm slowly teaching my fingers to dance the TextMate tango and unlearning old vim habits.

One resource has been James Edward Gray II's book on TextMate published by the Pragmatic Programmers. I finally sat down and got serious about writing some automations of my own.

I've got to say that I'm pretty impressed by TextMate. At the last Raleigh.rb hack night I was talking to another vim user. I'd mentioned to him that you can extend TextMate easily in Ruby, without really having experienced it. Today, I wrote a neat little TextMate command to help in building Rails database test fixtures.

It acts like a tab triggered snippet, but it's a smart little critter. If I'm editing a fixture file, say 'test/fixtures/users.yml', I can type item then tab and it will produce the skeleton yaml for a new record, with:

  1. A dummy name selected for overtyping.
  2. The id set to the next available primary key
  3. Each column name as a yaml key ...
  4. ... A tabstop on a value which shows the column type

For example:

Suppose I've got a fixture file for a model called Item:

	# == Schema Information
	# Schema version: 14
	#
	# Table name: items
	#
	#  id          :integer(11)   not null, primary key
	#  description :string(255)   
	#  name        :string(255)   
	#  price       :integer(11)   
	#

	# Read about fixtures at http://ar.rubyonrails.org/classes/Fixtures.html
	lotr1:
	  id: 1
	  name: LOTR1
	  description: The Road Goes Ever On
	lotr2:
	  id: 2
	  name: LOTR2
	  description: Introduction to Elvish

If I type a new line at the end of this file with the snippet command item and hit tab I see this:


lotr2:
  id: 2
  name: LOTR2
   description: Introduction to Elvish

  NameMe:
   id: 3
   description: string
   name: string
   price: integer

With NameMe selected so that I can easily replace it by typing. Note that the id was set to 3 since rows with ids 1 and 2 already exist.

Hitting tab again selects the first instance of string, and subsequent tabs step through the columns

Another feature which I won't show is that it looks at the table and if it doesn't have a primary key (e.g. for an association table in a has_and_belongs_to_many association), it won't generate the id: or a key.

You might be impressed, but I'm pretty happy with the results of a couple of hours of playing with TextMata and Ruby.

How it works.

The whole thing is done with a TextMate command implemented in Ruby. A few notes. First, it's just a coincidence that the snippet trigger is item and my example table is called items. Second, I'm not using the comments at the top of the fixture file which were generated when the model was created. The command works off of db/schema.db.

Here's the code:

  #!/usr/bin/env ruby -w  
  #
  #  Created by Rick DeNatale on 2007-10-09.
  #  Copyright (c) 2007. You may use this under the ruby license.                    

  $LOAD_PATH << "#{ENV['TM_SUPPORT_PATH']}/lib"
  require 'exit_codes.rb'


  class FixtureSnipGen
    attr_reader :rails_root, :table_name
    def initialize(fixture_file)
      match  = %r{(.*)test/fixtures/(.*).yml}.match(ENV["TM_FILEPATH"])
      @rails_root = match[1]
      @table_name = match[2]
      @no_id = false  
    end

    def next_id
      max_id = 0     
      while line = gets
        claimed = %r{\s+id:\s+(\d+)}.match(line) 
        max_id = [max_id, claimed[1].to_i].max if claimed
      end
      max_id + 1
    end

    def gen_snippet
      cols = gen_columns
      if cols
        puts "${1:NameMe}:"
        puts "  id: #{next_id}" unless @no_id
        puts cols.join("\n")
      else 
        puts "Couldn't find #{table_name} in schema.rb"
        TextMate.exit_show_tool_tip
      end
      puts "$0"
    end

    def schema_file_name  
      "#{rails_root}db/schema.rb"
    end 

    def value_tab(text)
      @tab_stop += 1
      (@tab_stop < 10) ? "${#{@tab_stop}:#{text}}" : " # #{text}"
    end


    def gen_columns
      #    raise Exception.new("No schema file") unless File.exist?(schema_file_name)
      started = false
      @tab_stop = @no_id ? 0 : 1
      result = []
      File.new(schema_file_name).each do | line |
        if started
          break if %r{^(\s*)create_table\s+}.match(line)
          col_match = %r{\.column\s+\"(.*)\"\s*,\s*:(\w+)}.match(line)
          result << "  #{col_match[1]}: #{value_tab(col_match[2])}" if col_match
        else
          started = %r{^(\s*)create_table\s+["']#{table_name}['"]}.match(line)
          @no_id = %r{:id\s+=>\s+false}.match(line) if started
        end
      end
      return result.empty? ? nil : result    
    end
  end

  begin
    FixtureSnipGen.new(ENV["TM_FILEPATH"]).gen_snippet
  rescue Exception => ex
    puts ex
    TextMate.exit_show_tool_tip
  end

In the TextMate bundle editor, I just added a new command with this code, and set the following options:

Save:
Nothing
Input:
Entire Document - which pipes the entire textmate buffer for the file into stdin.
Output:
Insert as Snippet - which triggers TextMate to interpret the output as a snippet once the script has finished.
Activation:
Tab Trigger = item
Scope Selector
source.yaml - this only works within yml files.

Finally note that although my example uses a table called items, it's just a coincidence that the tab trigger is item. It would be item in any fixture file.


Do You Ever Have Those Kind of Days?

Posted by Rick DeNatale Thu, 19 Jul 2007 18:19:00 GMT

Yesterday I wrote about some code I wrote to control what Time.now and Date.today return to support time-dependent testing.

This afternoon I discovered a critical, but easy to fix bug. I’ve updated the original article.

The good news is that the testcase for the time machine now has another test!


Time Flies While You're Having Fun Testing

Posted by Rick DeNatale Wed, 18 Jul 2007 19:45:00 GMT

I’ve been working on adding support for localization of the user’s time zone on an existing Rails app for a client. In order to test this, I found myself building a time machine.

At first, I did a fairly simple hack which monkey patched the system methods Time.now, and Date.today. This worked until I got into some testcases of code which was triggered off of updated_at fields in various ActiveRecord models. Since I had to deal with these implicit dates, I found that I needed to have finer control than my simple patch gave.


The code has now evolved to the point where I think that it shows some interesting aspects of basic Ruby metaprogramming.

The Initial Approach

In order to control the time, I decided to add a module in Test::Unit::Testcase called TimeMachine, so that a testcase could just include TimeMachine when it needed the function.

The way it’s used is to write something like:

def testSomethingYesterday
   now_as(1.days.ago) do
     #code to be run yesterday here
   end
end

The now_as method takes a time as a parameter. Within the block, Time.now will return that time, and Date.today will return the date corresponding to that time.

Doing this was fairly straightforward. The method aliased the two methods, redefined them, then, within a begin block with an ensure clause to restore the methods afterwards, yielded to the block.


Non-stop Time Trip Only

While this simple approach worked well, it had one drawback. It couldn’t be stacked. If you called now_as again within the block, the inner call would remove the monkeypatched methods when it returned. This first showed up when I had a bug in one of my testcase methods. That was fixed easily enough by rewriting that test.

But when I ran into the code which was using implicit times, I figured it would be easier to make my test helper a bit more sophisticated. I needed a time machine which could make side-trips along it’s round, trip in time.

The TimeMachine as of Now

So here’s my current implementation of Test::Unit:Testcase::TimeMachine:

class Test::Unit::TestCase                    # 1
  module TimeMachine

    def now_as(time)
      time_class = class << Time; self; end   # 5
      date_class = class << Date; self; end
      begin
        Time.class_eval do 
          @now_stack ||= []
          if @now_stack.empty?                # 10
            time_class.class_eval do
              alias_method :old_now, :now
              def now
                @now_stack.last.dup
              end                             # 15
            end
            date_class.class_eval do
              alias_method :old_today, :today
              def today
                Time.now.to_date              # 20
              end
            end
          end
          @now_stack.push(time.dup)
        end                                   # 25 
        yield
      ensure
        Time.class_eval do 
          @now_stack.pop
          if @now_stack.empty?                # 30
            date_class.class_eval {alias_method :today, :old_today}
            time_class.class_eval {alias_method :now, :old_now}
          end
        end
      end                                     # 35
    end
end

The basic idea is to maintain a stack of times in a class instance variable of Time. We define the pseudo now and today methods when the first time is placed on this stack, and restore them when the last time is removed.

The tricky part of this code is knowing when to talk to the Time and Date classes and when to talk to their respective metaclasses. In lines 5 and 6 I grab the two metaclasses so that I can refer to them in the code below (DRY). In line 7 I start the begin block which ensures that things will be restored when now_as has finished.

On line 9, running in the context of the Time class, I ensure that it has an instance variable to contain the stack of now times. Then if the stack is empty, I define the methods. Lines 12-15 run in the context of Time’s metaclass to define now as a class method of Time. Lines 18-21 handle the today method in a similar fashion.

After we’ve ensured that our newly patched methods are there, we push the time on line 24.

The yield, on line 26 runs back in the context of the testcase, which proceeds to do it’s thing.

Once that’s done, succeed or not, the ensure block cleans things up. Lines 29-33 go back to the context of the Time class to manipulate the stack and, if it’s empty again, restore the original methods.

What’s in the Future?

So that’s where the time machine sits right now. It does the simplest thing that could possbily work, right now. It might be nice if it could, perhaps optionally, have the current time change rather than being fixed, in other words, Time.now would return the originally stated time plus whatever time increment had elapsed since the as_now call. But for now, I haven’t needed it, and I’m not planning to use the time machine to artifically skip ahead to find future requirements before I discover them in real time.

Postscript

After posting this yesterday, I discovered that the nesting didn’t really work. I was only pushing the time on to the stack the first time. I’ve just corrected the code above.


Tracking Down Missing Fixtures

Posted by Rick DeNatale Tue, 17 Jul 2007 17:50:00 GMT

The other day I wrote about problems with undeclared fixtures in Rails ActiveRecord tests.

Here’s a little code snippet which might be useful in tracking down such problems.

class MyTest < Test::Unit::TestCase

  def self.been_here
    result = @been_here
    @been_here = true
     result
   end

  class RowCounter < ActiveRecord::Base
  end

  def setup
    unless self.class.been_here
      File.open("#{RAILS_ROOT}/data_dump_#{Time.now.strftime("%m%d%H%M%S")}", "w") do |file|
        Account.connection.tables.each do |table_name|
          RowCounter.table_name = table_name
          file.puts "#{table_name}:  #{RowCounter.count}"
        end
      end
    end
    # The rest of your setup code goes here
  end

# And the rest of the TestCase code here
end

This will dump a list of each table with a count of its rows the first time setup is run for the testcase. The file name is generated with a time stamp

Okay, so the example does go along with this weeks Harry Potter fever.

rails_developer@hogwarts.edu.magic.uk/cauldron_project/trunk$ cat data_dump_0717105533
students:  17
professors:  2
owls:  2
potions:  2
magical_creatures:  2
muggles:  0

Now you can run the test individually and then along with others. Then use a diff tool to see which tables have different numbers of rows before your test case starts for the first time.

A test helper

If you find this useful, why not add it to a test helper. If you don’t already have one, edit your test/test_helper.rb file, and add the following code inside of the Test::Unit::Testcase class:

  def self.been_here
    result = @been_here
    @been_here = true
    result
  end

  class RowCounter < ActiveRecord::Base
  end

  def dump_table_counts(file_name)
    File.open(file_name, "w") do |file|
      Account.connection.tables.each do |table_name|
        RowCounter.table_name = table_name
        file.puts "#{table_name}:  #{RowCounter.count}"
      end
    end
  end

I’ve moved the definition of the been_here method and the class Rowcounter from the individual testcase to the superclass. The only code in the testcase needed to use this is in the setup method, which now becomes:

  def setup
    dump_table_counts("#{RAILS_ROOT}/data_dump_#{Time.now.strftime("%m%d%H%M%S")}") unless self.class.been_here

    # rest of setup code here
  end

Qui ipso probo probatur?

Posted by Rick DeNatale Sat, 14 Jul 2007 19:28:00 GMT

The other day I was working on adding support for user selected time-zones to an existing rails app for a client. As usual I was doing test-first development. One of the things that makes rails such a pleasure is that a good set of tests give confidence that you aren’t breaking “legacy” code.

I also use, and really like, the rails plugin for vim which does lots of nice things like making navigation between the files of rails apps much easier. It also has a nice feature which adds a :Rake command to vim which “does the right” thing contextually. For example if you are editing a migration and enter :Rake, it runs rake db:migrate. If you are in a test file it runs just the single test selected by the cursor, or just that test file if you aren’t positioned to a particular test.



I was doing the latter, and my test was failing, and I was having a hard time debugging it. I tried executing just that single test from the bash command line with:

$ruby test/unit/my_test.rb -n"test_mytest"

and it still failed, no surprise. The same thing happened if I ran the entire test file, in fact, other tests which had worked before were now failing, and I was really mystified now because I didn’t see how I had done anything which had a remote chance of breaking those.

So I figured I probably should test everthing so:



$rake test

And, surprise of surprises, but every test worked. Not only my failing unit tests, but the functional and integration tests as well.

To make a long story short, the problem turned out to be that the tests were failing because, now that the particular model was sensitive to the user’s timezone, it needed access to it’s associated user model, and I hadn’t told the testcase that the users fixture was needed. Running the testcase in isolation, the users table wasn’t being populated for the test case. But running rake test, or rake test:units meant that other tests run before had left the data I needed behind.

Just a little thing that makes fixtures less than ideal.