A recent thread on ruby-core which touches on the issues of subclassing
prompts me to tell another war story.
Back in the late-1980s to early 1990s, the software industry was starting to warm to the possibilities of object-orientation.
I’d gotten to be seen as one of the experts on OO inside IBM so I found myself
in a series of interesting meetings, but inside IBM and with other
companies.
That’s how I found myself at the headquarters of
a well-known company in the Seattle area
which at that time was one of IBM’s major partners.
This company was the major supplier of IBM’s operating systems
for personal computers. I don’t want to give too many hints, so I’ll give a
pseudonym to the CEO, let’s just call him “Gill Bates”, and I’ll refer to the
company as “BigSoftware” or “BS” for short.
Now some of the guys I
used to work with at IBM might remember a co-worker in IBM Cary, and then RTP,
named Gil Bates, different guy.
The meeting was to discuss
the BS plans for a new operating system with an object-oriented
implementation
and application programming interface.
A small team of BS engineers presented to
a small team of IBMers. I found myself sitting across a conference table from
the presenters and right next to Gill.
Frameworks, Frameworks, Everywhere
Let me step back a bit and give a little techno-historical perspective
for those who weren’t involved in software in the late 1980s/early 1990s.
One of the things which was exciting people at that time was the emergence of
application frameworks.
MacApp had been around a bit, and had gotten some publicity. This was a
framework written in Object Pascal which made writing Macintosh applications
considerably easier. MacApp provided inheritable classes
which implemented a vanilla application conforming to the Mac UI guidelines.
Writing a MacApp application consisted of subclassing the framework classes
and adding the behavior which made the app what you wanted it to be.
MacApp spawned other frameworks, ET++ was a port to C++ done in
Zurich by a team, which included Erich Gamma, if I recall correctly.
MacApp itself later was ported to C++, which led to Apple’s ill-fated Pink
project. I’ll probably tell a war-story or two about Pink here sometime.
Of course the framework notion has long legs, for example Rails is such a
framework for writing web-based applications.
Back then, the idea of applying the idea not just to applications but to
operating systems seemed natural to many. MacApp and its ilk contained
object-oriented wrappers to OS apis. Why not just make the whole OS a
framework?
As an OO evangelist I’d been caught up in this as well, selling the idea of
frameworks. I was still pushing for application frameworks, but the idea of a
framework-based OS seemed pretty neat until I started thinking harder about it.
The Blessing and Curse of Inheritance
Object Orientation to my mind is about encapsulation. The implementation of
an object is hidden from the users of the object. Inheritance is a powerful
implementation technique, but it breaks the encapulation between superclass, class, and subclass. Like it or not, subclasses need to be written with knowledge
of,
and track changes to,
the implementation of their superclass. That’s okay as long as the
superclass remains relatively stable.
The problem is that you can’t expect that kind of stability over time. As
the framework evolves, new methods and features get added to the framework
provided superclasses, and worse from the point of view of the framework
subclasser, the inheritance relationships between the framework classes often
change. It’s the natural result of refactoring.
I’d played with about three releases of MacApp, and noticed that with each
release, I needed to rewrite substantial parts of my applications if I wanted
to step up to the new version. This was okay in that,
since the MacApp framework classes became part of my application,
the application was a self-contained
whole, and would continue to run as long as I didn’t rebuild it with the new
framework version.
Dealing with the inevitable refactorings is feasible if you have control
all of the code being refactored, or at least can delay refactoring by
ignoring new versions of code not under your control.
The problem becomes much more difficult
when the operating system interfaces require your code to inherit from the
system. New operating systems will break existing applications.
Inheritance is a mechanism for sharing code DNA.
In that sense it’s like sex.
It’s best when it’s restricted to consenting adults, practicing using
safe techniques, and aware of the potential consequences.
The State of the Art circa 1990
The difficulty of maintaining compatibility between releases is exacerbated
when the techniques used to provide inheritance are statically bound. At the
time of my tale, it was conventional wisdom that in order to get reasonable
performance out of an object oriented program the classical C++ approach was
the only way to go. This involved using arrays of pointers to the method
functions, called virtual function tables, and mapping method names to an integer sequence from 1 to the number
of methods needed. For a given class, the number of the first method it
introduced (i.e. a method which was not just an override of an inherited method)
would be one more than the maximum number assigned to its superclass’s methods.
These numbers were determined statically at compile time, and needed to be known
when a call to a method was compiled.
This meant that a simple act such as adding a new method definition in a
class would require recompilation of not just that class,
but all of it’s subclasses,
and any code which called a method on any instance of the class or its
subclasses. Besides making for long build times, it also meant that such a
change would break binary compatibility even in cases where subclassing or
client code which didn’t use the new method and so didn’t need to have its
source code changed.
Inside IBM, something called the System Object Model was being advocated.
This was an attempt to make the C++ implementation just described a little more
flexible to at least partially address this problem, an issue which was dubbed
the “fragile base class” problem.
But the number of refactorings which can be successfuly addressed just
by loosening up the method calling mechanisms is a small subset of the
refactorings which normally occur over the life-cycle of a framework.
Early binding issues or not, even with a dynamic language like Smalltalk
or Ruby, maintaining interface compatibility over time is a very difficult
problem.
Rails, like MacApp, addresses this by letting applications be frozen to
a particular version of the Framework.
Back to the Conference Room
So as you’ll recall, I found myself sitting next to “Gill Bates” while his
senior architects presented their framework based operating system to be written
in C++. I gently introduced the problems I just described, and it took a while
for it to sink in.
I kept re-asking my questions as they came up with inadequate answers.
Meanwile I noticed that, sitting next to me, “Gill” was starting to rock
back and forth in his chair.
Finally, I said:
I just don’t understand how you are going to maintain compatibility
after you refactor the framework for the second release.
The reply was:
We have our best people designing the system,
we won’t have to refactor.
At that point, I suggested that the best people would recogize the
inevitability of refactoring. At which “Gill” finally chipped in by saying
to his presenter:
You guys don’t explain our stuff very well.
I’ve been working on a little tool to peek behind the curtain and see a bit of what’s going on behind the scenes in the standard Ruby implementation (i.e ‘ruby’ as opposed to ‘Ruby’)
While doing this I was looking at the code which ruby runs when you include a module in a class or another module. I noticed that ruby 1.8 was going to some pains not to move the proxy for an included module in the inheritance chain.
To verify what my eyes seemed to be telling me, I wrote a silly little test program which created a module with one method, and several classes.
This test verified my reading of the 1.8 code. I then tried the same test using the latest ruby 1.9 and found that module mixin semantics have changed.
Read on
Here’s my test code:
module M1
def foo
"foo in M1"
end
endmodule M1
def foo
"foo in M1"
end
end
module M2
include M1
end
class C1
def foo
"foo in C1"
end
end
class C2 < C1
include M1
end
class C3 < C2
def foo
"foo in C3"
end
end
class C4 < C3
include M1
end
class C5 < C3
include M2
end
puts "C1 #{C1.new.foo}"
puts "C2 #{C2.new.foo}"
puts "C3 #{C3.new.foo}"
puts "C4 #{C4.new.foo}"
puts "C5 #{C5.new.foo}"
Module mix-in semantics in ruby 1.8
And here’s what we get when we run this using ruby 1.8, with commentary interspersed:
rick@frodo:/public/rubyscripts$ ruby include_test.rb
C1 foo in C1Note that class C1 defines a method :foo. There should be no surprise that invoking :foo on an instance of C1 gets that definition.
C2 foo in M1C2 subclasses C1, and also includes module M1, which also defines a method :foo. We can see that M1’definition takes precedence. The search order is, first a singleton class if it exists, then the object’s class, then any modules included in that class in the reverse order of inclusion, then the superclass, etc.
C3 foo in C3C3 subclasses C2 and overrides the foo method. Again no surprise.
C4 foo in C3Here’s the case which illustrates what I found in the 1.8 code. C4, a subclass of C3 again includes M1, but instances of C4 resolve foo to the method in C3. This seems to be odd, since by including M1 in C4, I would expect M1’s methods to take precedence over those of the superclass.
C5 foo in C3This final case is a class, C5 which like C4 subclasses C3, and includes a second module M2, which in turn includes M1. Once again foo resolves to the method defined in C3.
I find the 1.8 implementation counter-intuitive. As I said it goes to lengths to find an existing inclusion of a module, and re-use it, even if that inclusion was in a superclass.
Change in ruby 1.9
I guess that the core-team agreed with that assessment, or at least the current 1.9 implementation seems to do insertion in a less surprising manner. Here’s the result of running the same code with ruby 1.9:
$ ruby1.9 include_test.rb
C1 foo in C1
C2 foo in M1
C3 foo in C3
C4 foo in M1
C5 foo in M1For easier comparison here’s the output from ruby 1.8 without commentary.
$ ruby include_test.rb
C1 foo in C1
C2 foo in M1
C3 foo in C3
C4 foo in C3
C5 foo in C3I don’t know about you, but this seems more natural to me.
It’s a subtle change. I don’t know how much ruby code re-mixes in modules. I only wrote my test code to verify a quirk I saw in reading the ruby sources.
Since the ruby 1.8 implementation made doing so ineffective, I doubt that there is much code which does, but if there is such code, it might break under 1.9, but I think that the change is a good one.
Recently, a certain “gentle-reader” questioned my statement in my “mini-memoir” that Simula lacked classes and inheritance.
I stand by my guns. Simula (now known as Simula I) had neither. These were introduced by Simula-67 about five years later. It was Simula rather than Simula-67 which was one of the influences on Alan Kay’s early conception of Smalltalk.
For more details, see my reply to “gentle reader” in the comments to my mini-bio article.