Talk Like A Duck: Tag matz

A Chat with Matz, Classs Variable reversion, and A Mystery Explained

Rick DeNatale — Sat, 03 Nov 2007 02:51:00 -0400

I finally got to meet Matz in person today, and we had a nice little chat.

I was pleased to learn that he follows this blog. My readership might be small, but it's quality readership. One thing I told him was that as much as I love, and still love, Smalltalk, I really do feel that I love Ruby just a little bit more.

We talked a bit about the upcoming stabilization of 1.9, which will be frozen as far as language and standard libary changes for the Christmas 2007 1.9.1 release, at which time transition from development to stable status. He has been backing out some of the differences between 1.8.x and 1.9.0. He told me that he had recently, or was soon to, revert the changes to the scoping of class variables. It appears that class variables will continue to be shared between classes and subclasses.

He also cleared up a technical ruby mystery that's been puzzling me for some time.

About a year ago I wrote about a change in Ruby 1.9 which cleaned up the semantics when modules included by a class are re-included in a subclass. The mystery is that sometime after I wrote the article, in a subsequent revision of 1.9, this got dropped. I never could find out why this happened, so today I took the opportunity to ask Matz in person.

He agreed that the way that it worked temporarily was the right way for it to work, but it got dropped because YARV had difficulties with the change in implementing the super keyword.

We didn't go into further details, but here's my informed speculation on the situation.

When you send a message to a Ruby object, what happens, at least notionally, is that a linked list of (pseudo)objects is searched to find the first one which contains a method with that name in its method hash. Let's call this the method lookup chain. The head of the chain is pointed to by the receiving objects klass field, and is either the singleton class of the object if it has one, or the Class of the object, links in the chain represent modules and classes in the ancestry of the object. If the end of the chain is reached a method_not_found message is generated and searched for beginning again from the beginning of the receiving objects method chain.

Each link is either a class or a pseudo object marked as an "IClass" (for Included Class) whose method hash pointer points to the method hash of the Module it represents. Because module method hashes are shared in this way, changes modules methods will be visible to every class or module which includes the module immediately.

When the 'receiver' is super, the same kind of search occurs as for a send to self, but instead of starting with the receivers method chain, we want to search starting with the link in the chain after the one in which the currently executing method was found.

Smalltalk implementations typically reify methods as CompiledMethod objects which have an instance variable which refers to the class in which the method was defined. So a send to super simply starts the search with the superclass of the currently executing CompiledMethods definition class.

This approach doesn't work for Ruby because Modules can be included in multiple classes or modules, which means that methods defined in Modules can appear in multiple places in multiple method lookup chains.

One way to find the right place to start searching for a method with super as the receiver is to start from the beginning of selfs method lookup chain, search until we find the currently executing method, then start the super search with the next link in the chain. I believe that this is how Ruby works, except for that brief time in the evolution of 1.9.

In Smalltalk you must name the message when you do a send to super. In Ruby you can't. Smalltalk, unlike Ruby, allows you to send a different message than the current one to super. In Smalltalk, the keyword super effectively means "self, but resolve any messages starting with the superclass of the class which defined the currently executing method," while in Ruby it means something like, "the current message, but look for me after where you found the currently executing method." This difference simplifies finding the current method in the chain. We know we're looking for a method with the same name in both the first and second searches. Since each link in the chain has a hash of methods, it's generally faster to do a hash lookup on the name, and in the first stage of the search compare the value returned with the current method.

This two-phase search for super runs into a small snag when a modules method hash appears more than once in a given method lookup chain. The first stage super search can find the wrong place in the chain.

Here's some code similar to that in the earlier article.

module M
  def foo #foo_m
    puts "foo in M1"
    super
  end
end

class C1
  include M
end

class C2 < C1
  def foo #foo_c
    puts "foo in C2"
    super
  end
end 

class C3 < C2
  include M
end

Admittedly this code is contrived, it's specifically crafted to expose the issue in the simplest way, C1, C2, and C3 would probably have other methods which don't need to be shown here.

If every module inclusion in the above code created an IClass then the method chain for instances of C3 Would look like this:

The boxes with names inside represent the class objects, empty orange boxes represent IClass pseudo-objects, the blue box labeled M is the module object, and the blue labels represent the methods.

Okay, now let's say we evaluate the expression:

C3.new.foo

We first search the method chain for a foo method and the first one we find is foo_m in the first IClass, so we invoke it.
We reach the super in foo_m, so we start at the beginning of the method chain to find it which we do in the first IClass, since this is a send to super, we then look for the next mathod named foo, and we find foo_c in class C2, so we invoke it.
We reach the super in foo_c, so we start at the beginning of the method chain to find it which we do in the first IClass, since this is a send to super, we then look for the next mathod named foo, and we find foo_c in class C2, so we invoke that.
We reach the super in foo_c, so we start at the beginning of the method chain to find it which we do in the first IClass, since this is a send to super, we then look for the next mathod named foo, and we find foo_c in class C2, so we invoke that.
Houston, we have a problem!

So, including IClasses for a module more than once can lead to an infinite loop when there's a super, if the two-part re-search is used.

And this is most probably the reason why the code to include a module checks to see if the module is already included and doesn't actually re-include it in this, admittedly rare case.

So late last year, some other mechanism must have been being used to find the start of the super method chain, to allow real module re-inclusion. Perhaps, whenever a method was invoked, a pointer to the Class or IClass where it was found was placed on the execution stack so that it could be found if needed to resolve super, perhaps some other mechanism was used.

Whatever method was used, it was apparently incompatible, or hard to implement efficiently with YARV, so when YARV became a standard part of 1.9, the old module inclusion logic came back.

Oh well, being able to re-include really makes the semantics of Module inclusion clearer, seems a shame, albeit perhaps a rather small one.

Town Meeting with Matz at RubyConf 2007

Rick DeNatale — Fri, 02 Nov 2007 20:50:00 -0400

Photo by Nathaniel Talbott.

This was not quite the largest audience Matz has faced there were about 500 in attendance at this evening's "Town Hall Meeting", he asid that he had spoken to a group of over 1000 in Japan, but that was not to an exclusively Ruby audience.

I captured most of the questions and answers as best as I could. Any errors of paraphrasing are mine:

Q1
What about backwards compatibility. My program broke when Ruby 1.9 changed File.exists? to File.exist?
A1: We try to keep the language as backwards compatible as possible, but File.exists? was an old method inconsistent with the naming of predicates.
Q2: (From an Engineer from TI looking at putting Ruby into new calculators) I don't like the licence on the regexp library function in 1.8. The regexp engine in 1.9 isn't "encumbered" by the GPL?! When will 1.9 be out.
A2: Ruby 1.9.1 will be out by Christmas 2007, but might not be as stable as we want.

(Note from Rick) I'd talked earlier to Matz about this. The Christmas release will freeze the language and library APIs, it's the implementation which is expected to change to fix bugs and improve performance. That's what he means by "not as stable."

We had a problem with our app due to integer performance, why not change Fixnum to not go to Bignums automatically?

We won't "bend the language" for performance. Big Applause.

What do you think about the competition from JRuby, Rubinius, IronRuby?

Having the alternative implementations is wonderful. They open up a new world for people who couldn't use ruby for various reasons.

Is working on Ruby still something that makes you happy? What's the most enjoyable thing about it.

I started on Ruby in 1993 with the goal of enjoying myself, I enjoy programming in Ruby, I enjoy designing it, and I enjoy implementing it. I still enjoy it very much.

What's the Ruby community and presence like in Japan.

It's pretty much the same as the people here at RubyConf. They're nice people, fun, and interested in new technology. More people are actually earning their living from/with Ruby in America than Japan.

Does the increasing use of Ruby here in the States cause you to feel pressure? For example the TI question about licensing?

I try to stay away from the enterprise contracting issues.

What Editor do you Use?

Emacs. Mixed reaction.

(Charles Nutter) When can we expect a final 1.9 spec?

The features are almost fixed. I'm now in the process of auditing 1.9 changes. I'm going to reverse some of the 1.9 changes in a week or two. By the middle of this month (November 2007) the features will be pretty much stable.

Q10

In the matter of Ruby, do you consider yourself a Scientist, or an Artist and why?

A10

A Hobbyist. Ruby started out as my own toy language, I expected just a few users of the language. Now we have 500 ruby users in this room. I don't know how to count the number in the world, perhaps hundreds of thousands.

Q11

Would you ever consider exposing the parse tree to ruby programs?

A11

The parse tree is pretty implementation dependent so exposing it as a spec would be tough for us to support in the future.

Q12

What other languages are you looking at now?

A12

Erlang, I don't like the syntax but there are some interesting ideas in it.

Q13

Someone said that there wasn't much room to improve the implementation of Ruby/YARV, but a Smalltalk programmer told me about two techniques used by Smalltalk VMs, Polymorphic Inline Caching and Dynamic Compilation are they applicable?

A13

Did I say that there was no room for improvement? We know about such techniques and are looking at them for Ruby.

Q14

Rails has done a lot to boost the usage of Ruby. Do you see other areas which might provide similar frameworks/things to boost Ruby.

A14

We're looking at things to improve processing large data sets on multi-core platforms.

Q15

Some people in the Java world say that Ruby is great for fast development, fast compile times, but needing static typing and other less dynamic features to scale up, is this a problem for Ruby.

A15

Well there are things which make Ruby harder to use with extremely large teams. Java still shines in large scale enterprise development, but the rules are changing, there's more emphasis on smaller quicker projects, and Ruby shines there. We will probably look at some things like better name scoping to help with larger team projects but that won't be until the version of Ruby after 1.9.

A16

The beauty of Ruby to me is that it's "just one step ahead" of the programmer compared to say lisp, erlang, which are harder to understand.

Q17

What's your second favorite programming language?

A18

Several

Q18

What would you remove from Ruby, is there anything that doesn't belong?

Q18

Matz didn't seem to be able to think of anything.

Q19

Have you looked at the Toyota Lean Process? Are you familiar with it? Is it applicable to Ruby/Software development?

A19

It's interesting that there are so many parallels between auto manufacturing and softwere development.

Q20

Are continuations in or out of 1.9.

A20

You now need to require a library file, so they are out of the core language, but they're not completely out.

Q21

What about parallelism?

A21

We are looking at options like running multiple independent VMs on threads in the same process, with shared code.

Q22

You had talked before about selector namespaces

A22

It's tough to implement but we are looking at it for Ruby 2.0.

Q23

Do you see a difference in the coding style between American and Japanese rubyists?

A23

I don't really focus on it. Although recently code 'golfing' has seemed to become popular in Japan.

Q24

One of the key features of the IO language is that it has no keywords. Have you looked at that as a potential goal for future versions of Ruby.

A24

IO is interesting, but that's not the way that Ruby is going.