journal.stuffwithstuff.com

Playing with Generics in TypeScript 0.9.0

2013-04-23T00:00:00-07:00

This was just going to be a comment on this reddit thread, but then it seemed to take on a life of its own, so I figured I may as well ~~milk it for all it's worth~~ make a nice post out of it.

Yesterday, the TypeScript guys announced a preview of the new 0.9.0 version of the language featuring generics. I, like a lot of people, was really curious to see what approach they’d take with them. Retrofitting a type system onto a dynamic language is hard and generics are one of the places where that new suit of armor can really chafe the squishy flesh underneath.

This is a topic strangely near to my heart for another reason too. I’m on the Dart team and when Dart was announced, we were widely criticized for its type system. Generics are covariant in Dart, which is a mortal sin to many.

Now when any new type system comes out, my first thought is, “I wonder if it’s got covariant generics?” (My second thought is, “God, I need a hobby that doesn’t involve programming languages.”)

Some more caveats before I get going:

I literally spent five minutes poking at this, so I may have things wrong.
Compiling programs with a compiler that isn’t 1.0 yet and claiming that says something about the language specification is asking for trouble.
I, despite my current occupation, am surprisingly bad at reading language specs.
I am currently drinking a fairly strong beer.

OK, party time! After playing around with it a bit, as far as I can tell, TypeScript’s subtype relations are more permissive than I expected. This isn’t necessarily bad, just surprising. As a preamble, let’s define a supertype and subtype:

class Base {}
class Derived extends Base {}

Now consider:

class Box<T> {
  constructor(public value: T) {}
};

This is the simplest possible generic type.

var a : Box<Base> = new Box<Base>(null);
var b : Box<Derived> = new Box<Derived>(null);

These are both fine, as you would expect.

new Box<number>("not num")

This gives:

/generics.ts(5,0): error TS2081: Supplied parameters do not match any signature of call target.
/generics.ts(5,0): error TS2085: Could not select overload for 'new' expression.

Looks about right. Now let’s try covariance:

var c : Box<Base> = new Box<Derived>(null);

No errors. Contravariance?

var d : Box<Derived> = new Box<Base>(null);

Still no errors. This, I think, makes its type system looser than arrays in Java, and more permissive than Dart. Generics are bivariant in TypeScript.

var e : Box<number> = new Box<string>(null);

As a sanity check, this does give an error.

/generics.ts(73,4): error TS2012: Cannot convert 'Box<string>' to 'Box<number>':
Types of property 'value' of types 'Box<string>' and 'Box<number>' are incompatible.

So it doesn’t just ignore the type parameters, it really is bivariant: it will allow either a subtype or supertype relation for the type parameters, but not no relation at all.

Part of this may be because TypeScript’s type system is structural (neat!). For example:

class A {
  foo(arg : Base) {}
}

class B {
  foo(arg : Derived) {}
}

Here we have two unrelated types that happen to have the same shape (method names and signatures). Perhaps surprisingly, there’s no error here:

var f : A = new B();
var g : B = new A();

This is perhaps extra surprising because A and B don’t have the exact same signatures: their parameter types for arg are different. So the type system is both structural and allows either supertype or subtypes on parameters.

If the type system is structural, maybe the Box<T> examples only worked then because it had no methods (aside from the value property) that used the type parameter. What if we make sure T shows up in parameter and return positions?

class Box<T> {
  constructor(public value: T) {}
  takeParam(arg: T) {}
  returnType(): T { return null; }
};

Nope, this makes no difference. Still same behavior as before.

I believe the relevant bits of the spec are:

3.6.2.1 Type Arguments

A type reference to a generic type G designates a type wherein all occurrences of G’s type parameters have been replaced with the actual type arguments supplied in the type reference.

I think this basically says that generics are structurally typed and the type relation is determined based on the expanded type where type arguments have been applied. In other words, generic types don’t have type relations, just generic type applications. For example:

class Generic<T> {
  method(arg: T) {}
}

class NotGeneric {
  method(arg: number) {}
}

var h : NotGeneric = new Generic<number>();

This is fine in TypeScript unlike most nominally-typed languages. Pretty neat!

3.8.2 Subtypes and Supertypes

S is a subtype of a type T, and T is a supertype of S, if one of the following is true:

S’ and T are object types and, for each member M in T, one of the following is true:

M is a non-specialized call or construct signature and S’ contains a call or construct signature N where, when substituting ‘Object’ for all type parameters declared by M and N (if any),

for parameter positions that are present in both signatures, each parameter type in N is a subtype or supertype of the corresponding parameter type in M,

the result type of M is Void, or the result type of N is a subtype of that of M

…

…

…

My emphasis added. I believe this basically says that to compare two types, you walk their members and compare their types. For method parameters, “subtype or supertype” means bivariance: types go both ways. This is looser than the normal function typing rule which is contravariance for parameters and covariance for returns.

All in all, I find this pretty interesting. TypeScript has both a structural and prototypal type system, so it’s already pretty fascinating from a language design perspective. Allowing bivariance of parameter types is a pretty bold extension of that.

Oh, if you’d like to see this for yourself, here’s how:

$ git clone https://git01.codeplex.com/typescript
$ cd typescript
$ git checkout develop
$ jake local
$ chmod +x bin/tsc

Then you can run bin/tsc to compile stuff with the bleeding edge compiler. The latest spec is under doc/TypeScript Language Specification.pdf.

Well Done: A Sentinel Value

2013-04-17T00:00:00-07:00

This is kinda like part three of the Iteration Inside and Out posts, so you may want to check out part one and part two too. But you don’t have to.

Most programming languages have one or two singleton values floating around. These are special built-in objects that only have a single instance. null or nil are common. Some languages put true and false in the same bucket. JavaScript, ever the hipster, ironically defines one called undefined.

I wanted to talk a little bit about one I just added to my little language Magpie: a sentinel value called done. I’m still not certain it’s a great idea, but I’ll walk you through my thoughts. If nothing else, maybe it will serve as a cautionary tale for other, smarter language designers.

Channels

I ran into the problem that ultimately led to done when I started working on Magpie’s concurrency story. Its model is based on fibers and channels, much like Go or other CSP-inspired languages. You can spin up a new fiber using the async keyword:

async
    print("I'm in another fiber!")
end

Fibers are cooperatively scheduled so they only yield to other fibers when they do something that requires waiting. Usually that means IO. So if you do the above, you won’t see that message get printed until something causes the main fiber to yield. For example:

async
    print("I'm in another fiber!")
end

print("Before")
print("After")

This program will create a second fiber (but not switch to it). Then it gets to print("Before") in the main fiber. Since printing is IO, that switches to the second fiber. That in turn queues up its print and suspends again, so the main fiber resumes. Ultimately, it prints:

Before
I'm in another fiber!
After

You can spin up lots and lots of fibers because they don’t use OS threads, just some memory in the interpreter. This is swell for decoupling stuff and running things concurrently. But sometimes you do need to coordinate fibers with each other. For that, we’ve got channels.

A channel is a simple object that you can send objects into and then receive them from another channel. You can create one like:

var channel = Channel new

To send a value along it, just do:

channel send("My value")

And you can receive it like:

var result = channel receive

The fun bit is that when a fiber sends a value along a channel, it causes that fiber to suspend until some other fiber shows up to receive the value. A send doesn’t complete until the object has been received. So channels let you not just communicate but also synchronize.

Show’s Over Folks

There’s one other thing you can do with a channel, you can close it:

channel close

That puts the channel out of commission and tells everyone that they will receive no more values from it. The question I ran into was, “what happens to fibers that are waiting to receive on a channel when it gets closed?” For example:

var channel = Channel new

// Wait for a value and print it.
async print(channel receive)

// From the main fiber, close the channel.
channel close

Here, the second fiber is sitting there, relaxing, maybe having a cocktail while it waits for the channel to spew something forth. But it doesn’t, it gets closed. The fiber shouldn’t just stay suspended forever, so how should it get notified?

One option would be have receive throw an error when the channel gets closed. So you’d handle it something like:

async
    do
        print(channel receive)
    catch err is ChannelClosedError then
        print("Sorry, no value for you")
    end
end

Actually, any block can have a catch clause in Magpie, so the explicit do here is redundant. You’d really just do:

async
    print(channel receive)
catch err is ChannelClosedError then
    print("Sorry, no value for you")
end

That’s not too bad, but consider something a little more fleshed out. In many cases, you’ll receive something from a channel and do different stuff based on what you get. Channels are often used for messages and the specific message will often cause different behavior. You can imagine something like:

var channel = Channel new

async
    var count = 1
    while true do
        match channel receive
            case "inc" then print(count = count + 1)
            case "dec" then print(count = count - 1)
        end
    end
end

Here we’ve got a little concurrent counter that you can send messages to in order to increment and decrement the value. If it also needs to handle the channel closing, you’d need to do:

async
    var count = 1
    while true do
        match channel receive
            case "inc" then print(count = count + 1)
            case "dec" then print(count = count - 1)
        end
    catch err is ChannelClosedError then
        print("Closed")
    end
end

That feels redundant to me. Catching errors is basically pattern-matching (in fact, in Magpie it uses the exact same syntax and semantics as a match expression), so it feels dumb to have to have two matches, one for the set of normal values and then this extra catch for the one special “no more values” signal.

It’s not just redundant either. Even if you don’t care about the channel closing, you have to add the catch here. Otherwise, since it’s a thrown error, it will unwind the fiber’s callstack on you.

Both of those are no fun, so I figured, why not just make receive return a value when the channel is closed instead of throwing one. Magpie already has a null-like sentinel value called nothing, so I could just say receive returns nothing when the channel is closed.

But that’s a bit lame. While nothing doesn’t appear as often as null does in other languages, it’s still a general-purpose “no value here” object that users need to be free to use anywhere. If receive returns nothing when the channel is closed, there’d be no way to explicitly send nothing along an open channel. Receivers wouldn’t be able to tell the difference.

Are We Done?

So I made another singleton value: done. It exists solely to represent “no more values”. With this, the above code is like:

async
    var count = 1
    while true do
        match channel receive
            case "inc" then print(count = count + 1)
            case "dec" then print(count = count - 1)
            case done then print("Closed")
        end
    end
end

It may seem weird to have a method return values of different types, but this is natural in Magpie. The language is dynamically typed, and revolves heavily around pattern-matching. In, say, JavaScript, if you have a function that can return a number or a string and callers will want to handle those cases differently you get this ugly code:

var result = foo();
if (typeof result === 'number') {
  console.log("number");
} else if (typeof result === 'string') {
  console.log("string");
}

In Magpie, that’s:

match foo()
    case is Num then print("number")
    case is String then print("string")
end

Or, since the language is expression-oriented, you could even do:

print(match foo()
    case is Num then "number"
    case is String then "string"
end)

In fact, because Magpie also has multimethods, even this would do the right thing:

def showType(is Num) print("number")
def showType(is String) print("string")

showType(foo())

Where was I? Oh, right. So it’s totally kosher to have methods return values of different types and have callers dispatch on them. That was enough to talk myself into being OK with receive having a sort of ad-hoc variant-like return value.

The Next Iteration

Once I had that, I started fleshing out channels a bit more. One thing I noticed is that you often read from a channel repeatedly. See that while true do loop up there? That seemed a bit silly. Magpie has for loops. Like most languages, those are syntactic sugar for an iteration protocol. If I made channels support that directly, you could do:

async
    var count = 1
    for value in channel receive
        match value
            case "inc" then print(count = count + 1)
            case "dec" then print(count = count - 1)
        end
    end
    print("Closed")
end

This protocol has two levels: iterable and iterator. The iterable protocol is for an object that can be iterated: lists, collections, etc. There’s just one method:

def (is Iterable) iterate

An iterable type specializes iterate to return a new iterator for the object. This lets you iterate over objects without modifying the original object. It would be pretty weird if you could only use a list in one loop at a time.

In the case of channels, though, iterating it is mutating the original object, so iterate on a channel just returns the same object. Channels are both iterables and iterators. The iterator protocol was two methods:

def (is Iterator) advance

This tries to advance the iterator to the next item and returns true if it found another item.

def (is Iterator) current

This returns the current item. So there was a two method protocol here. Most other languages have something similar. You need two methods because one crank of the iterator needs to return two bits of data: whether or not you’re done iterating, and the next value if you aren’t.

I say “was a two method protocol” because now that I had this done value, I realized I could simplify it. All I need is advance. It returns the next item if there is one, or done if there isn’t. The only thing you lose is the ability to iterate over a collection that actually contains done as a value. But I’m OK with that. That seems like a small thing to give up to get a simpler protocol.

Now, not only can you use channels any place you work with sequences, but even implementing your own sequences is a little easier.

Your Favorite Functional Friends

When I say “any place you work with sequences” much of what I’m thinking of is those workhorse methods that transform sequences. Good old map, filter, fold, etc. This gives you a feel for what they look like in Magpie:

var list = [1, 2, 3, 4]
list map(fn(i) i * 3) // 3, 6, 9, 12
list where(fn(i) i % 2 == 0) // 2, 4

Since I specialized these on Iterable, that means any iterable type, including channels, gets them. For a longer example, Go has an example showing a concurrent Sieve of Eratosthenes. Ported to Magpie, it looks like:

// Send the sequence 2, 3, 4, ... to 'channel'.
def generate(channel is Channel) async
    for i in 2..9999 do channel send(i)
end

// Copy the values from 'input' to 'out',
// removing those divisible by 'prime'.
def filter(input is Channel, out is Channel, prime is Int) async
    input where(fn(i) i % prime != 0) pipeTo(out)
end

// The prime sieve: Daisy-chain Filter processes.
var channel = Channel new
generate(channel) // Spawn generate fiber.

for i in 0..50 do
    val prime = print(channel receive)
    val sieve = Channel new
    filter(channel, sieve, prime)
    channel = sieve
end

I won’t walk you through the whole thing, but I do think it’s pretty rad. The neat bit is this line:

input where(fn(i) i % prime != 0) pipeTo(output)

Here, where is Magpie’s filter function. It takes an iterable on the left and a predicate function on the right. It returns an iterable generating all of the items of the original sequence where the predicate returns true. Note that here, the argument on the left is a channel so this magically becomes an asynchronous filter.

Likewise, pipeTo is a method that takes an iterable and a channel and sends all of the values in the iterable to the channel. So we’re taking a channel, filtering it like a collection, then pumping that collection back into a new channel. This all happens asynchronously, but the code looks like you’re just playing with regular collections.

Crap, I’ve digressed again. I was supposed to be talking about done. I just think treating channels as sequences is really cool. OK, back to the matter at hand.

Now Are We Done Yet?

The simplest of the higher-order methods for working with iterables is each. All it does is call the given function for each item in the sequence. In other languages this is sometimes called forEach. Most of the time you don’t need it since it is almost identical to just using a built-in for loop. But sometimes it’s convenient, so Magpie has it.

If you’ve worked with forEach in JavaScript, one nasty problem is how do you stop early if you don’t actually need to walk the whole sequence? Ruby has some special syntax built into the language, break and next, that are designed to work specifically with blocks and iterators. JS ain’t so lucky.

Magpie has break for for loops, but a method like each is just a method that takes a function. You can’t break from within a function. But… since we have this done value laying around, we can use it here too. So the way each (and map and where and others) work is if the callback ever returns done that means “stop iterating now”.

For example:

[1, 2, 3, 4] map(fn(i) if i < 3 then i * 2 else done)

This will not only transform the values (i.e. what map usually does), but it will also truncate the sequence and just return [2, 4].

So we’ve got an easier way to handle closed channels, a simpler iterator protocol, and more expressive collection-manipulation methods. Not too bad for one little sentinel value.

Iteration Inside and Out, Part 2

2013-02-24T00:00:00-08:00

You’ll probably want to have read part one first unless you’re feeling brave.

In our last episode, we learned that iteration involves two chunks of code: one generating values, and one consuming them. During a loop, these two chunks take turns in a cycle of generate, consume, generate, consume, like some sort of weird incremental Ouroboros. This means that one chunk has to fully return and unwind its stack frames before it can hand off to the next one.

We also learned that external and internal iterators each nail some problems and totally fail at others. The discrepency boils down to which chunk of code has more useful stuff to store on the callstack.

With external iterators, the code consuming values has control over the stack, so it works well with problems where most complexity is in consuming values. For example, short-circuiting or interleaving multiple iterators is trivial in an external iterator.

Conversely, internal iterators put the code generating values in control. They excel when generating values is complicated, like walking a tree and iterating over the nodes.

Now we’ll see some techniques to deal with this. The basic idea is reification. If you’ve got some data on the callstack that you want to hang onto, you need to find a place to store it.

Iterators and generators

Say we want to define a method that concatenates two sequences. We don’t want to actually create a data structure that contains the elements of both, we just want to return an iterator that walks the first sequence and then the second one. Here’s how you could do that in C#:

IEnumerable Concat(IEnumerable a, IEnumerable b)
{
  return new ConcatEnumerable(a, b);
}

class ConcatEnumerable
{
  IEnumerable a;
  IEnumerable b;
  ConcatEnumerable(IEnumerable a, IEnumerable b)
  {
    this.a = a;
    this.b = b;
  }

  IEnumerable GetEnumerator()
  {
    return new ConcatEnumerator(
        a.GetEnumerator(), b.GetEnumerator());
  }
}

class ConcatEnumerator
{
  IEnumerator a;
  IEnumerator b;
  bool onFirst = true;
  ConcatEnumerator(IEnumerator a, IEnumerator b)
  {
    this.a = a;
    this.b = b;
  }

  bool MoveNext()
  {
    // Which sequence are we on?
    if (onFirst)
    {
      // Stay on first.
      if (a.MoveNext()) return true;

      // Move to the next sequence.
      onFirst = false;
      return b.MoveNext();
    }

    // On second.
    return b.MoveNext();
  }

  Object Current
  {
    get { return onFirst ? a.Current : b.Current; }
  }
}

Oof, that seems like a pile of code for such a simple goal. The problem is that we’ve got all of this state to maintain: the two sequences being iterated, which one we’re in, and where we are in it. Since this is an external iterator, we can’t just store that as local variables on the stack because we have to return from MoveNext() between each item.

But but but! C# has something called iterators. (A confusing name. What other languages call “iterators”, C# calls “enumerators”. So “iterator” means something special in C#-land.) The above code can also be written:

IEnumerable Concat(IEnumerable a, IEnumerable b)
{
  foreach (var item in a) yield return item;
  foreach (var item in b) yield return item;
}

How’s that for an improvement? (Note: if your employer pays you by the line, you’ll want to avoid this.) The magic here is yield return. When a method contains it, the compiler turns the method into an iterator. You can think of it sort of as a “resumable method”. When you call Concat(), it runs to the first yield, then stops. Then when you resume it, it picks up where it left after the yield.

So here it iterates through the first sequence, stopping at each item and returning it. Then it does the same thing with the second sequence. But what does it mean to “resume” a method?

The return type here clarifies that. When you call Concat(), what you get back is an IEnumerable. This is C#’s iterable sequence type. So what you get back is a “collection”. “Resuming the method” just means “get the next value in the sequence”.

So, given the above, we’ve got a nice solution to our original problem. We can use it like:

foreach (var item in Concat(stuff, moreStuff))
{
  Console.WriteLine(item);
}

Using yield lets us store all of the interesting state—the two sequences and our current location in them— just as local variables right in the Concat() method. C# will reify that stuff for us so that when the Concat() method “returns”, that data gets squirreled away somewhere safe.

You may wonder how it does that. That’s kind of the funny bit. In the case of C#, the answer is that the compiler itself will automatically generate a little hidden class exactly like our ConcatEnumerator one up there. The actual runtime (the CLR) doesn’t have any support for yield. It’s done purely by automatically generating code. It’s large delicious lump of syntactic sugar.

In the last post, I crafted some glorious ASCII art showing where all of the state is being stored. The main problem was that the state for the code generating values and the state for the code consuming them both live on the stack. Using an iterator, though, gives you this:

  stack                       heap
+---------------------+
| iterator.MoveNext() |
+---------------------+     +-------------------+
| loop body           | --> | DesugaredIterator |
+---------------------+     +-------------------+
  ...

  main()

So the stack has the state for the code consuming values. But the state needed to generate values lives on the heap. There’s an instance of this little class that the compiler created for us. When MoveNext() returns, we don’t trash the state because it’s still over there in the heap available the next time we call MoveNext().

There are a few other languages that do (or will) work this way. Python calls these “generators” and uses a similar yield statement. The next version of JavaScript will have something similar. The language that invented generators and the yield keyword was Barbara Liskov’s CLU, immortalized forever in Tron. (I’m not making that up.)

There’s a limitation here, though. You can only yield from the method itself. Let’s say (for whatever reason) we wanted to organize our C# code like:

IEnumerable Concat(IEnumerable a, IEnumerable b)
{
  WalkFirst(a);
  WalkSecond(b);
}

IEnumerable WalkFirst(IEnumerable a)
{
  foreach (var item in a) yield return item;
}

IEnumerable WalkSecond(IEnumerable a)
{
  foreach (var item in a) yield return item;
}

What we want to have happen is that the yield return in WalkFirst() and WalkSecond() will cause Concat() itself to yield and return, but it doesn’t work that way. Iterators/generators reify a stack frame for you, but they only reify one. If you want to have your iteration logic call other methods which also yield, you have to manually reify every level yourself by walking the sequence at each level. Something like:

IEnumerable Concat(IEnumerable a, IEnumerable b)
{
  foreach (var item in WalkFirst(a)) yield return item;
  foreach (var item in WalkSecond(a)) yield return item;
}

IEnumerable WalkFirst(IEnumerable a)
{
  foreach (var item in a) yield return item;
}

IEnumerable WalkSecond(IEnumerable a)
{
  foreach (var item in a) yield return item;
}

You see how we’re doing foreach and yield return both in the Walk_ methods and in Concat itself? We’re explicitly making every level of the callstack an iterator. That makes sure every call frame gets reified like we need. Can we do better?

Python 3.3: Delegating generators

The above example can be translated to Python like so:

def concat(a, b):
  for item in walkFirst(a): yield item
  for item in walkSecond(a): yield item

def walkFirst(a):
  for item in a: yield item

def walkSecond(a):
  for item in b: yield item

Aside from being more terse, this is a one-to-one mapping with the C# code. But Python 3.3 adds something new for us here. Let’s use it:

def concat(a, b):
  yield from walkFirst(a)
  yield from walkSecond(a)

def walkFirst(a):
  for item in a: yield item

def walkSecond(a):
  for item in b: yield item

The explicit loops in concat() have been replaced with a new yield from statement. Nice. This makes composing generators a little cleaner. But there’s no real magic here. We still have to have yield at every level of our iteration code.

In most cases, this is just a bit tedious but not a showstopper. But if you have higher-order functions this can actually prevent code reuse. If you have some function that takes a callback, it doesn’t know if that callback wants to yield or not, so it doesn’t know if it needs to yield. You may end up having to implement that function twice, once for each style.

So yield from is only a tiny improvement. Can we do better?

Ruby: Enumerables, Enumerators, and Fibers

If you ever find yourself in a game of “which language has a better feature than X”, Ruby is usually a safe play. Matz has culled an impressive array of features from Smalltalk and Lisp. (And let’s not forget Perl, the ugly duckling paddling around Ruby’s Pond of Inspiration.)

In Ruby, iteration is usually internal. The idiomatic way to go through a collection is by passing a block (more or less a callback, if you’re not familiar with Ruby/Smalltalk parlance) to the each method on a collection. But it also supports external iterators and a for expression. Impressively, it can convert the former to the latter. (Going the other way is trivial in any language.)

Let’s dig up an example from the previous post. Here’s some Ruby code for defining a tree and iterating over the nodes of the tree in order:

class Tree
  attr_accessor :left, :label, :right

  def initialize(left, label, right)
    @left = left
    @label = label
    @right = right
  end

  def each(&code)
    @left.each &code if @left
    code.call(self)
    @right.each &code if @right
  end
end

We can use it like:

tree.each { |node| puts node.label }

This is using internal iteration. We pass in that { |node| ... } block and the Tree class itself recursively walks the nodes and invokes the callback on each node.

Now let’s say we want this to be an external iterator. Maybe we want to walk two trees in parallel to see if they have the same labels. We can do this something like:

class Tree
  # Mixin all of the enumerable methods to our class.
  include Enumerable
end

a = some tree...
b = another tree...

if a.zip(b).each.all? { |pair| pair[0] == pair[1] }
  puts "Equal!"
end

The zip method takes an enumerable on the left and another on the right and “zips” them together one pair at a time. The result is an array of pairs of elements. If you zip [1, 2, 3] and ['a', 'b', 'c'] together, you get [[1, 'a'], [2, 'b'], [3, 'c']]. Neat.

Then all? walks an array, testing each element using the given block. If the block returns true for every element, all? returns true too. Swell.

But there’s a subtle problem here. The zip method converts its arguments to arrays before doing anything. So we’ve got our nice each method on Tree that generates values incrementally without wasting memory, but then we throw it at zip and it goes ahead and allocates big arrays to store all of these intermediate values. If the two trees we’re comparing are huge, that’s a lot of wasted memory.

What we’d like is a way to walk those two trees iteratively without creating any intermediate arrays. The each method on Tree does that, but it’s an internal iterator. External iterators are perfect for this task. Can we convert it? In Ruby, that’s as easy as:

a = some tree...
b = another tree...

a_enum = a.to_enum
b_enum = b.to_enum

That little to_enum method takes an object that implements each and returns an external iterator. We can use these iterators like so:

loop do
  if a_enum.next != b_enum.next
    puts "Not equal!"
  end
end

The protocol here is that next returns the next item in the sequence. If there are no more items, it raises a StopIteration error (which loop conveniently handles).

Double-plus good! We’ve got nice support for both internal and external iterators, and we can easily convert back and forth between them. It’s like having cake and pie for dessert. The one remaining question is how does this work? Look at what we have here:

The internal iterator (the each method on Tree) recursively calls itself and builds a deep callstack. At any point during that, it can emit values.
The to_enum method takes that and returns an enumerable object. When you call next it runs that recursive code then somehow suspends it whenever a value is generated.
The next time you call next it picks up exactly where it left off. Somehow that entire callstack gets frozen and then thawed between each call to next.

There must be some kind of data structure that represents an entire callstack. It doesn’t reify a strack frame like generators, it reifies the whole stack.

A fiber by any other name

This mystery data structure is what Ruby calls a fiber. Its sort of like a thread in that it represents an in-progress computation. It has a callstack, local variables, etc. Unlike a “real” thread, though, it doesn’t involve the OS, kernel scheduling and all of that other heavyweight stuff.

It’s also cooperatively scheduled instead of pre-emptively. That’s a fancy way of saying fibers have to play nice with other. If you want a fiber to run, you have to give it control. It can’t take it from you.

These constructs have been given as many names as languages that support them. Lua calls them “coroutines” (which is, I think, the oldest name for the idea). Stackless Python calls them “tasklets”. Go’s “goroutines” are similar, though with some interesting differences.

This is the special sauce we need for to_enum. When you call it, it spins up a new fiber. Then it runs the internal iterator on that new fiber. When you call next on the enumerator, it runs that fiber until it generates a value. When it does, it suspends the fiber, returns the value, and runs the main fiber. When we need the next value, it just suspends the main fiber and resumes the spawned one again.

In other words, a simplified implementation of to_enum looks a bit like:

class Object
  def to_enum
    MyEnumerator.new self
  end
end

And the MyEnumerator class (which is simplified from this excellent StackOverflow answer) is:

class MyEnumerator
  include Enumerable

  def initialize(obj)
    @fiber = Fiber.new do  # Spin up a new fiber.
      obj.each do |value|  # Run the internal iterator on it.
        Fiber.yield(value) # When it yields a value, suspend
                           # the fiber and emit the value.
      end
      raise StopIteration  # Then signal that we're done.
    end
  end

  def next
    @fiber.resume          # When the next value is requested,
                           # resume the fiber.
  end
end

Iteration or concurrency?

I started this two-parter with a question about how you can make iteration beautiful and easy to work with. This leads us to wanting both internal and external iteration, and the ability to go from one to the other. The piece needed to really make that work is an easy way to create a new callstack. And fibers are a great answer for that.

But what I find interesting is where we arrived at. We started talking about iteration, one of the most shallow of flow control structures. But look where we ended up. Fibers are a concurrency mechanism. Concurrency is way over in the deep end of language features.

I don’t think this is a coincidence. If you look at iteration, it actually is about concurrency. You’ve got two “threads” of behavior: one that’s generating values and one that’s consuming them. You need to run these two threads together and coordinate them. That is concurrency. We’re just so used to it, that we don’t think of it that way.

Wait, what about Magpie?

I screwed myself over here. The secret agenda of this pair of posts was to trick you into reading about my language by promising to teach you something about other languages you may actually use in real life.

If we’re both lucky you did learn something, but I haven’t gotten to my language yet. Alas, I’m already 5,000 words in and I’ve surely exhausted your patience. I guess Magpie will have to wait for a later post. Trust me, though. It’s awesome. Promise.

Iteration Inside and Out

2013-01-13T00:00:00-08:00

You would think iteration, you know looping over stuff, would be a solved problem in programming languages. Seriously, here’s some FORTRAN code that does a loop and would run on a computer fifty years ago:

do i=1,10
    print i
end do

So when I started designing loops in my little language Magpie, I figured it would be pretty straightforward:

Look at a bunch of other languages.
See what the awesome-est one does.
Do that.

Now, of course, the first wrinkle is that this isn’t just about looping a certain number of times, or through just a range of numbers. That’s baby stuff. Hell, C can do that.

This is about iteration: being able to generate and consume arbitrary sequences of stuff. It’s not just “every item in a list,” it’s “the leaves of a tree,” or “the lines in a file” or “the prime numbers”. So there’s an implied level of abstraction here: you need to be able to define what “iteration” means for your own uses.

What I found kind of surprised me. It turns out there’s two completely separate unrelated styles for doing iteration in languages out in the wild. Gafter and the Gang of Four (also an excellent band name) call these “internal” and “external” iterators, which sounds pretty fancy.

Each of these styles is just beautifully elegant for some use cases, and kitten-punchingly awful for others. They’re like Yin and Yang, or maybe Kid and Play.

External iterators: OOPs, I did it again.

The first side of the coin is external iterators. If you code in C++, Java, C#, Python, PHP, or pretty much any single-dispatch object-oriented language, this is you. Your language gives you some for or foreach statement, like this:

var elements = [1, 2, 3, 4, 5];
for (var i in elements) print(i);

(This is Dart if you were wondering.)

What the compiler sees is a little different. If you squint through the Matrix, then a loop like the above is really:

var elements = [1, 2, 3, 4, 5];
var __iterator = elements.iterator();
while (__iterator.moveNext()) {
  var i = __iterator.current;
  print(i);
}

The .iterator(), .moveNext(), and .current calls are the iteration protocol. If you want to define your own iterable thing, you create a type that supports that protocol. Since a for statement compiles down to that (or “desugars” if you’re hip to PL nerd lingo), supporting that protocol lets your type work seamlessly inside a loop.

In statically typed languages, this “protocol” is actually an explicit interface:

In dynamically-typed languages, it’s more informal, like Python’s iterator protocol.

Beautiful example 1: Finding an item

Here’s a simple example where it works well. Let’s write a function that returns true if a sequence contains a given item and false if it doesn’t. I’ll use Dart again because I think Dart actually works pretty well as an Ur-language that most programmers can grok:

find(Iterable haystack, needle) {
  for (var item in haystack) {
    if (item == needle) return true;
  }

  return false;
}

Dead simple. One key property this has is that it short-circuits: it will stop iterating as soon as it finds the item. This is not just an optimization, but critical when you consider that some sequences (like reading the lines in a file) may have side-effects, or you may have an infinite sequence.

Beautiful example 2: Interleaving two sequences

Let’s do something a bit more complex. Let’s write a function that takes two sequences and returns a sequence that will alternate between items in each sequence. So if you throw [1, 2, 3] and ['a', 'b', 'c'], you’ll get back 1, 'a', 2, 'b', 3, 'c'.

interleave(Iterable a, Iterable b) {
  return new InterleaveIterable(a, b);
}

This just delegates to an object, because you need some type to hang the iterator protocol off of. Here’s that type:

class InterleaveIterable {
  Iterable a;
  Iterable b;
  InterleaveIterable(this.a, this.b);

  Iterator get iterator() {
    return new InterleaveIterator(a.iterator(), b.iterator());
  }
}

OK, again just another bit of delegation. This is because most iterator protocols separate the “thing that can be iterated” from the object representing the current iteration state. The former is not modified by being iterated over, but the latter is. So now let’s get to the real meat:

class InterleaveIterator {
  Iterator a;
  Iterator b;
  InterleaveIterator(this.a, this.b);

  bool moveNext() {
    // Stop if we're done.
    if (!a.moveNext()) return false;

    // Swap them so we'll pull from the other one next time.
    var temp = a;
    a = b;
    b = temp;
    return true;
  }

  get current => a.current;
}

This is a bit verbose, but it’s pretty straightforward. Each time you call moveNext(), it reads from one of the iterators and then swaps them. It stops as soon as either one is done. Pretty groovy.

Kitten-punch example: Walking a tree

Now let’s see the ugly side of this. Let’s say we’ve got a simple binary tree class, like:

class Tree {
  Tree left;
  String label;
  Tree right;
}

Now say we want to print the tree’s labels in-order, meaning we print everything on the left first (recursively), then print the label, then the right. The implementation is as simple as the description:

printTree(Tree tree) {
  if (tree.left != null) printTree(tree.left);
  print(tree.label);
  if (tree.right != null) printTree(tree.right);
}

Later, we realize we need to do other stuff on trees in order. Maybe we need to convert it to JSON, or just count the number of nodes or something. What’d we’d really like is to be able to iterate over the nodes in order and then do whatever we want with each item. So the above function becomes:

printTree(Tree tree) {
  for (var node in tree) {
    print(node.label);
  }
}

For this to work, Tree will have to implement the iterator protocol. What does that look like? It’s best just to swallow the whole bitter pill at once:

class Tree implements Iterable<Tree> {
  Tree left;
  String label;
  Tree right;
  Tree(this.left, this.label, this.right);
  Iterator get iterator => new TreeIterator(this);
}

class IterateState {
  Tree tree;
  int step = 0;
  IterateState(this.tree);
}

class TreeIterator implements Iterator<Tree> {
  var stack = [];
  TreeIterator(Tree tree) {
    stack.add(new IterateState(tree));
  }

  bool moveNext() {
    var hasValue = false;
    while (stack.length > 0 && !hasValue) {
      var state = stack.last;
      switch (state.step) {
      case 0:
        state.step = 1;
        if (state.tree.left != null) {
          stack.add(new IterateState(state.tree.left));
        }
        break;

      case 1:
        state.step = 2;
        current = state.tree;
        hasValue = true;
        break;

      case 2:
        stack.removeLast();
        if (state.tree.right != null) {
          stack.add(new IterateState(state.tree.right));
        }
        break;
      }
    }

    return hasValue;
  }

  Tree current;
}

Sweet Mother of Turing, what the hell happened here? This exact same behavior was a three line recursive function and now it’s a fifty line monstrosity.

I’ll get back to exactly what went wrong here but for now let’s just agree that this is not a beautiful fun way to abstract over an in-order traversal. Now let’s cleanse our palate.

Interal Iterators: Don’t Call Me, I’ll Call You.

Right now, the Rubyists are grinning, the Smalltalkers are furiously waving their hands in the air to get the teacher’s attention and the Lispers are just nodding smugly in the back row (all as usual). Here’s what they know that you may not:

Those languages (Smalltalk, Ruby by way of Smalltalk, and most Lisps) use internal iterators. When you’re iterating you’ve got two chunks of code in play:

The code responsible for generating the series of values.
The code that takes that series of values and does something with it.

With external iterators, (1) is the type implementing the iterator protocol and (2) is the body of the for loop. In that style, (2) is in charge. It decides when to invoke (1) to get the next value and can stop at any time.

Internal iterators reverse that power dynamic. With an internal iterator, the code that generates values decides when to invoke the code that uses that value. For example, here’s how you print the Beatles in Ruby:

beatles = ['George', John', 'Paul', 'Ringo']
beatles.each { |beatle| puts beatle }

That each method on Array is the iterator. Its job is to walk over each item in the array. The { |beatle| puts beatle } is the code we want to run for each item. The curlies define a block in Ruby: a first-class chunk of code you can pass around.

So what this does is bundle up that puts expression into an object and send it to each. The each method can then iterate through each item in the array and call that block of code, passing in the item.

Beautiful example 1: Walking a tree

Let’s see what our ugly external iterator example looks like in Ruby. First, we’ll define the tree:

class Tree
  attr_accessor :left, :label, :right

  def initialize(left, label, right)
    @left = left
    @label = label
    @right = right
  end
end

To walk the tree using an internal iterator style, we’ll want this to magically work:

tree.in_order { |node| puts node.label }

Implementing that iterator in Dart (or Java, or C#) was about 50 lines of code. Here it is in Ruby:

class Tree
  def in_order(&code)
    @left.in_order &code if @left
    code.call(self)
    @right.in_order &code if @right
  end
end

Yup, that’s it. It looks pretty much like the original recursive function, because it is just like that function. The only difference is where that Dart function was hard-coded to just call print(), this one takes a block, basically a callback to invoke with each value. In fact, we can implement the same thing in any language with anonymous functions. Here’s Dart:

inOrder(Tree tree, callback(Tree tree)) {
  if (tree.left != null) inOrder(tree.left);
  callback(tree);
  if (tree.right != null) inOrder(tree.right);
}

You couldn’t do this in Java (…yet), but in most OOP languages you can passably fake internal iterator style. It’s just not idiomatic in those languages.

Internal iteration is definitely beating external style in this tree example. Let’s see how it fares on the others.

Beatiful example 2: Finding an item

OK, let’s say we’re using Ruby and we want to write a method that, given any iterable object, sees if it contains some object. By “any iterable object”, we’ll mean “has an each” method, which is the canonical way to iterate. Something like:

def contains(haystack, needle)
  haystack.each { |item| return true if item == needle }
  false
end

Not bad! So we’re two-for-two on internal style. Let’s transmogrify this into Dart:

contains(Iterable haystack, needle) {
  haystack.forEach((item) {
    if (item == needle) return true;
  });
  return false;
}

Still pretty terse! Except there’s one problem: it doesn’t actually work.

What’s the difference? In both examples, there’s a little chunk of code: return true. The intent of that code is to cause the contains() method to return true. But in the Dart example, that return statement is contained inside a lambda, a little anonymous function:

(item) {
  if (item == needle) return true;
}

So all it does is cause that function to return. So it ends, and returns back to forEach() which then proceeds along its merry way onto the next item. In Ruby, that return doesn’t return from the block that contains it, it returns from the method that contains it. A return will walk up any enclosing blocks, returning from all of them until it hits an honest-to-God method and then make that return.

This feature is called “non-local returns”. Smalltalk has it, as does Ruby. If you want internal iterators, and you want them to be able to terminate early like we do here, you really need non-local returns.

This is a big part of the reason why internal iterators aren’t idiomatic in other languages. It’s really limiting if your each or forEach() function can’t early out easily.

Kitten-punching example: Interleaving two sequences

The other example that worked well with external iterators was interleaving two sequences together. It was a bit verbose, but it worked just fine and could be used with any pair of sequences. Let’s translate that to an internal style. This post is plenty long, so I’ll leave it as an exercise. Go do it real quick and come back.

…

Back so soon? How’d it go? How much time did you waste?

Right. As far as I can tell, you simply can’t solve this problem using internal iterators unless you’re willing to reach for some heavy weaponry like threads or continuations. You’re up a creek sans paddle.

This is, I think, a big reason why most mainstream languages do external iterators. Sure, the tree example was verbose, but at least it was possible. (It’s also probably why languages that do internal iterators also have continuations.)

What’s the problem?

It appears we’re at a stalemate. External iterators rock for some things, internal at others. Why is there no solution that’s great at all of them? The issue boils down to one thing: the callstack.

You probably don’t think about it like this, but the callstack is a data structure. Each stack frame (i.e. a function that you’re currently in) is like an object. The local variables are the fields in that object.

You get another bit of extra data for free too: the current execution pointer. The callstack keeps track of where you are in your function. For example:

lameExample() {
  print("I'm at the top");
  doSomething();
  print("I'm in the middle");
  doSomething();
  print("Dead last like a chump");
}

We kind of take this for granted, but each time doSomething() returns to this lameExample(), it picks up right where it left off. That’s handy. Remember our recursive tree traversal:

printTree(Tree tree) {
  if (tree.left != null) printTree(tree.left);
  print(tree.label);
  if (tree.right != null) printTree(tree.right);
}

After calling printTree() on the left branch, it resumed where it left off, printed the label, and went to the next branch. Once you throw in recursion, you also get the ability to represent a stack of these implicit data structures. The callstack itself (hence the name) will track which parent branches we’re in the middle of traversing.

When we converted that function to an external iterator, that fifty lines of boilerplate was just reifying the data structures the callstack was giving us for free. The IterateState class is exactly what each call frame stored. The tree field in it was the tree parameter in the printTree function. The step field was the execution pointer. The stack in TreeIterator was the callstack.

The lesson here is that stack frames are an amazingly terse way of storing state. You don’t realize how much it’s doing for you until you have to write it all out by hand. If anyone ever asks me what my favorite data structure is, my answer is always: the callstack.

Who owns the callstack?

This is the key we need to see why each iteration style sucks for some things. It’s a question of who gets to control the callstack. Earlier, I said that there are two chunks of code involved in iteration: the code generating the values, and the code doing stuff with them. In an external iterator, your callstack looks like this:

+------------+
| moveNext() |
+------------+
| loop body  |
+------------+
  ...

  main()

The method containing the loop calls moveNext(), which pushes it on top of the stack. It can in turn call whatever it wants, so it temporarily has free reign on the callstack. But it has to return, unwind, and discard all of that state to return to the loop body before it can generate the next value.

That’s why the tree example was so verbose. Since all of that state would be trashed if it was stored in callframes, it had to reify it—stick it in that stack of IterateState objects stored in the iterator object. That way it’s still around the next time moveNext() is called.

With an internal iterator, it’s like this:

+------------------------+
| each                   |
+------------------------+
| method containing loop |
+------------------------+
  ...

  main()

Now the iterator is on top. It can build up whatever stack frames it wants, and then, whenever its convenient, invoke the block:

+------------------------+
| block                  |
+------------------------+
  stuff...
+------------------------+
| each                   |
+------------------------+
| method containing loop |
+------------------------+
  ...

  main()

The block now has to return to each (or whatever each calls). So the iterator can keep whatever state on the callstack it wants, since it’s in control. But, as you can see, you really need non-local returns for this to work well. Because, when the block does want to stop iteration, it needs a way to unwind all the way through stuff... and each all the way back to the method.

That’s the issue. Whoever gets put on top of the stack is in the weaker position, because it has to return all the way to the other one between each generated value. In some use cases, the generator of values needs that power (recursively walking a tree), and internal iterators work great. In others, the consumer of values needs that power (interleaving two iterators) and external ones win.

Since there’s just one callstack, that’s the best we can do. Or is it?

Check out part two to see what some languages have done to try to deal with this.

The Impoliteness of Overriding Methods

2012-12-19T00:00:00-08:00

Over the weekend, I was reading one of the shagadelic papers on Self, Parents are Shared Parts of Objects: Inheritance and Encapsulation in SELF. What can I say, I have a weird idea of fun. If you’re interested in prototypes, or you’re a Javascripter (but I repeat myself) you owe it to yourself to read these papers. They are gems.

But this post isn’t about prototypes, it’s about something the Self folks mention in passing:

In BETA, virtual functions are invoked from least specific to most specific, with the keyword inner being used to invoke the next more specific method. This mechanism is a product of the philosophy in BETA that subclasses should be behavioral extensions to their superclasses and therefore specialize the behavior of their superclasses at well-defined points (i.e. at calls to inner).

It took me a while to tease out what this is saying, but once I did, it was like a dim little light bulb flickered on in my head.

What’s BETA?

Before I get into the lightbulb part, a bit of history. BETA is a language that came out of the “Scandinavian School” in Denmark, the same people that brought you Simula and kicked off the object-oriented revolution. Alan Kay may have coined “object-oriented programming”, but it was Simula that gave him the idea. Chances are, the language you should be coding in right now instead of slacking off reading my blog was directly inspired by these guys.

So after Simula, they went off and made BETA. I think this is more or less equivalent to “famous rock band goes into hiding for ten years and emerges with avant garde free jazz album”. BETA was used as a teaching language, I think, and there were some papers about it, but I don’t know if many people seriously used it in anger.

(Trivia time! Some of the guys who made V8, the famously-fast JavaScript engine in Chrome did use BETA. “V8” got its name because it’s the eighth virtual machine that Lars Bak created. His first VM? A BETA one.)

Part of the reason BETA didn’t flourish may have to do with terminology. Instead of classes and methods, BETA has patterns which subsume both, somehow, and aren’t related to other uses of the term in other languages. I wrote that sentence, and I don’t even know what the hell that means.

The BETA book is a bit… dense. Or maybe it’s just that the syntax is so… weird:

Account:
  (# balance: @integer;
    Deposit:
      (# amount: @integer
      enter amount
      do balance+amount->balance
      exit balance
      #);

    Withdraw:
      (# amount: @integer
      enter amount
      do balance-amount->balance
      exit balance
      #);
  #)

I pride myself on being able to grok syntax on a pretty wide variety of languages but I’m not even sure what’s a comment there. I think if you translated that to JavaScript, it would be something like:

var account = {
  balance: undefined,

  deposit: function(amount) {
    return this.balance += amount;
  },

  withdraw: function(amount) {
    return this.balance -= amount;
  }
};

Like avant garde jazz, this may be genius, but it’s so out there and unapproachable, it’s hard to tell. Fortunately, the Self guys have deciphered some of the mystery and left that little nugget in their paper.

Overriding and `super()`

Let’s cover one last bit of context before I get to the point. If you’re using any OOP language, you’re hopefully familiar with overriding methods. Details vary between languages, but the two main points are:

A subclass can override a method in its superclass. When you invoke the method on an instance of the subclass, the derived method gets called first.
In the body of the overriding method, it can invoke the base class method directly in order to chain the two methods together. In Java, you do so by calling super.someMethod(). In C# it’s base.someMethod(). In CLOS, you use call-next-method. You get the idea.

I’m using class terminology here, but all of the above applies equally well to prototypal languages too, with a couple of names changed.

Here’s an example:

class Account {
  int balance = 0;
  int deposit(int amount) {
    return balance += amount;
  }
}

class CrappyBankAccount extends Account {
  int deposit(int amount) {
    // Service charge, sucker!
    CrappyBank.mainAccount.deposit(2);
    super.deposit(amount - 2);
  }
}

So now, if you do something like this:

Account account = new CrappyBankAccount();
account.deposit(23);

First, it invokes CrappyBankAccount#deposit(). Then, when that calls super.deposit(), it chains to the base Account#deposit() method.

Who’s in Charge Here?

What this means is that the subclass is in control of the dispatch chain. When you override a method in Java, you get to decide if you do stuff before calling super or after. You can change the arguments you pass to it, or even skip calling it entirely.

This is great for flexibility, but as an API designer, that can be frustrating. When I’m making a class that’s designed to be subclassed, I often have constraints that I want my class to ensure.

For example:

class GameObject {
  float x, y;

  void render(Renderer renderer) {
    renderer.setTransform(x, y);
  }
}

class ScaryMonster extends GameObject {
  void render(Renderer renderer) {
    super.render(renderer);
    renderer.drawImage(Images.SCARY_MONSTER);
  }
}

Here we’re making a game with a base class for a character in the world. It provides a default render() method that tells the renderer where to render. The subclass overrides it and draws the specific image that’s appropriate for that character.

There’s an implied requirement here: if you override render() in a subclass, you must call super.render() before you do any drawing. If you don’t, the transform won’t be set and it’ll draw wrong.

These hidden requirements rub me the wrong way. If you’re implementing a subclass of GameObject, how are you supposed to know that you need to do that? You can document it, but it would be better if the base class itself made sure you did the right thing.

`render()` and `onRender()`

To solve this, what I (and lots of other people) do is split these into two methods, like so:

class GameObject {
  float x, y;

  void render(Renderer renderer) {
    renderer.setTransform(x, y);
    onRender(renderer);
  }

  protected abstract void onRender();
}

class ScaryMonster extends GameObject {
  protected void onRender(Renderer renderer) {
    renderer.drawImage(Images.SCARY_MONSTER);
  }
}

Our public render() method is now designed to not be overridden. (In C++ or C# you’d make it non-virtual.) It does the set-up it needs and calls the protected abstract onRender() method. That method is intended to be overridden, and by making it protected and abstract, it’s clear you must override it. Marking it abstract also makes it clear that you don’t need to call super().

This lets the base class stay in control of the dispatch process. It can do set-up before and after the subclass’s “overridden” method gets called. The dispatch order is reversed now. When you call render(), you hit the superclass first and then it calls onRender() in the subclass.

This is almost always how I design classes that I intend to be subclassed. It’s rare that I override methods in my code that aren’t abstract, and I’ve been on teams with style guides that enforced this pattern.

Back to BETA

Of course, the problem with this is that it is a pattern. You have to make a pair of methods, and every time you have another level of subclassing, you need a new name. (If there was a subclass of ScaryMonster that wanted to override onRender() then ScaryMonster would have to add a onOnRender().) This brings us back to BETA.

Overriding methods in BETA works exactly like this pattern, but baked right into the language. Instead of calling super() in the derived class, you call inner() in the base class. That tells it to chain down to the subclass at that point.

When you invoke an overridden method, dispatch starts at the base class (just like we want in the GameObject example) and then walks down to the subclasses at the superclass’s whim. In other words, our example with BETA-style overriding would look like:

class GameObject {
  float x, y;

  void render(Renderer renderer) {
    renderer.setTransform(x, y);
    inner(renderer);
  }
}

class ScaryMonster extends GameObject {
  void render(Renderer renderer) {
    renderer.drawImage(Images.SCARY_MONSTER);
  }
}

If you chain more than two levels of subclasses, BETA scales better because you don’t need to keep coming up with new names. It has some other neat attributes too.

class GameObject {
  float x, y;

  void render(Renderer renderer) {
    renderer.setTransform(x, y);
    inner(renderer);
    renderer.restoreState();
  }
}

class ScaryMonster extends GameObject {
  void render(Renderer renderer) {
    renderer.drawImage(Images.SCARY_MONSTER);
  }
}

Here, we’ve added a call to restoreState() after the call to inner(). By giving control of the dispatch to the base class, it can execute code both before and after the derived class code. super() doesn’t let you do that. (Though super() does let you handle the opposite case: you can put code before and after the base class code in the derived method that calls super().)

It also gives you a convenient way to control which methods are virtual and which aren’t: if a method doesn’t call inner() it implicitly can’t be overridden since it cedes no control to a subclass.

What this means is that base classes have explicit control over how they can be extended. Outside of programming “override” has negative connotations: it means you’re hijacking something without its consent and indeed overriding does kind of work like that in most languages.

In the early days of class-based OOP, people thought any old class could be spontaneously subclassed. You could just override some stuff and it would all magically work out. What we’ve finally realized is that the API you expose to subclasses is another boundary layer that needs to be carefully designed. Ad-hoc subclassing rarely works and classes generally need to be designed up front in order to be subclassed.

BETA was designed around that model. With typical Scandinavian politeness, you don’t override your base class, you politely request permission to extend it. I think right now, the style of a lot of OOP code today fits that model better.

Most languages chose a different path than BETA, but this makes me wonder if Kristen and company had it right all along.

Multimethods, Global Scope, and Monkey-Patching

2012-06-12T00:00:00-07:00

What I mean is that if you really want to understand something, the best way is to try and explain it to someone else. That forces you to sort it out in your mind. And the more slow and dim-witted your pupil, the more you have to break things down into more and more simple ideas. And that’s really the essence of programming. By the time you’ve sorted out a complicated idea into little steps that even a stupid machine can deal with, you’ve learned something about it yourself.

Douglas Adams

I’m interested in all kinds of languages, so I’d read about multimethods and generic functions in Common Lisp, Clojure and Dylan. Even lesser-known languages like Cecil and Nice. But it wasn’t until I implemented them in my own language that I ran into a seemingly innocuous question: how are they scoped?

This question consumed a great number of showers and morning commutes. I implemented a bunch of different versions and watched them break. What I finally stumbled on is really simple, but… wrong-feeling. This post is an explanation of how multimethods are scoped in Magpie, but also sort of a rationalization since the answer turns out to be “globally”.

But I’m getting ahead of myself. First, let me rewind and set up some preliminaries. OK, maybe a lot of preliminaries. This is a corner of language design that I think few people wander into and previous explorers don’t seem have left a map so maybe there will be some value in this.

What Is Scope?

When you hear “scope” you probably think something like this:

{
  int a = 1;
  {
    int a = 2;
    printf("%d", a);
  }
  printf("%d", a);
}

The curly braces in C++ define block scopes. When you declare a local variable, it goes in the nearest enclosing scope. When the compiler sees a use of a variable (here a), it looks in the surrounding scopes to figure out what it’s bound to.

Scopes can nest, like you see here. When a variable is defined in multiple scopes, the innermost one shadows the others and wins. So this program will print 2 followed by 1.

Some languages like JavaScript (before ES6) and C (before C99) don’t have block scope like this. Instead they just have function scope. Each function body can have its own local variables, but there are no nested scopes inside that (unless you actually nest a function in JS). You still have variables outside of functions, so you still have to think about nesting, though.

Both of these kinds of scope have something in common: they use lexical scoping. It’s called that because you can tell what a variable name is bound to just be looking at the text (hence “lexical”) of the program. This is in contrast to dynamic scoping where you can only tell what a name is bound to at runtime. Lexical scoping was one of the big innovations of ALGOL, and almost every language works like it these days.

This is all well and good for scoping variables, but for the sake of this post, I’ll widen the term to talk about any kind of identifier that appears in a program. Variables aren’t the only place names appear in many languages. Consider this bit of JavaScript:

function(who) {
  alert(who.favoriteColor);
}

Here we have two names in the program (ignoring alert), who and favoriteColor. We need to figure out what they represent in order to run the code. who is easy, it’s just a lexically scoped variable and we can see that it’s bound to whatever you pass to the function.

What’s about favoriteColor? Since it comes after a ., that means it’s a property accessor. Each object in JS carries a little bag of named properties around with it. When we do .favoriteColor here, it looks for a property named favoriteColor in that bag at runtime and returns its value. We’ll call this object scope.

Being a prototype-based language, JS is pretty simple here. It only has lexical scope and object scope. Class-based object-oriented languages often have a couple more scopes. Methods will be looked up on the class of the receiver, and there is usually a separate scope for “static” methods.

What Scopes Does Magpie Have?

OK, so we have block scope, object scope, method scope, static scope. There’s lexical scoping and dynamic scoping. Which of these does Magpie have?

Just one: lexical block scope.

Magpie is a class-based object-oriented language, and its syntax is designed to look (more or less) like one. It doesn’t use a dot to separate the “receiver” from the method or property, but it otherwise looks like it would have object, method, and static scope:

list add(item)

When it’s compiling list add(item), the variables list and item are looked up in lexical scope like you expect. But the method add is too. In Magpie, methods are not bound to classes. Instead, they are defined separately. If you know C’s model of “structs + functions”, you have roughly the right idea. If you know CLOS or Dylan, you have exactly the right idea.

When you see list add(item) it looks like you’re calling add “on” some list object, but what it really means is “call this add method, passing in list and item as arguments”. It’s the same as add(list, item) in other languages. Magpie just has a syntax that lets you stick the method name in the middle.

In addition, “property accessors” in Magpie are just methods too. So when you see person favoriteColor, that’s just favoriteColor(person). To compile that, we just look up favoriteColor in lexical scope like you would a variable.

So how far up do these scopes go? In Magpie, each source file is a module and each module has its own top-level scope. There is no single global shared scope. Instead, each module is its own little island that only sees the names that it defines or explicitly imports.

What About Polymorphism?

“Polymorphism” in the object-oriented sense means that the same method can do different things at runtime given different kinds of objects. “Single-dispatch” OOP languages, which are probably what you’re familiar with, achieve this by treating the receiving object (i.e. this) as a scope. By looking up the method on the receiver at runtime, you get behavior that varies based on that object. Magpie on the other hand, just looks up the method in lexical scope.

If methods aren’t looked up on the object (or on its class), how do we get polymorphism in Magpie? How can I make an add() method that does the right thing given a list, a set, or a map?

The answer is that all methods in Magpie are multimethods. The docs have the full story, but here’s the TL;DR: You can define multiple methods with the same name but different argument patterns, like so:

def (list is List) add(item)
    // add stuff to list...
end

def (set is Set) add(item)
    // add stuff to set...
end

def (map is Map) add(item)
    // add stuff to map...
end

In C, this would just be an error because the names collide. In Magpie, it’s A-OK. What this does is a create a single add() multimethod that contains those three methods. When you call add(), it looks at the types of the arguments at runtime and picks the right method to call. Instead of using scoping for polymorphism, it essentially uses pattern matching.

Time For a Snack Break

That probably all sounds a bit hand-wavey. Let’s walk through a concrete example and see how all of the moving parts mesh together. First, let’s make a module that defines a class and some methods for it.

// sandwich.mag
defclass Sandwich
    val meat
    val condiment
end

def (sandwich is Sandwich) isVegetarian
    sandwich meat == "tofu"
end

def (sandwich is Sandwich) toString
    "A tasty " + meat + " and " + condiment + " sandwich"
end

So we have a class Sandwich. This will also automatically give us getter methods for meat and condiment that will return the appropriate fields given a Sandwich instance on the left. In addition, we add another method isVegetarian. It takes a sandwich and returns true if it doesn’t have any meat in it. Finally, we give it a toString method so you can print it and stuff.

Now let’s make another module that uses this one:

// main.mag
import sandwich

val sandwich = Sandwich new(meat: "ham", condiment: "mayo")
print("veggie? " + sandwich isVegetarian)
print(sandwich)

Seems pretty simple, right? It turns out that not binding methods to classes leads to a couple of subtle but deep problems. (At least they were subtle to me. I didn’t realize them until I’d implemented the intepreter and ran straight into them.)

These problems all turn out to be related exactly to the original question of this post, how methods are scoped, and solving them is what led to Magpie’s current (and perhaps surprising) answer.

The First Problem: “Overriding” Methods

So what happens when we run this example? Let’s say for now that methods are scoped just like variables, which is how Magpie used to work. We’ll walk through it a line at a time.

import sandwich

This takes all of the top-level variables and methods in sandwich.mag and binds them to the same names in main.mag. After that Sandwich, meat, condiment, isVegetarian, and toString are all available for use.

val sandwich = Sandwich new(meat: "ham", condiment: "mayo")

This creates a new Sandwich instance and stores it in a variable sandwich.

print("veggie? " + sandwich isVegetarian)

This calls isVegetarian. We’ve imported that method, so there’s no problem here. Now consider the last line:

print(sandwich)

print() is a built-in method that takes an object, converts it to a string by calling toString on it, and then displays it. “Built-in” just means it’s defined in another core module and is automatically imported, so we do indeed have access to it from main.mag. So we call it and pass in the sandwich. What happens next?

What doesn’t happen is that it doesn’t call the toString that we actually defined for Sandwich. That method is scoped to sandwich.mag. We imported it into main.mag so we could call it there. But we aren’t calling toString in main.mag. It’s being called from print, which is in core. It has no idea there’s this other toString method specialized for sandwiches because the toString multimethod core knows about is unrelated to the one sandwich.mag and main.mag have.

Ouch. Our intent in sandwich.mag is that defining toString would work like overriding in other OOP languages. Any place that is calling toString should see that new specialization even if it hasn’t directly imported the module that contains it.

The First Solution: Shared Multimethod Objects

My fix for this was to change what it means to import a multimethod. In our little example, the import graph is like:

core
  ^
  |\
  | \
  | sandwich
  |  ^
  | /
  |/
 main

main.mag imports sandwich.mag and they both also (implicitly) import core. Core itself contains a toString method with specializations for the atomic types like numbers and strings.

The first fix was that when you import a method, you import the exact same multimethod object. So sandwich.mag imports the toString multimethod from core. When it then defines a new (sandwich is Sandwich) toString specialization, that goes into the exact same multimethod that core is using. There is basically a single toString object in memory that all of those modules have a reference to.

This works because core is the first module that created it, and our other two modules are both importing it. So every place that’s using toString has a path of imports that ultimately traces back to the root in core.

The Second Problem: Colliding Imports

Problem solved, right? Well, not so fast. After I did this, I ran into the next issue. Here’s another example. We’ve got these two modules:

// pal.mag
defclass Pal
    val name
end

// pet.mag
defclass Pet
    val name
end

They both define classes that have name fields. That means they both implicitly create name methods. Then we use them:

// main.mag
import Pal
import Pet

def greet(who)
    print("Hi, " + who name + "!")
end

greet(Pal new(name: "Fred"))
greet(Pet new(name: "Rex"))

The intent here is clear: greet() should be able to print anything that has a name getter. What happens if we run this with our current semantics? It turns out we don’t get very far.

The import Pal line works fine. It brings Pal and name into main.mag. Then the second import comes along. It defines Pet fine, but there’s already a name multimethod defined now. We have a name collision.

We didn’t have a name collision in the first example. Even though toString got imported into main.mag both from core and sandwich.mag, they were the exact same object, so we could just safely ignore it. Here, though, that isn’t the case. The import graph is like:

pal  pet
 ^    ^
  \  /
   \/
  main

There is no root module where a single name multimethod is being created. Instead, pal.mag and pet.mag both create their own unrelated name multimethods. When main.mag tries to import them both, they aren’t the same object, so they collide.

The Second (Failed) Solution: Just Deal With It

My first stab at “fixing” this was to just declare those semantics are How It Should Be. If you have colliding method names, you just rename on import, like:

// main.mag
import Pal with
    name as pal.name
end
import Pet with
    name as pet.name
end

That works, but it’s really ugly. I found in practice that method collisions were surprisingly common, almost always coming from fields with simple names like name. Having to qualify those at every use site sucks:

greet(who)
    print("Hi, " + who pal.name + "!")
end

Worse, it breaks duck typing. Using this scheme, there’s no way to create a single greet() method that works on both pets and pals since there isn’t a single name method it can call on both.

The Third (Failed) Solution: Do Something Really Complex

At first, I tried to fix this by doing some hairy method merging when you imported. If you had a method name collision on import, it would create a new local multimethod object that contained all of the specializations of both of the ones you’re importing. That fixes the above problem. main.mag will end up with one name method that can accept both pals and pets.

But it unfixes the first problem. If you import a multimethod (like toString in the first example) and then add a new specialization, you need to push that specialization back up to the modules that you imported.

So to get that working, I made it track where you had all of the modules you had imported a multimethod from. When you defined a new specialization, that would get pushed back up to those modules too.

This did sort of work, but it was complex, felt brittle, and was hard to reason about.

The Fourth Solution: Go Global

I spent a lot of time thinking about it and finally asked if there was a radically simpler answer that would work. One came to mind: make methods globally scoped.

Any time you define a method, in any module, it would just go into a single global pool of multimethods that all modules share. Overriding works, because you’re always overriding: there is only a single multimethod with a given name across the entire program. Duck typing works for the exact same reason.

But global scope is bad, right? I felt like I was committing some cardinal sin by even contemplating this. After much gnashing of teeth, I concocted a rationalization for why this might not be so bad. Here goes…

Consider your favorite conventional OOP language. We’ll do Ruby because it looks nice here:

class Cow
    def speak()
        puts "moo"
    end
end

class Dog
    def speak()
        puts "woof"
    end
end

We have two classes that both define speak methods. We don’t think of speak as being “global” here because you get to the speak methods through an instance of Cow or Dog. Once you do, the method will correctly be associated with the right type and do the right thing. You can’t call Dog’s speak on an instance of Cow or some other unrelated type. Likewise, if you don’t have a dog or a cow, you can’t get to speak at all.

Now consider the Magpie equivalent, and assume that multimethods are globally scoped:

defclass Cow
end

def (is Cow) speak
    print("moo")
end

defclass Dog
end

def (is Dog) speak
    print("woof")
end

How does it compare? Like the Ruby solution, you can’t call Cow’s speak method using a Dog or some other type. If you don’t have a dog or a cow, you can’t get to speak at all.

What this means is that the multimethod is globally scoped. But actual specializations, the stuff you care about, functionally aren’t. Since they are specialized to types, if you don’t have an object of that type, you can’t get to the method. The multimethod is sitting there in global scope where anyone can get to it, but unless you have the key (an object of the right class), you can’t crack it open and get at the methods.

In other words, globally scoped multimethods in Magpie are isomorphic to methods in a single dispatch language. Actually, that’s only half true. Single-dispatch languages only support a subset of what multimethods let you do. Magpie can also express a bunch of stuff that single-dispatch languages can’t.

Better than Monkey-Patching

Multimethods match on all of their arguments, not just the “receiver” on the left-hand side. Let’s go back to our Ruby example. Say we need to serialize Dogs and Cows to a stream. Since classes in Ruby are open for extension, we can do:

class Dog
    def serialize(stream)
        stream write("dog")
    end
end

class Cow
    def serialize(stream)
        stream write("cow")
    end
end

Swell. But then someone else on our team gets tasked with making dogs and cows support serializing to XML. If they aren’t aware of our monkey-patch and add:

class Dog
    def serialize(xmlWriter)
        xmlWriter write("<dog></dog>")
    end
end

class Cow
    def serialize(xmlWriter)
        xmlWriter write("<cow></cow>")
    end
end

Then those serialize methods will obliterate the ones for writing to a stream. Or maybe the other way around. It depends on whose code gets to run last. Nasty.

Meanwhile, in Magpie:

def (dog is Dog) write(stream is Stream)
    stream write("dog")
end

def (cow is Cow) write(stream is Stream)
    stream write("cow")
end

def (dog is Dog) write(writer is XmlWriter)
    writer write("<dog></dog>")
end

def (cow is Cow) write(writer is XmlWriter)
    writer write("<cow></cow>")
end

Here, there is no collision at all. When you call write, it will look at both the left-hand argument (a Cow or a Dog) and the right-hand side (a Stream or an XmlWriter) and pick the one perfect method for those types.

This means you can effectively “monkey-patch” with much finer-grained control and less chance of collision. The entire argument signature forms a key that’s used to select the right method. If you don’t want people to accidentally hit your specific method, just ensure that it requires an argument of a type that’s hard to get a hold of.

Higher Order Macros in C++

2012-01-24T00:00:00-08:00

My hobby project lately has been working on a little bytecode interpreter for Magpie in C++. As an ex-game programmer, I’m pretty sad at how rusty my C++ has gotten. To try to make things a bit easier on myself, I’ve been borrowing from the masters whenever possible. That means I usually have v8’s source code open in another window.

(Aside: if you’re interested in programming languages, it is so awesome that implementations like v8 and spidermonkey are open source and just a click away. Learning from them is a bit like taking your first martial arts lesson from an angry Bruce Lee, but it’s still amazing that industry-leading codebases are just there waiting for your perusal.)

In all honesty, my usual process looks a bit like:

“Hmm, I need to code up a floobinator. v8 has one. Let me see how they do it.”
hunt through v8 code
“Ah, here it is.”
OH GOD, WHAT WIZARDRY IS THIS.

About 90% of it is over my head, but I figure 10% of v8 is still a pretty good chunk of smart. There is one clever technique I learned from them that I do understand: macros that take macros as arguments. That’s the point of this post.

The Problem

C++ is a pretty powerful language for defining abstractions which let you get rid of redundancy. Functions and methods address duplicate chunks of imperative code. Base classes let you reuse data definitions. Templates let you do… well… almost anything.

Even so, there’s still often hunks of repetition that you can’t seem to eliminate. For example, let’s say we’re working with a language’s syntax. Typically, the parser generates an AST which then gets passed to the compiler. The compiler walks the AST using Ye Olde Visitor Patterne and generates some lower-level representation for it.

Depending on how rich your language is, you’ll have quite a few different AST classes to represent the different syntactic elements: literals, unary operators, infix expressions, statements, flow control, definitions, etc. V8, for example, has 40 classes to cover everything you can express in JavaScript.

These are relatively simple types. A (greatly!) simplified one looks a bit like:

class BinaryOpExpr : public Expression {
  BinaryOpExpr(Expression* left, Expression* right)
  : left(left),
    right(right) {}

  virtual void accept(AstVisitor& visitor) {
    visitor.visitBinaryOpExpr(this);
  }

  Expression* left() { return left; }
  Expression* right() { return right; }

private:
  Expression* left;
  Expression* right;
};

Imagine thirty-something-odd more classes like this and you’ve got the right idea. There isn’t too much we can do in C++ to simplify these definitions themselves. Each class is different enough that it’s simplest and clearest to just write them out.

Where the tedium really comes in is all of the surrounding code that uses these classes. First up is the aforementioned visitor. To make it easy for the compiler to dispatch to different code based on the different AST classes, you’ll typically define a class like:

class AstVisitor {
public:
  ~virtual AstVisitor() {}

  virtual void visitBoolLiteral(BoolLiteral* expr) = 0;
  virtual void visitNumLiteral(NumLiteral* expr) = 0;
  virtual void visitStringLiteral(StringLiteral* expr) = 0;
  virtual void visitUnaryOpExpr(UnaryOpExpr* expr) = 0;
  virtual void visitBinaryOpExpr(BinaryOpExpr* expr) = 0;
  virtual void visitAssignmentExpr(AssignmentExpr* expr) = 0;
  virtual void visitConditionalExpr(ConditionalExpr* expr) = 0;
  virtual void visitIfThenStmt(IfThenStmt* expr) = 0;
  // 30 more of these, you get the idea...
};

That code really is just repetitive boilerplate. There’s more. It’s useful to also have an enum for each AST node type so that we can also switch directly on the type of a node without having to go through a visitor for everything. So you’ll want something like:

enum AstType {
  kBoolLiteral,
  kNumLiteral,
  kStringLiteral,
  kUnaryOpExpr,
  kBinaryOpExpr,
  // again, you get the idea...
};

For debugging, it’s handy to be able to get a string representation for an AST node’s type too:

const char* typeString(AstType type) {
  switch (type) {
    case kBoolLiteral:  return "BoolLiteral";
    case kIntLiteral:   return "IntLiteral";
    case kNumLiteral:   return "NumLiteral";
    case kUnaryOpExpr:  return "UnaryOpLiteral";
    case kBinaryOpExpr: return "BinaryOpLiteral";
    // yup...
  }
}

C++’s usual abstraction facilities won’t help us here: in all of these cases the repetition is in the middle of some type definition or statement. C++ is really only designed to let you abstract over entire statements (by making functions) or types (by making templates or base classes).

Let’s Get Dirty

But there is, of course, one grease-covered rusty tool in the C and C++ toolbox that doesn’t give a damn about actual syntactic elements: the preprocessor. It doesn’t even know what a statement is! It just sees chunks of text.

More often than not, that fact makes it too blunt of an instrument to be wielded indiscriminately without risking bloodshed but here it’s just what we need. In all of our problem examples, we want to be able to say “for each AST node type, insert this chunk of code but with the node’s class name inserted in it in a few places.”

Let’s try doing something like this:

#define DEFINE_VISIT(type)  \
    virtual void visit##type(type* expr) = 0

With this, we can simplify our visitor class to:

class AstVisitor {
public:
  ~virtual AstVisitor() {}

  DEFINE_VISIT(BoolLiteral);
  DEFINE_VISIT(NumLiteral);
  DEFINE_VISIT(StringLiteral);
  DEFINE_VISIT(UnaryOpExpr);
  DEFINE_VISIT(BinaryOpExpr);
  DEFINE_VISIT(AssignmentExpr);
  DEFINE_VISIT(ConditionalExpr);
  DEFINE_VISIT(IfThenStmt);
  // 30 more of these, you get the idea...
};

That’s a little better, I guess. But not really. This trick doesn’t help at all with the enum, and only helps a little in typeString(). The problem is that it’s the “for each AST type” part of our problem where the repetition really is. It’s the loop itself we want to abstract over more than the loop body.

What we want is a macro that will itself walk over all of the types, like:

#define AST_NODE_LIST   \
    BoolLiteral         \
    NumLiteral          \
    StringLiteral       \
    UnaryOpExpr         \
    ...

But of course, that one doesn’t do anything useful. We don’t want it to just expand to the type names themselves. It needs to do something with them. But that something is different for each problem area. We need it to take a parameter that is the chunk of code that we generate for each type. Like:

#define AST_NODE_LIST(code) \
    code(BoolLiteral)       \
    code(NumLiteral)        \
    code(StringLiteral)     \
    code(UnaryOpExpr)       \
    ...

Macros Taking Macros… We Must Go Deeper

Now the fun part. When we use that AST_NODE_LIST macro, what is that code argument going to be? It needs to be a thing that’s available at preprocess time, can take an argument, and can generate a chunk of code. That leaves only one answer: a macro.

Until I saw this in V8, I didn’t even know you could pass macros to macros. But indeed you can. Using the AST_NODE_LIST we just defined, our visitor becomes:

class AstVisitor {
public:
  ~virtual AstVisitor() {}

  #define DEFINE_VISIT(type)  \
  virtual void visit##type(type* expr) = 0
  AST_NODE_LIST(DEFINE_VISIT)
  #undef DEFINE_VISIT // Clean it up since we're done with it.
};

When AST_NODE_LIST is expanded, it will expand to one call to DEFINE_VISIT for each of the AST node types. Then those will in turn be expanded to define the visitor method for that type.

Likewise, our enum becomes:

#define DEFINE_ENUM_TYPE(type) k##type,
enum AstType {
  AST_NODE_LIST(DEFINE_ENUM_TYPE)
};
#undef DEFINE_ENUM_TYPE

That’s the whole thing in its entirety. Finally, the function for converting it to a string:

#define DEFINE_TYPE_STRING(type) case k##type: return #type;
const char* typeString(AstType type) {
  switch (type) {
    AST_NODE_LIST(DEFINE_TYPE_STRING)
  }
}
#undef DEFINE_TYPE_STRING

Note that we’re using stringification and token pasting here to not just literally substitute in the AST node class’s name, but also to use it as a string, or to build a larger name (like kBinaryOpExpr) out of it.

This gets rid of some tedious code, but it has another nice side-effect. If you later need to add a new node class, you just add it to AST_NODE_LIST. Once it’s there, every place that’s using that will automatically pick it up. That way you don’t have to remember to touch AstVisitor, AstType and typeString().

I can’t tell if this is crazy awesome, or just plain crazy, but it’s not something you see every day. What other interesting stuff is hiding in the bowels of your favorite open source project?

A Proposal for Null-safety in Dart

2011-10-29T00:00:00-07:00

Page 75 of the current (0.04) draft of the Dart language spec has this note in it:

Should we do something with respect to non-nullable types?

If you asked me I’d answer a resounding yes! (Alas, no one did, but that’s never stopped me before.) So, here’s my attempt at an answer. I’m posting it here on my blog just to clarify that this is my strawman proposal, and not something that’s got any official Dart or Google stamp of approval on it.

The Proposal In a Nutshell

By default, all types are non-nullable. If you declare a variable of type int, it is a static error to try to assign null to it. In checked mode, it is a dynamic error to assign null to a non-nullable variable.

You can define a nullable type by adding ? after the type name. int is non-nullable, int? is nullable. Non-null types are a property of Dart’s type system, but (unlike Maybe or option in other languages) not a part of its runtime behavior.

At runtime, Dart continues to work like a dynamic language where any variable may hold a value of any type, including null. “Nullability” is as optional as the rest of Dart’s type system.

Why Default to Non-nullable?

One of the first decisions a non-null proposal has to make is which is the default: nullable or non-nullable?

I prefer defaulting to non-null for a couple of reasons:

Most of the time, you don’t want to allow null. I went through SwarmViews.dart, one of the largest Dart source files that I’m familiar with and annotated it for nullability. I found 16 nullable variables or fields and 172 non-nullable ones. In other words, defaulting to non-nullability there is right about 90% of the time.
A very large number of type annotations are for atomic types (int, bool, etc.) and people expect those to be non-nullable like they are in other languages. It would be strange to have to go out of your way to say “I want a non-nullable bool”.
In Dart, you don’t have to provide a type annotation at all. It seems to me then that if you’ve chosen to go out of your way to say a variable is of type Foo, it should really be a Foo and not a Foo-or-null.

Semantics

A nullable type is essentially the union of the original type and the Null type. In other words, bool? contains all of the values of the bool type (true and false) as well as all of the values of the Null type (null).

Subtyping

A non-null type is a subtype of its nullable type. In other words, you can always pass a Foo to something that expects a Foo?.

Also, the special Null type is a subtype of every nullable type. This means you can initialize nullable types with null like you’d expect. Since Null is not a subtype of non-nullable types, you cannot initialize those with null. That implies that all variables of non-null types must be initialized. This is an error:

int a;

As is this:

class Point {
  int x, y;
  Point();
}

Requiring fields of non-nullable types to be initialized is a challenge in languages like Java and Scala where you can access a field before it’s been initialized in the constructor. Fortunately, Dart already has constructor initialization lists, and they solve this problem handily. We can fix that Point class like so:

class Point {
  int x, y;
  Point() : x = 0, y = 0;
}

Since the constructor initializers are run before you can access this, we’ve ensured that those fields will be initialized before anyone can see them. Neat.

Working with Nullable Types

One challenge with nullable types is that working with them can be tedious. With things like Maybe in Haskell and option in ML, you have to manually unwrap the “nullable” value to get at the delicious real value hidden inside.

This proposal doesn’t have that problem (granted, it also doesn’t have some of the benefits of option types). It tries to stick close to Dart’s dynamic sensibilities, so there are no option-like values that you have to extract the real value from using pattern matching.

Instead we rely on the fact that a variable can hold a value of any type. A nullable type just means the type checker won’t yell at you if it knows null is one of those values.

To test for null, you can just do a vanilla if (foo == null) statement.

To go back and forth between nullable and non-nullable types, we rely on assignment compatibility. Dart’s assignment compatibility rules are looser than other languages. Like most, they allow implicit upcasts. Given this:

class Base {}
class Derived extends Base {}

Then this is allowed:

Base b = new Derived();

But Dart also allows implicit downcasts (in other words from supertype to subtype):

Derived d = b; // declared as type Base above

(The reasoning here is that many times a downcast will work correctly at runtime, and Dart’s type system is optimistic that you know what you’re doing.)

Thanks to this, nullable types are easy to work with. It is, of course, always safe to go from a non-null type to a nullable one:

int? a = 123;

But it’s equally easy (though not equally safe) to go in the other direction:

int b = a; // declared as int? above

If you actually want to be safe, you just need to check for null first:

if (a != null) {
  int b = a; // safe at runtime now too!
}

So here, as in other places in Dart, the type checker won’t get in your way.

Wait, What Good is It?

You couldn’t ask for a less intrusive null-tracking system, but it seems a little too unintrusive. Isn’t silently assigning from a nullable type to a non-nullable one the exact thing we’d want the type checker to flag?

Well, yes, sort of. But Dart’s specified static checker isn’t that strict in general. It also doesn’t flag implicit downcasts, which this is just a special case of. Even without that, I think we still get a lot of mileage out of this:

Gilad describes Dart as having a “documentary” type system. With nullable and non-nullable types, we can now use type annotations to tell users of our APIs which things accept null and which don’t. In other languages you have no choice but to spell that out in the documentation. With this, it’s right in the type signature for a method. If you call method(Foo a, Foo? b), it’s clear that it can handle a null for the second argument but not the first.
In checked mode, if you try to assign null to a non-nullable type, it will flag that as an error. This means you’ll know when a null snuck in as soon as it happens, and not later on in the program when you accidentally try to call a method on it. That is, I think, a big part of what you want null-checking for.
As I mentioned before, it ensures non-nullable types are initialized. Sure, you can implicitly assign from a nullable type to a non-nullable one, but what you can’t do is implicitly assign from null to a non-nullable type. This is a static error:

int a = null;

That goes a long way towards flushing out uninitialized variable bugs. In
my experience with Dart, that's one of the most common errors I run into.

While the specified type check behavior allows implicit downcasting, tools are encouraged to do their own smarter analysis. If we get nullability annotations in Dart, it will be easy to have tools do data flow analysis and report warnings like this:

double(int? i) {
  return i * 2; // WARN that we didn't test i for null first
}

double(int? i ) {
  if (i == null) return 0;
  return i * 2; // OK now
}

Nullables and Generics

A key question that comes up with nullables is how they play with generic types. I’m not a type system expert, so it’s entirely possible that I haven’t thought this all the way through, but I think it falls out correctly from the subtyping between nullable and non-nullable types.

Generics are covariant in Dart. That means that generics also allow subtyping between a nullable and non-nullable type argument. In other words, this is allowed:

List<int?> list = new List<int>();

As long as you use your types in a way that makes covariance safe (i.e. you read from them and don’t write to them) this will work as well for nullable types as it does with subclassing.

The other question is how using a type parameter in an annotation plays with nullability. I.e.:

class SomeClass<T> {
  T? someField;
}

This is really beyond the edge of my type system fu, but I think the fact that nullable types flatten (A?? becomes A?) will alleviate some of the nasty corner cases this may lead to. I’m not sure though, and this is definitely something I’d like feedback on.

Two Corner Cases

There are two types in Dart where nullability will work a little strangely: Null and Object. The former’s behavior is pretty obviously weird. Given that a nullable type is the union of some type and Null, there’s no dinstinction between Null and Null? (since nullability flattens). Likewise, you can’t have a non-nullable Null type (since no possible values could inhabit it).

The other case is a bit more surprising. You can’t have a non-nullable Object type. Object is the top type which means every object is an instance of Object, including null. In practice, I think this works OK. If you have a variable of type Object, the only operations you can perform on it are ones that all types support, like toString(). Even null supports those, so a non-nullable Object type isn’t needed to avoid type errors.

That covers most of the implications of the proposal as far as I can tell. To get more concrete, let’s see how we’d need to modify the spec and libs to support it:

Spec Changes

I definitely don’t have the skills Gilad has at specifying these semantics precisely, but I’ll at least make an honest try. Most of the changes, as you’ll see, are simplifications. Null shows up as a special case in much of the spec. One of the reasons I like this proposal is that it eliminates most of those.

Section 10.2

Change:

The static type of null is ⊥.

to:

The static type of null is Null.

Section 10.16 Assignment

In both:

In checked mode, it is a dynamic type error if o is not null and the interface induced by the class of o is not a subtype of the actual type (13.8.1) of v.

and:

In checked mode, it is a dynamic type error if o is not null and the interface induced by the class of o is not a subtype of the static type of C.v.

remove “o is not null and”.

Section 10.13.2 Binding Actuals to Formals

From:

In checked mode, it is a dynamic type error if oi is not null and the actual type (13.8.1) of pi is not a supertype of the type of oi,i ∈ 1..m.

remove “oi is not null and”. And from:

In checked mode, it is a dynamic type error if om+j is not null and the actual type (13.8.1) of qj is not a supertype of the type of om+j,j ∈ 1..l.

remove “om+j is not null and”.

Section 11.9 Try

From:

A catch clause catch (T1 p1, T2 p2) s matches an object o if o is null or if the type of o is a subtype of T1.”

remove “o is null or”.

Then we need to add a section:

13.9 Nullable Types

A nullable type is a parameterized type that allows both values of the type parameter’s type or the singleton value of type Null, null.

Let A? be the nullable type of type A. Nullable types do not nest or wrap: B?? is equivalent to B? for any type B. The subtype relations are:

A <: A? (non-nullable is a subtype of a nullable)

Null <: A? (non-nullable is also a subtype of Null)

It is a static error to declare a variable of a non-nullable type without giving it an initializer whose static type is assignment compatible with the variable’s type.

Corelib Changes

This proposal implies some modification to the corelib APIs to be null-aware. There are two kinds of changes we’d need: minor changes to type annotations and a few actual semantic changes.

Type Annotation Changes

The minor change is that a few type annotations need to be tweaked. Operations that are defined to return something “or null” need to be made explicitly nullable.

For example, the Map<K,V> interface has a subscript operator (i.e. map[key]) that returns null if the key isn’t found. Right now it’s return type is declared to be V, the value type. It will need to be V? instead to explicitly permit that null.

Likewise, there are a number of methods that take named optional parameters whose types aren’t nullable, like:

class Expect {
    static void isFalse(actual, [String reason = null])
    // more...
}

Here, that String will need to be made nullable.

Semantic Changes

The more intrusive change is that operations that implicitly create null entries in collections might need to be modified. For example, if you do:

var list = new List<int>(4);

then you create a list whose four elements are automatically set to null. That’s a problem here because the element type (int) is non-nullable.

The safest way to handle this is to change the constructor to let the user explicitly provide the value to initialize the new entries with. So instead of the above, you could do:

var list = new List<int>(4, fill: 1);
print(list); // [1, 1, 1, 1]

or maybe:

var list = new List<int>.generate(4, (index) => index * 2);
print(list); // [0, 2, 4, 6]

I may have missed something, but I believe List is the only type that needs any real API changes like this.

Super Summmary

If you made it this far, you deserve a reward. Sadly, the best I can offer is a bullet list of what you’ve already read:

All types are non-nullable by default.
You define a nullable type by adding ? after the type.
Nested nullable types flatten: A?? -> A?.
A non-nullable type is a subtype of its nullable type.
No nullable values: no option or Maybe.
Can assign back and forth between nullable and non-nullable types.
Must initialize non-nullable variables to non-null values.
Lets types document which things expect null.
Catches invalid nulls early in checked mode.
Most spec changes are just removing “or null”.
Most corelib changes are making some annotations nullable.
Probably want to change List constructor to avoid implicit null fill values.
This proposal is awesome.

Do I have that right? Thoughts?

Wrapping My Head Around Optional Typing

2011-10-21T00:00:00-07:00

One of the really cool parts about being involved with Dart is that I get a lot of first-hand experience with an optionally-typed language. I’ve been fascinated by optional typing for a while, but there are few opportunities to actually try it out on non-trivial code. With Dart, I get to use an optionally-typed language whose lineage goes right back to one of the original wellsprings.

What I found was that it was surprisingly hard to wrap my head around. I initially considered it just sort of halfway between dynamic and static typing, like a 50/50 blend. It turns out, I think, that is more different than that. If there is a line between dynamic languages and static ones, optionally typed ones aren’t on that line. They float off in their own axis like imaginary numbers. In fact, I think they go off in multiple axes.

Before I go into how I think about them, I should probably lay down the basic semantics. Here’s the super science breakdown on how optional types work in Dart. If you want a more, uh, professional treatment, you can also check out Gilad’s less rambling version.

Type Annotations are Optional

At it’s heart, Dart is a dynamically-typed language, so you can code without any annotations:

sum(a, b) {
  var result = a + b;
  return result;
}

Here, you’ve said that a, b, and result can be any type at all and that’s OK. But you can also choose to provide a type annotation in all of the usual places: variable declarations, function parameters, or fields, and return types. So this is valid too:

num sum(num a, num b) {
  var result = a + b;
  return result;
}

Now we’ve stated our intent that a and b be numbers. We’ve also said that sum should return a number. Note that we didn’t annotate result. It can still be anything. Dart lets you mix and match untyped and typed code, so here the result of a a + b is assigned to an untyped variable. Likewise, we get the untyped result and use it as the return value for a function with a typed return.

This mixing and matching is important because it means you can gradually fold types into your code. You can leave them all off while you’re prototyping and then start filling them in as you’ve nailed down the design.

Tools Can Use Them

OK, so you’ve sprinkled some types through your code. Why bother? What do they do? The most basic “feature” that, surprisingly, does add value, is that they help document your code. Other people reading it can see what types you expect variables to be.

The next step up on the scale of usefulness is that, since they’re in the code, tools like IDEs, compilers, and linters are free to do whatever analysis they want using them. Dart does two favors for anyone writing an editor for it:

Types have a built-in declarative syntax. Unlike JavaScript, Python and other dynamic languages, Dart doesn’t use an imperative syntax for defining types. The grammar for classes and interfaces is essentially static. That means an editor can figure out all of the methods a type supports just by parsing a source file. There’s no runtime modifcation or monkey-patching that it needs to worry about.
Variables can have a known type. If you choose to annotate them (or the editor can infer them), then it knows the type of a variable. If it knows the type, then thanks to the previous point, it knows what you can do with it. Ta-da: auto-complete and refactoring are now possible for a dynamic language. It can also do static type-checking like you get in most statically-typed languages.

They Can Be Checked at Runtime

But tooling is gilding the lily. When you’re talking about types, you expect, you know, a type-checker. You have that too (at least in the VM— what types mean when compiled to JS is another interesting story). When you run in checked mode, every type annotation gets checked at runtime. It’s as if every line of code like this:

int i = someFunction();

Turns into (more or less, simplifying things a bit) this:

var _temp = someFunction();
if (_temp is! int) throw 'Type error! Run for your life!';
var i = _temp;

You can think of every type annotation as an expectation: this thing should be a number here. In checked mode, the VM will constantly validate your expectations and stop if something doesn’t hold.

It’s important to note that these checks are done dynamically, at runtime. There isn’t a separate static type checking pass. For example, if you have code like this:

if (2 == 3) {
  // should never get here
  int i = 'not int';
}

You won’t see a type error here because it will never actually get inside the if block. You may be rightly wondering why in the hell you’d want to wait until runtime to find a type error instead of doing it statically. Dart’s take is that you can do both.

Note that we said earlier that tools can do static type-checking if they want. What Dart does is also give you the option to perform those checks at runtime. This is actually how most static languages work too. Very few languages are not unsound, and in the presence of unsoundness, they have to check at runtime. Consider this Java code:

String[] array = new String[5];
Object[] untyped = array;
untyped[2] = 123; // not a string

Here we’re creating an array of strings. Then we assign it to a variable whose type is an array of objects (i.e. anything). Then we try to stuff something that isn’t a string (but is an object) in it.

Update 2011/10/23: I was wrongly using an int array here. Changed it to string[]. I didn't realize only arrays of reference types are covariant in Java.

The static type checker won’t catch this because arrays are covariant in Java. That means that to ensure the last line doesn’t crash your VM, it will do a runtime check every time you set an element in an array to make sure it’s the right type.

There’s another common case where you skirt around the static checker and rely on dynamic type tests: casts.

void callback(Object data) {
  // It's my callback, so I know the data is an int
  int value = (Integer) data;
}

There are times when you know more than the type system does and you just forcibly assert your knowledge. Doing so shouldn’t let you just take down the VM (unlike in C++ where an improper type cast can set your house on fire), so every cast does a runtime check too.

Since Dart lets you mix untyped and typed code, it just embraces this model of validating at runtime more fully. Doing so has a couple of other advantages:

You don’t have to rely on your type system for security. Java tries to rely on the type system and bytecode verification to ensure that code can’t maliciously break the security guarantees of the VM. From what I’ve heard, doing so turned out to be unbelievably complicated.

Dart, on the other hand, can rely on a much simpler runtime model (isolates) to ensure security boundaries.
The type system can be less pessimistic. Static type systems, by their fundamental nature, are pessimistic. Since they don’t know what actual code paths will be run at runtime and which actual types a variable will have, they err on the side of caution. They will report errors for code that may run, or variables that may have the wrong type.

This is good for ensuring real errors don’t get missed, but it’s drag when it reports false positives. Consider:

bool contains(List<Object> collection, Object needle) {
  for (final item in collection) {
    if (item == needle) return true;
  }
  return false;
}

var numbers = <int>[1, 2, 3, 4];
print(numbers.contains(2));

There's a type error here (according to most static type systems). We're
passing a `List<int>` to a function that takes a `List<Object>`, which
relies on [covariance](http://en.wikipedia.org/wiki/Covariance_and_contravariance_%28computer_science%29), but that
isn't statically safe. This `contains` method *could* call `collection.add("not an int")` and that would be an error.

However, it *doesn't actually do that*. It's using the collection in a way
that's perfectly safe with covariance. By loosening the type system, and
relying on dynamic checks to catch *actual* errors at runtime, we can
reduce the number of false positives that the type system chokes on.

They Can be Ignored at Runtime

This is where things get weird. (Actually, if you’re a type system person, they’re already weird because covariant generics are wrong wrong wrong.) I’ve been talking about checked mode, but there’s another mode: production mode. In that mode, the type annotations completely ignored. In other words, it will let you run this:

int i = 'not int';
bool b = 'not a bool either';
num wtf = i + b;
print(wtf); // not intnot a bool either

This probably seems a little odd.

Maybe more than a little odd. In production mode, Dart behaves exactly as if it were a dynamically-typed language. Imagine if you decided to write your JavaScript like this (not that anyone would be crazy enough to do that):

var /* int */ i = 'not int';
var /* bool */ b = 'not a bool either';
var /* num */ wtf = i + b;
print(wtf); // not intnot a bool either

Unsurprisingly, those comments won’t do anything at runtime. That’s how Dart runs in production mode.

How Should You Think of This?

OK, so what do we have? We have a nice little syntax for jamming type annotations into your program. If you use them, then tools can take advantage of that to help you work with your code. Also, in checked mode, the VM will validate them for you. But in production, they are ignored.

That… doesn’t really sound much like a “type system” compared to other languages. In fact, if you try to think of it as a type system, it’s pretty disappointing. It’s more like some type… stuff. Instead of thinking of it in terms of type systems, I tried to find something else I was familiar with from other languages that I could map it to. It finally clicked when someone at work referred to them as type assertions.

Now I get it.

If you’ve done C or C++ programming, you’ve probably used assert() or some flavor of it. If not, it looks like this:

float divide(float num, float denom) {
  assert(denom != 0);
  return num / denom;
}

That assert statement evaluates the condition in it. If it comes up false, then it stops the show and starts ringing the alarm bells. Actually, that’s not entirely true. If you run it in debug mode and the assertion fails, then it halts. Most projects have at least two build configurations.

Debug is what developers use day in and day out. It has extra diagnostic stuff like symbol table information for debugging, and also includes all of the assertions. But there is also usually a release mode. This is what you ship to customers and run in production. In that mode, the assertions are compiled out. They are erased completely.

Sound familiar?

Now why on Earth would you want to disable your asserts in release mode? Isn’t that like wearing your life jacket in the boating classroom and then taking it off when you go out on the water?

It turns out that removing your asserts at runtime actually has a few advantages:

It’s faster. All of those assert conditions have to be executed and checked at runtime. That can add a lot of runtime overhead. When I used to be a game developer, debug builds of games typically ran much slower than release. Playing a videogame running at four frames a second is a strange skill to cultivate.
The app should try its hardest to continue. Once your program is in the customer’s hands, you really don’t want it to crash. It’s possible that some of those assertions are bogus and the app will still actually work if you run past a failed one. Sure, in rehearsal you stop on the first wrong note, but once the audience sits down and the curtains go up, the show must go on.
The user can’t handle a failed assertion anyway. If you were to let the asserts remain in release mode, what do you do when one fails? In debug mode, a failed assertion will do all sorts of helpful stuff like show a stack trace with line numbers, maybe do a heap dump. All that is really helpful… if you’re a programmer on the project.

If you’re just Joe User, that’s utterly useless (and may be a security hazard!) The best the app could hope to do is show an sad face error message and restart. There’s nothing an end user can productively do with the knowledge that a bug in the code itself has manifested.

So what Dart does is apply that reasoning to the type assertions themselves. You can really think of type annotations in Dart as being syntactic sugar for this:

var _temp = someFunction();
assert(_temp is int);
var i = _temp;

As a developer, you can run in checked mode and Dart gives you much of the benefit of a typed language. All those asserts help you enforce API requirements just like they do in C++ or other languages. In fact, you could adopt this style in JavaScript if you really wanted to.

By baking a certain flavor of assert (asserting on type) into the syntax of the language itself, Dart makes it easy for tools to parse those type annotations too and provide more contextual information about your code.

Is It a Type System?

“Type system” carries a lot of implied assumptions and meaning with it. If you take what you know about type systems and look at Dart through that lens, you will be one or more of:

Disappointed
Infuriated
Confused

This doesn’t mean it’s a bad feature, just that it’s not what you think it is. Meatloaf is a pretty terrible dessert, but it’s a fine entrée. If you don’t think about type systems and just ask yourself “is the set of features that Dart provides helpful in writing code?”, I think the answer is “yes”. It’s just not helpful in exactly the same way that type systems in other languages help.

I don’t really think of Dart as having a type system. I think of it as having type requirements. I can use types to define what the APIs of my library expect (preconditions) and promise to deliver (postconditions). At runtime, those assertions will be validated so I can figure out which side is failing to live up to its obligation.

I find that’s a pretty handy tool to have, and it’s really nice to get that without having to give up the simplicity and flexibility of a dynamically typed language.

Semicolons are a Shibboleth

2011-10-12T00:00:00-07:00

For better or worse, anything that Google does attracts a ton of attention. New programming languages also bring out the inner critic in most people. Type systems are prone to enraging the LtU set, especially unsound ones. And doing anything that approaches JavaScript is bound to infuriate a set of people that are passionately devoted to JS and The Way of the Prototype.

So, really, Dart was a perfect storm. It’s been strangely fun watching the complaints, critiques, feature requests and condemnation roll in. One particular issue has incited way more attention than I expected given its significance in the grand scheme of things: semicolons.

Dart requires semicolons as statement terminators just like C, C++, Java, and C# do. (And JavaScript unless you’re ~~brave~~ insane enough to use ASI.) Making them optional is currently the second-most starred bug on the tracker. It’s one of the most frequently discussed points on this insanely long reddit thread.

Whatever your opinion of them, you have to admit they’re a relatively innocuous language feature. Making them optional has very little impact on your code, and there isn’t anything you can express without them that you couldn’t with (and vice versa). What’s the big deal?

Here’s my theory: semicolons are a shibboleth to tell which language camp the designers come from. If you don’t know the term, “shibboleth” comes from this passage in the Old Testament:

Gilead then cut Ephraim off from the fords of the Jordan, and whenever Ephraimite fugitives said, ‘Let me cross,’ the men of Gilead would ask, ‘Are you an Ephraimite?’ If he said, ‘No,’ they then said, ‘Very well, say “Shibboleth” (שבלת).’ If anyone said, “Sibboleth” (סבלת), because he could not pronounce it, then they would seize him and kill him by the fords of the Jordan. Forty-two thousand Ephraimites fell on this occasion.

Judges 12:5-6, NJB

The Two Camps

I hate generalizations, and I particularly hate dichotomies, so I apologize for doing both right here. There is a huge amount of untruth to what I’m about to say, but I hope enough truth for this to be worth your time. Here’s two opposing philosophies for programming languages:

A language is a tool to tame the complexity of the problems we’re solving. It should protect me from my fallibility (and more importantly the fallability of the jack-asses on my team). The more it can help me control things and prevent problems, the better. Good fences make good neighbors.
A language is a tool for expressing my ideas. It should amplify my creativity (and that of the smart people writing the cool libraries I use). The fewer restrictions it gives me, the better. We’re all consenting adults here.

I think you can divide languages pretty effectively into those two camps. The former gets the “serious business” languages: Fortran, C++, Java, C#. The latter gets you the hipster languages: JavaScript, Ruby, Python, Lisp.

Which camp is Dart in? Usually the presence of a static type system alone is enough to toss it into the first camp, but Dart’s type system is so… optional, that it sort of hovers in the uncanny valley between static and dynamic.

If you’re trying to figure out whether you should give a damn about Dart, you’ll want to know if its philosophy lines up with yours. You can look at the language today, but it’s hard to know how much that tells you since the it’s clearly not done yet.

Semicolons

So you ask the shibboleth question: “mandatory semicolons?”

By itself, semicolons mean little to the language, but almost all of the second camp languages make them optional or eschew them completely. They say punctuation is ugly, makes the code less beautiful, and is needless boilerplate since they are almost always at the ends of lines anyway. Requiring them is just making the programmer do mindless gruntwork.

But a “serious business” programmer likes them: they are simple and unambiguous to parse. You don’t have to worry about some other guy’s weird coding style making it impossible for you to understand their code. We’ve been using semicolons for years, why stop now? They’re safe and familiar.

Asking which way Dart is going to with semicolons is, I think, a way of asking which group of programmers the language wants to cater to. Either choice carries an implied signal about how other choices in the language will likely be made: will it favor individual expressiveness over group consistency? Freedom over safety?

So How Does Dart Pronounce It?

So the question is, what will Dart do?

I honestly have no idea. I’m not one of the designers, and I think they may be split on the issue. I have my own personal preference, but that carries about as much weight as yours. Even if they were to make semicolons optional, I don’t think that necessarily says much about your or my other pet features being included.

I think Dart really is trying to inhabit the ground between these two camps, so each attempt to pull in one direction or the other will be handled individually.

But, like everyone else, I hope they decide to go with my preference, which is of course the right one.