We're using maven and tycho to manage builds for the Scala IDE. Our dependencies come both from maven and Eclipse P2 repositories, and http://download.eclipse.org is slow as molasses. Every build loses 4-5 seconds connecting to that repository (even when there's nothing to download). I finally made a local mirror (wget didn't work, so I had to use the official way).

The next step was to make this mirror available to my builds, without modifying the existing poms (it's an open source project, with contributors in several locations). Maven conveniently provides mirrors in a per-user settings.xml file, but for some reason P2 repositories didn't pick it up. It turns out you need to explicitly say it's a P2 repository (thanks to good people on this mail thread):

[xml highlight_lines=12,13]



local-helios
Helios
Local Mirror of Eclipse Helips p2 repository.
http://iulians-imac.local/~dragos/mirrors/helios/
p2
p2


[/xml]

Using Java inner classes in Java

Let's call nested a class defined inside another class that is static, and let's call inner class a non-static class defined inside another class. Java uses the same syntax to refer to both nested and inner classes: import Outer.Inner. We'll consider the following example throughout this post:

[java gutter=false]
package test;

public class Outer {
public int base = 10;
public class Inner {
public int x = 10 + base;
}
}
[/java]

You can use the Inner class either by subclassing Outer, or (less common), by importing and instantiating through an outer instance:

[java gutter=false highlight_lines=5]
import test.Outer.Inner;
import test.Outer;
...
Outer o = new Outer();
Inner i = o.new Inner();
i.x;
[/java]

Java does not distinguish between instance and static members, so the import line looks unsurprising. However, the instantiation syntax is rather weird, making the keyword new look like a member of o, instead of making Inner look like a member of o.

An inner class has access to all members of the enclosing class, final and non-final alike. Therefore, whenever an inner class is instantiated it needs to be passed (implicitly) a reference to the outer instance.

Inner classes in Scala

Scala treats inner classes like any class members. Therefore, selection (both for importing and instantiation) proceeds with an outer value and goes on to select the inner type.

[Scala gutter=false]
val outer = new Outer
val inner = new outer.Inner
[/Scala]

Note that due to type inference, there was no need to specify the type of inner. What is the type of inner?

[Scala gutter=false]
val outer = new Outer
val inner: outer.Inner = new outer.Inner
[/Scala]

We can use the same syntax to import the Inner type in scope: import outer.Inner.

In Scala, types that depend on a value (the object outer in this example) are called path-dependent types. If we had a second instance of Outer, say outer2, the two types would be incompatible:

[Scala gutter=false]
val outer2 = new Outer
val inner2: outer.Inner = new outer2.Inner
[/Scala]
We'd get the following error:

outer.scala:7: error: type mismatch;
 found   : Test.outer2.Inner
 required: Test.outer.Inner

This is different behavior from Java, where type Outer.Inner matches any instance of Inner, regardless of the outer object instance. It turns out that Scala does have an equivalent type: Outer#Inner. The hash operator performs a type selection. Unlike Java, Scala reserves the dot exclusively for selection on terms.

[Scala gutter=false]
var inner: outer.Inner = new outer.Inner
var inner2: Outer#Inner = new outer2.Inner

inner2 = inner // works
inner = inner2 // fails
[/Scala]

And the confusion?

There is still one important difference between the way Scala and Java treat inner classes: mutability. Scala requires paths to be immutable, therefore both instantiation and imports have to mention only vals. The following code doesn't work:

[Scala gutter=false]
var outer = new Outer
val inner = new outer.Inner
[/Scala]

outer.scala:9: error: stable identifier required, but outer found.
var inner = new outer.Inner


You may ask yourself why this restriction? It has to do with type members (remember, Scala classes may have abstract type members). If the path to the inner class contains mutable variables, the type of Inner members may change during execution. For example, a field may change from Int to String. In short, it's unsound.

I believe it's this restriction that confuses most people. So what do you do if you need to have mutable outer instances? First of all, you need to use type selection (Outer#Inner) to refer to such types. Second, you need to instantiate them through an immutable alias, for instance by defining a helper method:
[Scala gutter=false]
var outer1 = new Outer // outer1 is mutable, we need a factory method
val inner1 = mkInner(outer1)

def mkInner(o: Outer): Outer#Inner = new o.Inner // o is immutable
[/Scala]

The Outer#Inner syntax is not allowed in imports (it's not a path).

With this observations, it should be always possible to use Java inner classes from Scala. I hope this post clarifies a (sometimes shady) part of the Java and Scala interoperability.

Here's the result (lower is better)

The first benchmark is the total time for a full build, the following are the bootstrapping steps of the Scala compiler and standard library. For the total time, the laptop is almost 1:30 minutes faster than the desktop!

Here are the two configurations (I ran both builds with a 2.5 GB heap, using Java 1.6/64 bits)

• MacBook Pro: 2.3 GHz quad-core i7 (sandy bridge) with Intel 320 SSD
• iMac: 2.96 GHz quad-core i7 (first generation core i7)

For sure the SSD on my MBP helps a lot, but the processor can't be bad either. I repeated the benchmark by having both computers write to a RAM-disk, and the results were very similar. Of course, the JDK, bootstrap compiler and sources were still on disk (HDD on the iMac, SSD on the laptop), so the laptop still has a signifficant advantage.

While this is certainly a flawed benchmark, and no general conclusions can be drawn, it is nevertheless something I do very often. It's great to see that my new laptop can deliver so well!

The GData Scala Client Library

Google Data (or GData) is an Atom-based publishing protocol that allows CRUD operations over Google services, such as Calendar, YouTube, GMail, etc. It is a very powerful mechanism that allows programatic access to all your Google-based thingies, but being XML-based it is very cumbersome to use directly. Google maintains a bunch of client libraries in different languages that hide all the ugly XML from programmers' eyes.

While at Google, I wrote a Scala client library, and I am still very pleased with the result. I strayed away from their Java client architecture, and for XML handling I wrote a pickler library. However, unlike the original paper that deals with sequential data, I had to make it work with XML trees, which proved a lot more interesting. Briefly, the library allows one to define bi-directional mapping between user types and XML using syntax that's reminiscent of Relax-NG schemas:

[Scala gutter=false]
def pickler: Pickler[Rating] =
elem("rating",
opt(attr("average", doubleVal))
~ attr("min", intVal)
~ attr("max", intVal)
~ opt(attr("numRaters", intVal))
~ default(attr("rel", text), "overall")
~ opt(attr("value", intVal)))(gdNs)
[/Scala]

This code defines an element called 'rating' with an optional attribute average, of type Double, two required attributes of type int (min and max), an attribute with a default string value, and so on. This pickler defines two methods, one from XML to instances of Reminder, and another from Reminder to XML. The library provides type-safe, user-extensible picklers, meaning that the result of unpickling an XML element like

[xml gutter=false]

[/xml]

will be an instance of the Rating class, and serializing an instance of Rating would generate conforming XML. One of the difficulties I faced was to allow permutation of child elements when parsing (the Atom protocol is very liberal in this regard, and Google servers use this liberty). I won't go further into the details of the implementation, as it is nicely described in the Developer's guide.

The other part of the library deals with HTTP requests and binds together the XML serialization code with the network protocol.

Updating birthday entries

A couple of months ago I decided to organize my birthday calendar, and realized that most of the entries had a reminder, but some did not. This resulted in the suboptimal situation that I forgot some birthdays, but not all. I decided to fix this.. programmatically.

The code to do it in Scala is surprisingly simple:

[Scala gutter=false highlight_lines='27,32']
package anivs

import com.google.gdata.calendar._
import com.google.gdata.data.kinds.Reminder
import Schemas._

object ChangeAnivs extends Application {
final val calName = "Birthdays"
val s = new CalendarService("test")
s.setUserCredentials("username@gmail.com", "secret")

val cal = s.getOwnedUserCalendars.find(_.title.content.startsWith(calName)) match {
case Some(bcal) => bcal
case _ =>
println("couldn't find calendar")
exit(1)
}
var link = cal.link(EVENT_FEED)

while (link.isDefined) {
val events = s.getEvents(link.get.href)
println("* got back " + events.length + " events")
addReminders(events)
link = events.link("next")
}

implicit def toOpt[T](s: T): Option[T] = Option(s)

def addReminders(events: s.eventsFeed.Feed) {
for (event <- events if event.reminders.isEmpty) {
// alert 10 min before event
val rem = Reminder(method = "alert", minutes = 10)
event.reminders = List(rem)
val e1 = s.updateEvent(event)
println("Updated event " + e1.title.content)
}
}
}
[/Scala]

That's it! It adds a popup reminder to each event that has no reminders. Notice that the reminder uses named parameters, and the class has default values for most of its parameters (Atom has plenty of optional elements and attributes). I also added an implicit conversion from Any to Option, so that I don't need to wrap Some around the parameters I passed to Reminder.

Next steps

I plan a number of improvements to the code. GData-scala-client was developed before sbt came to the scene, so I'd like to move the build from ant to sbt and publish it to the scala-tools.org maven repository. I already went through most GData-defined types and added default arguments and plan to more consistently use named parameters where possible. This should make the code much easier to understand. And lastly, I might add support for other services (currently, it supports Calendar, YouTube and Contacts) -- I'm a bit reluctant to say this too loud, though..

(I could change the size of the RSS feed in b2evolution's settings).

My previous blog is here.

• asserts should behave exactly like Java asserts
• they should not require any compiler changes (entirely done as library)
• they should be as efficient as in Java

Java asserts

Since Java 1.4, the assert keyword has been added, allowing Java programs to have runtime checks. Assertions can be turned on or off "on-site", as a VM parameter, at the granularity of classes. The nice thing is that they incurr almost no cost at all when disabled: the condition and the optional error message are not evaluated unless assertions are enabled for the enclosing class. This suggests using call-by-name parameters to encode it in Scala.

Scala asserts

The VM hook to determine the assertion status is a new method in java.lang.Class, so here is a first attempt to code this in Scala (normally this method goes in Predef, but to keep samples simple I use another object):

object Prelude {
  def assert(prop: => Boolean, msg: => String): Unit = {
    if (this.getClass().desiredAssertionStatus() && !prop)
      throw new AssertionError(msg)
  }
}

One problem is that the assertion status we get is wrong: it belongs to the Prelude object, instead of the caller's. We can add an implicit parameter to hold the Class of the caller. However, we don't want to have the user write any more code than in Java, so we need to add an implicit value in ScalaObject that is inherited by all Scala classes (for simplicity, we don't change ScalaObject, but create a new trait):

trait Assertable {
  implicit final val $cls = this.getClass()
}

object Prelude {
  def assert(prop: => Boolean, msg: => String)(implicit $cls: Class): Unit = {
    if ($cls.desiredAssertionStatus() && !prop)
      throw new AssertionError(msg)
  }
}

We can test this code using:

object Foo extends Object with Assertable with Benchmark {
  val iters = 100000

  def run = {
    var i = 0
    while (i < iters) {
      methodWithAssert(-1)
      i += 1
    }
  }

  def methodWithAssert(x: Int) {
    Prelude.assert(x > 0, "Negative x")
  }
}

Compiling with -optimise turns the above call to Prelude.assert into very much what javac outputs (like, no closure objects for the call-by-name parameters). The only difference is the call to Class.desiredAssertionStatus, which javac optimizes by storing its result in a final static field, initialized in the static initializer. Unfortunately, our code is 20-30 times slower than the Java version. It seems the native method is really expensive, and we'd better find a way to optimize this call. However, Scala does not have statics the way Java has, and we don't want to touch the compiler. The next best thing we can do is to make it an instance field. We can simply reuse the field used to hold the java.lang.Class:

trait Assertable {
  implicit final val $ea = this.getClass().desiredAssertionStatus()
}

Now the code is as fast as Java code!

Our solution costs one extra word per object, as opposed to one extra word per class in Java (in fact, javac won't generate such fields unless there is at least one call to assert in the enclosed class). This might be a worry... is it worth it? What we gain is less compiler magic, and show that we can extend the language with powerful libraries, without paying much in performance.

trait Automaton {
  /** A type for signals. */
  type Signal

  /** The initial state. */
  val initialState: State

  private var st: State = initialState
  /** The current state of this automaton. */
  def state: State = st

  trait State {
    val name: String
    def react(sig: Signal): State
  }

  type Transitions = Map[Signal, () => State]

  /** React to the given symbol by making a transition. */
  def react(sig: Signal) {
    st = state.react(sig)
  }

  /** An end state that makes no transitions. */
  def finish(nme: String): State = new State {
    val name = nme
    def react(sig: Signal) = this
  }

  /** A state that can have multiple successors. */
  def peekOne(nme: String, ts: Transitions) = new State {
    val name = nme
    def react(sig: Signal) =
      ts.get(sig) match {
        case Some(st) => st()
        case None => this
      }
  }
  implicit def stateToLazy(s: => State): () => State = () => s
}

An automaton has an initial state, and can react to signals by changing state. States have a name and implement the react method to return the next state for a given input signal. Signals are fully abstract. We provide two methods for constructing states, finish creates a an end state, and peekOne which given a map from signals to states, returns a state that makes a transition according to this map. Ignore for the moment the fact that transitions take signals to thunks (functions that take unit and return a state).

Lazy values come in handy when one defines an automaton using this framework:

object Game extends Application {

  val game = new Automaton {
    type Signal = String

    lazy val initialState = start
    lazy val start: State = peekOne("start",
                               immutable.Map(("run",   running),
                                             ("over",  over),
                                             ("reset", start)))
    lazy val over = finish("over")
    lazy val running = peekOne("running",
                               immutable.Map(("over",  over),
                                             ("reset", start),
                                             ("hit",   dead)))
    lazy val dead = peekOne("dead",
                            immutable.Map(("reset", start),
                                          ("end",   over)))
    //...
  }

Defining the automaton in a natural way means some forward references of states, or recursive references, will exist. Without lazy values, the automaton would not be correctly initialized, as some states would be null. Of course, one could order the definitions according to the graph of dependencies, but this is cumbersome.

I get back now to the type of transitions, which uses delayed states:

  type Transitions = Map[Signal, () => State]

The reason is that a cycle in the automaton (like the one formed in state start with transition reset) would lead to an error, unless the recursive reference to the value being defined is delayed. This makes sense, since the value to the state being defined is not needed when constructing the state, but only when the automaton makes a transition. Scala provides a mechanism for such situations: call-by-name parameters. Unfortunately, they are an evaluation strategy, not a type, so we can't use them directly here: we can't define a Map[Signal, => State]. A => type may only appear in the parameter section of a definition, so we need an additional implicit conversion:

  implicit def stateToLazy(s: => State): () => State = () => s

This has the disadvantage that it creates two closures for each entry in the transition map, but the code looks much nicer. If this is too expensive, we can of course prepend () => to each reference to a state in the automaton definition, and change the type of transitions to Map[Signal, State].

A few ideas:

• Could we extend call-by-name parameters to be real types, and therefore allowed everywhere a type is allowed? I see no real showstopper now. This would allow the definition of the transition type above, and save some closure classes.
• Could we add a way to reference but not dereference lazy values? Right now this can be achieved by passing a lazy value to a call-by-name parameter, and then storing it back to a lazy value. Maybe a more traditional delay/force model of laziness makes sense for Scala as well.

You can try the thing by downloading the the modified Ovm and the examples.