by Malcolm Rowe

Books of 2018

[Insert obligatory “well, it’s been a while since I’ve written anything for this blog” paragraph here.]

With 2018 finally complete, I thought it might be fun to take a quick look at the books I read last year. All of these are from my Goodreads profile, though I tend not to write reviews for individual books there.

Goodreads has a “reading challenge” each year wherein you can set a target number of books to read. In 2016, I hit my target of 34 books, albeit only by cramming both the SRE book and The Calendar of the Roman Republic (long story) on the last day of that year. Buoyed by success, I increased it to 38 books for 2017… and then got distracted by life and fell a bit short.

So, for 2018, I kept the same target as for 2017, and tried to not get distracted. A few weeks ago, I’d got a little bit ahead of that — woohoo me! — and decided it might be fun to put together a short review of each. So here are all the books I read in 2018, in (roughly) chronological order.

To sum up: I managed to read 40 books last year, almost all of which were fiction, mostly urban fantasy and sci-fi, to nobody’s surprise. (I also started and failed to finish a bunch of non-fiction books).

I think I did a better job of picking books with diverse protagonists this time round, and while most of the books I read were published in the last few years (40% were published in 2018), I managed to also seek out a few older ones (Kindred, for example, I’m really glad I got round to reading).

Onward to 2019!

  1. I’d have called it sci-fi purely because it has time-travel, but I ran across an interview with Butler in which she points out, “Kindred is fantasy. I mean literally, it is fantasy. There’s no science in Kindred.” She has a point. 

  2. … though from what I can tell, 6 Ma is squarely in the Miocene epoch, not the Pliocene. In A Pliocene Companion, Word of God resolves this by stating that, in-universe, the Pliocene is considered to start around 11 Ma (not 5.6 or 5.33 Ma, as in our reality). 

  3. And to a large extent, discrimination that’s still present today: there’s a line where our heroine says that “people would ignore what I said until [my husband] repeated it”, which sounds familiar enough. 

This is Just to Say

I have invalidated
the assumptions
that your code
depended upon

Forgive me
they were so well hidden
and so fragile
Reid McKenzie, Twitter

Deadlocks in Java class initialisation

I recently ran across the fact that it’s possible to make the Java runtime deadlock while initialising a class — and that this behaviour is even mandated by the Java Language Specification.

Here’s a Java 7 program that demonstrates the problem:

public class Program {
  public static void main(String args[]) {
    new Thread(new Runnable() {
      @Override public void run() {


  private static class A {
    private static final B b = new B();
    static void initMe() {}

  private static class B {
    private static final A a = new A();
    static void initMe() {}

In addition to demonstrating that lambdas are a good idea (all that boilerplate to start a thread!), this also shows how cycles during class initialisation can lead to a deadlock. Here’s what happens when you run it1:

$ javac Program.java
$ java Program

That is, it hangs.

In Java, classes are loaded at some arbitrary point before use, but are only initialised — running the static {} blocks and static field initialisers — at defined points2.

One of these points is just before a static method is invoked, and so the two calls to A.initMe() and B.initMe() above will both trigger initialisation for the respective classes.

In this case, each class contains a static field that instantiates an instance of the other class. Instantiating the other class requires that that class is initialised, and so what we end up with is that each class’s initialisation is blocked waiting for the initialisation of the other class to complete.

If you trigger a thread dump at this point — by sending a SIGQUIT or hitting Ctrl-\ (or Ctrl-Break on Windows) — then you’ll see something like this:

Full thread dump OpenJDK 64-Bit Server VM (24.79-b02 mixed mode):

"Thread-0" prio=10 tid=0x00007efd50105000 nid=0x51db in Object.wait() [0x00007efd3f168000]
   java.lang.Thread.State: RUNNABLE
        at Program$A.<clinit>(Program.java:13)
        at Program$1.run(Program.java:5)
        at java.lang.Thread.run(Thread.java:745)

"main" prio=10 tid=0x00007efd5000a000 nid=0x51ca in Object.wait() [0x00007efd59d45000]
   java.lang.Thread.State: RUNNABLE
        at Program$B.<clinit>(Program.java:18)
        at Program.main(Program.java:9)


Interestingly, you can see that while both threads are executing an implicit Object.wait(), they’re listed as RUNNABLE rather than WAITING, and there’s no output from the deadlock detector. I suspect that the reason for both of these is that the details of class initialisation changed in Java 7:

In Java 6, the runtime would attempt to lock the monitor owned by each Class instance for the duration of the initialisation, while in Java 7, attempting to initialise a class that’s already being initialised by another thread just requires that that the caller be blocked in some undefined fashion until that initialisation completes.

There are other ways to trigger the same problem, too. Here’s another problematic snippet:

public class Foo {
  public static final Foo EMPTY = new EmptyFoo();

public class EmptyFoo extends Foo {}

Here we have Foo, and EmptyFoo, a special — presumably empty, in some fashion — version of Foo. EmptyFoo is usable directly, but it’s also available as Foo.EMPTY.

The problem here is that initialising EmptyFoo requires us to initialise the superclass, and initialising Foo requires initialisation of EmptyFoo for the static field. This would be fine in one thread, but if two threads attempt to initialise the two classes separately, deadlock results.

Cyclic dependencies between classes have always been problematic in both Java and C#, as references to non-constant static fields in classes that are already being initialised see uninitialised (Java) or default (C#) values. However, normally the initialisation does complete; here, it doesn’t, and here the dependencies are simply between the classes, not between their data members.

Unfortunately, I don’t know of any convenient way to detect these cycles in Java: OpenJDK provides -XX:+TraceClassInitialization, which I suspect might be useful, but it’s only available in debug builds of the OpenJDK JRE3, and I haven’t been able to confirm exactly what it shows.

And for what it’s worth, I’m not aware of a better solution for detecting cycles in C# either. For Noda Time, we used a custom cycle detector for a while; it spotted some bugs resulting from reading default values, but it was too brittle and invasive (it required modifying each class), and so we removed it before 1.0.

I suppose that if we assume that class initialisation occurs atomically and on multiple threads, then this kind of problem is bound to come up4. Perhaps what’s surprising is that these languages do allow the use of partially-initialised classes in the single-threaded case?

If videos are your thing, the folks at Webucator have turned this post into a video as part of their free (registration required) Java Solutions from the Web course. They also offer a series of paid Java Fundamentals classes covering a variety of topics.

  1. Or at least, what happens when I run it, on a multiprocessor Debian machine running OpenJDK 7u79. I don’t think the versions are particularly important — this behaviour seems to be present in all Java versions — though I am a little surprised that I didn’t need to add any additional synchronisation or delays. 

  2. A similar situation exists in C# for classes with static constructors (for classes without, the runtime is allowed much more latitude as to when the type is initialised). 

  3. You can trace class loading with -XX:TraceClassLoadingPreorder and -XX:TraceClassLoading, but this doesn’t tell you when class initialisation happens. 

  4. He says, with a sample size of one. I haven’t managed to confirm what C# does, for example, and C++ avoids this problem by replacing it with a much larger one, the “static initialisation order fiasco”. 

pip install --isolated fails on Python 3.0–3.3

(This is a quick post for search-engine fodder, since I didn’t manage to find anything relevant myself.)

If you’re using pip install --isolated to install Python packages and find that it fails with an error like the following:

Complete output from command python setup.py egg_info:
usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: -c --help [cmd1 cmd2 ...]
   or: -c --help-commands
   or: -c cmd --help

error: option --no-user-cfg not recognized

… then you might have run into an incompatibility between pip and Python versions 3.0–3.3.

pip version 6.0 added an isolated mode (activated by the --isolated flag) that avoids looking at per-user configuration (pip.conf and environment variables).

Running pip in isolated mode also passes the --no-user-cfg flag to Python’s distutils to disable reading the per-user ~/.pydistutils.cfg. But that flag isn’t available in Python versions 3.0–3.3, causing the error above.

I ran into this because I recently migrated the Python code that generates this site to run under Python 3.x. I’m using a virtualenv setup, so once I had everything working under both Python versions, I was reasonably confident that I could switch ‘production’ (i.e. the Compute Engine instance that serves this site) to Python 3 and discard the 2.x-compatibility code.

Good thing I tested it out first, since it didn’t even install.

It turns out that:

  1. virtualenv ships an embedded copy of pip and setuptools, but setuptools will use the system version of distutils1, and
  2. --no-user-cfg was added in Python 2.7, but wasn’t ported to 3.x until 3.42, and
  3. the distribution I’m using on my real server (Debian 7) ships with Python 3.2.3, rather than the 3.4.x I’m using elsewhere.

I worked around this by just omitting the --isolated flag for Python verions [3.0, 3.4) — though since I don’t actually have any system config files in practice, I probably could have set PIP_CONFIG_FILE=/dev/null instead (which has the effect of ignoring all config files).

I’m not the first person to have noticed that virtualenv isn’t actually hermetic. Though some of that rant is out of date now (Python wheel files provide prebuilt binaries), and some isn’t relevant to the way I’m using virtualenv/pip, it’s definitely true that the dependency on the system Python libraries is the main reasons I’d look to something more like Docker or Vagrant for deployment were I doing this professionally.

So did I finally manage to switch to Python 3.x after that? Not even close: Python 3.x didn’t gain the ability to (redundantly) use the u'foo' syntax for Unicode strings until 3.3, and some of my dependencies use that syntax. So I’m waiting until I can switch to Debian 8 on Compute Engine3, at which point I can cleanly assume Python 3.4 or later.

  1. This is a rant for another day, but it looks like virtualenv monkeypatches pip, which monkeypatches setuptools, which either monkeypatches or builds upon distutils. Debugging through this edifice of patched abstractions is… not easy. 

  2. It’s a bit more complex than that: Python 3.0 and 3.1 were released first, then the feature was implemented in both 2.7 and 3.2, but then distutils as a whole was rolled back to its 3.1 state before 3.2 was released. That rollback was reverted for Python 3.4. 

  3. I can apt-get dist-upgrade from the Debian 7 image just fine, but it’s a bit slow and hacky, so I’d rather wait for official images. (I also need to fix some custom mail-related configuration that appears to have broken under Debian 8.)