60.0 Hours

Records, it turns out, are not always the kind you want to break.

Back in January 1998, I managed to spend 56.5 continuous hours in an office tracking down a single corrupt print job that was cheerfully destroying a Novell Netware server. It was exhausting, educational, and something I fully intended to never repeat. I filed it away under "formative experiences" alongside "always check the obvious things first" and "print queues are the source of all evil."

Ten years later, I beat that record by three and a half hours.

60.0 hours. Not 59.9, not 60.1. Exactly sixty. Which is either a remarkable coincidence or evidence that my body has an internal timer that fires at multiples of 56.5 and adds a bit for inflation.

Why Hong Kong?

This is the question everyone asked, usually immediately after "are you insane?", and the answer is: physics.

Specifically, the speed of light, which travels at approximately 299,792 kilometres per second through a vacuum and somewhat slower through fibre optic cable, and which does not particularly care about your architectural preferences or your preferred data centre pricing. FiveTen had twenty offices spanning Europe, Asia-Pacific, and North America, and the applications they were running had strict latency requirements. We needed to keep round-trip times to all of those offices within 70 milliseconds. Not 71. Not 80. Seventy.

When you do the maths — and you absolutely have to do the maths, because the speed of light is one of those things that won't negotiate — you pull up a map of the global fibre network and start looking for somewhere that sits at the centre of it. Somewhere that can reach Europe, reach North America, and reach Asia-Pacific without light having to travel so far that it arrives apologetically late. Hong Kong, as it happens, is exactly that place. It sits at a natural convergence point of the major transoceanic cable routes, which makes it the kind of location that network engineers describe as "well-connected" and everyone else describes as "very far away."

Hong Kong was not chosen because it was exciting. It was chosen because when you draw lines from twenty offices on three continents and look for the point that minimises the worst-case latency, Hong Kong keeps winning. I have argued with many things in my career. I have never successfully argued with 299,792 kilometres per second. It simply sits there, being a constant, entirely unmoved by deadline pressure.

The Scope of the Thing

FiveTen had twenty offices across Europe, Asia-Pacific, and North America, each with its own data, its own applications, its own users, and its own entirely reasonable opinion about how important it was relative to the others. The plan was to migrate all of them onto a new shared global infrastructure in the new Hong Kong data centre, simultaneously modernising the platform and consolidating what had been a sprawl of independent systems into something that could actually be managed as a coherent whole.

This is the kind of project that looks clean on a slide deck. The migration plan has colour-coded swim lanes. The timeline has contingency built in. The risk register has mitigations for everything except the mitigations themselves going wrong. Everyone nods. The slide deck is declared excellent. Then you arrive at the data centre and discover that reality does not use the same slide deck software.

Twenty offices. Twenty sets of dependencies. Twenty different people you were trying not to wake up in the middle of the night, spread across time zones in which "the middle of the night" occurred at usefully different times — which was at least one thing that actually worked in our favour.

The Boardroom Becomes a Warroom

Somewhere around hour fourteen, the boardroom officially stopped being a boardroom.

The transformation was not marked by ceremony. There was no plaque unveiling or ribbon cutting. It was simply that at some point a row of cot beds appeared along one wall, a whiteboard covered entirely in migration state diagrams replaced the usual projection screen, and the catering had shifted from the kind of biscuits you get in proper business meetings to the kind of takeaway containers you get when someone has placed a delivery order so large that the restaurant sent a handwritten note of concern along with it.

The IT team were magnificent. I want to be absolutely clear about this, because it is the most important sentence in this entire post. What follows is a story with a good ending, and the reason it has a good ending is not me. The reason is a team of people who stayed in that room, working through the night, and then working through the next day, and then working through the night again, because they understood what we were trying to accomplish and they were committed to accomplishing it.

I was overseeing. They were doing. These are not the same thing, and anyone who has been on either side of that relationship knows the difference.

The Particular Texture of Night Migrations

There is a specific quality to the small hours during a large-scale infrastructure migration that is unlike any other experience in technology. Time dilates in a peculiar way. 3am and 4am become indistinguishable except by the timestamp on the status updates. The delivery food, which was excellent at 7pm when everyone was merely tired, achieves a kind of transcendent meaninglessness by 2am, when you are eating it not because you are hungry but because eating is a thing that humans do and you are still, technically, a human.

Decisions that would take twenty minutes in a normal meeting get made in thirty seconds. Not because the decisions are trivial, but because the context is so thoroughly understood by everyone in the room that language becomes almost unnecessary. A look at the whiteboard, a quick exchange, a nod. Done. Move to the next dependency.

There is also, if you are doing it right, a kind of dark humour that emerges. By hour thirty, the team had developed an elaborate taxonomy of migration states ranging from "nominal" through "interesting" to "we may need more curry." The last category appeared twice. We managed it both times.

What 70ms Actually Means

People outside infrastructure sometimes hear "70 millisecond round-trip time" and think it sounds like a made-up constraint invented by engineers to make their lives interesting. It is not. Seventy milliseconds is the difference between an application that feels responsive and one that feels like it is communicating via particularly sluggish carrier pigeon. Users notice. Users do not notice in a way that they can articulate — they do not say "the round-trip latency is unacceptable" — they say "this is slow" and then they find a workaround, and the workaround is usually worse than the original problem in ways that take years to fully manifest.

The entire 60-hour exercise existed to ensure that nobody ever had to experience that carrier pigeon. Physics defined the location. The location defined the work. The work defined the sixty hours. It is a chain of causality that is, in retrospect, almost elegant.

The End of Hour Sixty

The last business came live at roughly the same moment that the sun was doing something optimistic over Hong Kong harbour, and there was a moment of genuine quiet in the warroom that had been the boardroom, as everyone registered that the whiteboard was clear of open items.

Not "clear" in the sense that everything had gone perfectly. Clear in the sense that everything had been addressed. There is a difference, and it matters. Nothing in a migration of this scale goes perfectly. Things go wrong in exactly the ways they were supposed to, and a few additional ways that were creative and new, and the team handles them, and they get added to the whiteboard, and then they get addressed, and then they come off the whiteboard, and eventually the whiteboard is clear.

That is what sixty hours looks like from the inside.

The Record

The 56.5-hour record stood for ten years, and I am genuinely not sure whether 60.0 hours represents an improvement or a regression. On the one hand, we delivered a shared global infrastructure platform that connected twenty offices across three continents. On the other hand, I appear to have spent two and a half days in a building without going home, which is not a lifestyle that most occupational health guidelines would endorse.

What I know for certain is that I could not have done it alone. This is not false modesty — it is simple arithmetic. One person cannot migrate twenty offices simultaneously. One person cannot maintain the situational awareness across that many parallel workstreams without eventually losing track of something important. The team made this possible. Every migration state that came off the whiteboard came off because someone on that team understood their piece of it well enough to execute it under pressure, in the small hours, in a boardroom that had been converted into a warroom and would shortly be converted back again as if none of this had ever happened.

Somewhere on a server rack in Hong Kong, the infrastructure we built that night is still, in some form, doing something useful. The speed of light continues to be 299,792 kilometres per second. Physics remains entirely unimpressed by any of this.

I, on the other hand, went home and slept for eleven hours.

All Posts