Some of my earliest engineering memories smell of diesel and hot metal.
Before I wrote software for a living, I trained as a marine engineer in Jamaica. After that I worked as a power technician, ran gas turbines, and designed parts of a power system. Machines like that teach you one thing very quickly: something is always about to go wrong. A bearing runs hot. A breaker trips. A storm arrives a day early. The job is not to stop things from breaking. Nobody can do that. The job is to know what to do next, and to do it fast.
I didn’t know it then, but that lesson would follow me everywhere.
The same problem, in different clothes
I moved from machines to electronics, then to telecommunications, then to computer science. In Canada I spent about ten years building machine learning systems that had to run every day, at scale, with real people depending on them. The tools changed completely. The problem didn’t.
Pipelines fail at 3 a.m. Data arrives late, or wrong. A model that worked last month quietly stops working. Once again, the interesting question isn’t will it break? It’s what happens when it does?
Since then I’ve noticed the same pattern almost everywhere I look:
- A parent on the school run hits an accident at 8:10 and has to rethink the whole morning.
- A youth basketball program loses its gym the night before a session.
- A small business owner’s plan for the week collapses because one person calls in sick.
- A factory line stops because a machine fails, or because someone has broken into the systems that control it.
Same situation, different size. Murphy’s Law doesn’t care how big you are.
Rerouting for everything else
We already accept that some plans should fix themselves. When there’s an accident on the highway, your phone finds you a new route in seconds, and nobody thinks that’s strange. But for most of the plans that run our lives and our businesses, the schedules, the crews, the production runs, recovery is still manual. Someone gets a phone call, opens a spreadsheet and starts over.
A slow or sloppy replan is a second disruption. Getting back on track quickly and well is a hard problem, and it’s the one I keep coming back to.
It’s a big part of why I started my company, and it runs through my PhD research too, which looks at the security of the operational technology that runs plants and grids: what keeps the lights on, and the line moving, when something goes wrong on purpose.
Why a blog, and why now
I’m building a lot right now, mostly with AI as a working partner, and I’ve learned more in the last few months than in some whole years. Some of it worked. A good amount didn’t. I’d like to write both down.
So this is where I’ll think out loud about:
- engineering, from turbines to software
- building real products with AI, honestly, including what goes wrong
- recovery: how systems, plans and people get back on track
- lessons from a long and non-linear road
If any of that interests you, there’s an RSS feed. Thanks for reading the first one.