Are you technical? (1)

August 8, 2024

I read this great article about Heroku by someone who used to work there. You may have heard that Heroku had a huge data breach, where all of the access tokens it was storing for Github integration were compromised. This makes me sad because it will probably be the final nail in the Heroku coffin, and Heroku is a lot of the reason that I’m doing the job I do today.

Sometimes people ask me if I’m technical and I don’t really know how to reply to this question. To most software developers I say “not really”, which I judge by the fact that I’ve never written production code for a system before (well, I have once, story for another day) but I have given out a lot of opinions on how that should happen. So I’m roughly trying to say something along those lines — take my words with the caveat that I’ve never done your job.

But in other contexts I’m relatively technical. When my parents were moving out of their house recently I found a “Teach yourself Linux in a week” book from when I was about 16 (complete with distro on CD) where I’d made notes in the margins. I made my first money from computer stuff after teaching myself a HTML/CSS/grid based design and writing wordpress themes. This was in the era of “you can solve any problem using wordpress plugins” (maybe that era hasn’t ended?)
I came into contact with Heroku because I was writing little apps in Ruby. And I was writing apps in Ruby because of Why’s Poignant Guide. This was a book that spoke to me deeply. It was delightfully weird and compelling. I completely did not understand it but I wanted to. So I set out learning to write Ruby code. Probably the only mental models I have of writing code were developed from this book — I still think of all functions in terms of passing variables down a chute and doing things to them along the way.
At the beginning I was just writing little scripts and running them on my laptop to learn the language. But after a while I started to put together the HTML and css I’d learned with the ruby using a delightful little framework called Sinatra (I love that it still has basically the same website as it did in 2009). Who thought I was going to say Rails there? Well I did use Rails as well and it was a lot smaller in those days (I think it was 2.3ish that I started on) but I never really used a database for any of my apps (that was fun, I stored things in arrays and everything got “wiped” when the app restarted) so I didn’t really need the “power” of rails. As I started experimenting with writing apps I started to get confused about how I was going to get the app off of my laptop and onto the internet. In the past I’d always either bought some space from a web hosting company and ftp-d my files into the right location or used wordpress.com and hacked around with it. Now I started digging into setting up a server so that it could run ruby, and including a web server? And somehow hooking that up to the internet? This was a bit of a puzzle. But I could probably work most of this out, I knew my way around a linux box. But what was worse is I had no money. So how was I going to get my hands on a server in the first place?

Enter git push heroku master. It is difficult to explain the magic of this command to me. I mean for a start it made me put everything I’ve ever subsequently done under source control, which was wild power. But the joy of typing the command, hitting enter and then seeing your app live on cuddly-koala-539.herokuapp.com in a few seconds transformed for me what you could do with code. I was writing all sorts of weird silly apps after that. Memorably I wrote an app for a friend’s 30th birthday that asked you what you’d eaten in the last few days and predicted what kind of consistency your poo would be as a result [content warning: there’s pictures at the end of that url].

Beyond the core user experience of making it super simple to deploy an app in pretty much any language with one command, Heroku also made some great breakthroughs in other areas of developer experience. The documentation was superb — until Stripe came along it was known as the gold standard in docs for a developer focused product. The 12 factor principles! Why instant deployment matters! Using containers before docker! The postgres service was so well designed and reliable that it contributed to postgres getting alongside mysql as an open source sql choice. The whole structure of addons, making it simple to extend your app with a db or queue etc was great. There were so many ideas that transformed our thinking about what building software on cloud infra should look like. When I first ran a team building infrastructure I shamelessly used Heroku both in my own learning and as the default example of how we should approach user experience.
Fast forward a bunch of years and I was on stage in New York at Velocity conference 2014 giving a talk about how my team had built “Heroku for government”. It was git push heroku master but the build pipeline included performance tests, compliance checks, making sure you’d set up monitoring and alerting, logs, databases etc. And all the while I pushed my team to give people the experience of that one command. I was chasing the speed and simplicity of that first deploy, sometimes successfully, sometimes less so. Really I still am — there’s a lot more space for that kind of magic in what we do. We’re still selling a variant of the “set up your own server” world, when I suspect at least a subset of our users would be empowered to create amazing things if we had the feedback loop speed and simplicity of git push heroku master.

So if it was so awesome, why wasn’t Heroku more successful? Why aren’t we all running our infrastructure on the platform?

Cost was a major factor. Not so much the actual cost, but the feeling of it. You went from being able to do quite a lot for free to the next tier of payment being close to $100 a year — not a lot by any means but a bit of a jump for the casual user. At the other end of the scale, you had a lot of people complaining that Heroku was expensive because they were comparing with EC2 costing on AWS (“just throw elastic beanstalk on it, it’s basically heroku”). This massively underestimated both the additional cost of managing that EC2 instance (i.e. running a team doing upgrades, patches, connectivity, config management, secrets, etc etc) and the value of making these things smooth from a developer experience, something we’ve always been bad at factoring into our calculations.
I think configurability was the big one though. Heroku’s service was initially optimised around a particular case — running rails with postgres — and it was very very good at this. It did expand out beyond this, in particular with great language support through buildpacks. But once you got beyond running one app and a database, well things got a little complicated. If you were running a lot of apps, there was a bit of a question of how you link them together. But more than that, Heroku worked if you bent your infrastructure to the assumptions of the platform. But anyone bringing their infrastructure to Heroku would find that it wasn’t really the kind of thing you could alter to fit your world. This is wildly different from the “winner” of the cloud infrastructure, AWS, where everything is a low level configurable building block. Even the higher level abstractions expose tonnes of configuration and interoperability. AWS met people where they were, in the total messiness of their current situation. Heroku asked people to mould themselves to a particular way of doing things, with the promise of speed and smoothness. A lot of post-Heroku platform work is about this core trade off, between exposing everything and having to deal with supporting every possible configuration, and exposing a lot less but making your users fall in line with the choices you’ve made. There are ways that we can try to do both but I don’t think as a discipline we’ve worked out what the right path is.
People get weird about infrastructure. We fall in love with it, with every knob and dial that we can turn to make it just so. But the configurability is also our curse. I’m still of the opinion that we are waiting for our iPhone moment — we’ve got the feature phones and you can send an email if you get the exact right incantation of steps but we haven’t reached the level where we know what the user is trying to do, and can give them an awesome way of getting them straight there.

That went in a different direction to what I was thinking. Consider it part 1 in a potentially repeating series of answers to the question “Are you technical?”.