Choosing an orchestrator

We’re moving to containers, which means picking something to run them. Here’s the shortlist and why I landed where I did.

The four worth considering

Swarm, the simplest. It comes preinstalled with Docker and you turn it on with docker swarm init. That’s genuinely hard to beat for getting started. It also doesn’t feel ready for what we need yet. Simplistic in the way that stops being charming about six months in.

Kubernetes, the most mature. It’s been in production for about two years at a lot of companies, it comes out of Google, and the configuration is declarative, which matters more than it sounds like it does. The cost is that it’s hard to set up properly for production and there are a great many moving pieces.

Marathon, the apache. It runs on Mesos, which looks like the most thoroughly engineered thing in the category, and it has genuinely fine-grained control over sharing a pool of CPU, memory and disk. It’s also very complex. Training people on it, or hiring people who already know it, would take a long time.

Nomad, the spartan. One binary, deploys plenty of things including Docker, and built for very large deployments. It’s also new and sparse. It does cluster management and scheduling and leaves the rest to you.

The two things that actually decided it

The technical comparison doesn’t separate them as much as you’d hope. All four would work. So the decision came down to two things that aren’t about the software at all.

We don’t have many people with deep DevOps knowledge. That’s the real constraint, and it points at picking something the other companies in the group can also run, so that when something breaks at two in the morning there’s somebody outside my team who recognises the error message. A slightly worse tool that more people around you understand beats a slightly better one you’re alone with.

It should be good for the careers of the people who learn it. I think this is a legitimate thing for me to weigh and I’d rather say so out loud. Whatever we pick, my engineers will spend years getting good at it, and that skill either travels with them or it doesn’t. Choosing something that makes them more employable is not a cost I’m absorbing for their benefit. It’s how you get people who want to be here.

On both counts the answer is the same one, so: Kubernetes.

The part I’m less sure about

Kubernetes has the steepest setup cost of the four and the largest surface area to learn. Kubelet, kubectl, minikube, kops, kubespray, kubeadm, and that’s before any of the concepts. Pods, replica sets, deployments, stateful sets, daemon sets, jobs, services, ingresses, and three different things called volumes that are not what volumes mean in Docker.

That’s a lot of vocabulary to take on, and I don’t want to pretend the decision is free. What makes it worth it is that the vocabulary is the same vocabulary everywhere else, which is precisely the argument above.

LATER EDIT: The career argument turned out to be the durable half of this. Four years later we chose Go over Java, Kotlin and C++ for the backend, and the same reasoning did a lot of the deciding: the team wanted to learn Go, and C++ lost partly because nobody believed it was where their career was going.