Moving away from PHP at eJobs

Business issue faced

At the end of a rewrite of eJobs.ro, Romania’s largest online recruitment marketplace, we needed to migrate the current website data to a new structure without stopping the old service for more than 8 hours.

Situation

The product was written in PHP and most of the developers on the team were experienced in only PHP

We created live data migration services in PHP but once we started testing them we observed very low throughput with migrating individual data items so a decision has been made to batch data items for better performance

In order to batch incoming lists of data items we needed to concatenate smaller batches into bigger ones, an operation which is not supported by the PHP array which is actually implemented as a dictionary and concatenation is actually simulated through merging leading to a O(n*m) complexity.

This merging actually has 2 problems: one of efficiency as each element must be compared with all existing elements and a second problem of data correctness as element keys can be identical and data gets overwritten.

It became clear that we needed to use a different programming language as the migration ran for more than the allowed 8 hours. In fact, it needed around 7-8 days of downtime to migrate the whole data.

Conditions

We wanted to use a mainstream programming language because 3 reasons: 1) we could later recruit new software engineers to work with it; 2) existing developers had career supporting reasons to adopt it; 3) needed libraries already written and battle proofed by the community.

The programming language of choice had to support real lists where concatenation would be an O(1) job.

The programming language had to be suitable for our Kubernetes infrastructure which translates into having small independent binaries and fast restarting processes.

Excellent concurrency support to enable easy parallel processing of data streams was also important for the transition success.

Finally we wanted mainstream support and tooling on Linux which was the main OS for the live environment at eJobs but also wanted good cross-environment support.

Alternatives

Alternatives considered after filtering through all of the above conditions were 2 JVM languages (Java, Kotlin), C++ and Golang.

At this point the plan was to rewrite a piece of a data migration in each language and judge based on the outcome which will better suit us and our problems.

The JVM languages were tried first, starting with Java which had good performance but unpleasantly looking code being quite verbose. Java’s focus on OOP was not very appreciated either. Kotlin was second and it actually improved over Java on the maintainability of the code by having shorter and easier to read code. JVM helped as it was easy to find libraries for data retrieval and processing but also didn’t help because of the size of the final builds which had to include the runtime adding a few hundred megabytes to each image. Both languages produced excellent migration speeds around 20k rows per second which would fit perfectly in our 8 hour window with some time to spare.

C++ port was started but soon we discovered the team wasn’t fond of it and it was canceled before completion. A fault was partially that the resulting code was not so easy to digest, and partially there was a feeling that, careerwise, C++ was not the future.

Golang port happened last but was quite fast and we had minor hurdles in finding the right driver for the database but once we had it running, it was running at around 20k rows per second which was on par with the JVM alternatives.

The code was a lot more simple and thus more maintainable.

Software engineers on the team also wanted to learn it believing that it would be a great advantage to their careers because of the overwhelming use of Golang in the current DevOps tools. The Golang docker images were also in the 2-3 megabytes range.

Choosing Golang

Observing the many advantages compared to PHP but also to the above mentioned JVM languages, my decision was to go ahead with Golang. The first step was to teach my team Golang basics so they could easily participate in Pair Programming sessions where acquisition of the language and its best practices would be the fastest according to my experience.

It took about 6 months to get the backend software engineers to an independent contributor level in Golang.

After completing the data migrations we felt like it would be best to put Golang to good use in our microservices which so far were PHP or Python. One microservice was moved first and we discovered that many things got simpler or better by using Golang including: a lot less type related errors, less hard to understand abstractions, easier GRPC interaction because of native support, clear and more integrated concurrency using goroutines, extensive error checking mandated by the language, accessible testing framework included in the default language tools and so on.

We then took this initiative a step forward and decided that Golang would be our main and default programming language for backend development.

Now, about 2 years after Golang was introduced we are now close to having almost 100% Golang backend code. This led to 3 main outcomes: 1) 10x API response speed 2) higher stability proven by our 99.98% uptime now compared to 99.93% before the language change and 3) better maintainability as most developers can now easily intervene and extend others’ people code. Gofmt is also wonderful at avoiding code formatting inconsistencies and opinion clashes between different software engineers.

LATER EDIT: Four months on I ran a workshop on simplicity as an engineering practice, which is the general form of the argument behind this decision.

LATER EDIT: The career reasoning in here isn’t new. I used the same argument in 2018 when picking an orchestrator, and for the same reason: whatever we standardise on, the team spends years getting good at it, and that skill either travels with them or it doesn’t.