When the output is somebody's application

We score candidates against jobs. A recruiter opens a role and sees applicants ranked by how well they fit it, and that score is computed by a model.

It didn’t get built because ranking is an interesting problem. It got built because of arithmetic.

Above some number, the order is the product

A popular role at eJobs.ro collects more applications than any person is going to read. That isn’t a crisis, it’s the ordinary state of a marketplace at national scale, and a recruiter’s day does not get longer to accommodate it.

Below some number of applicants a human reads all of them and the order barely matters. Above it they read the first screen and stop, and at that point the order is the product. Not a feature of the product. The product.

Which means you are ranking whether or not you meant to. Doing nothing is also a ranking, and the one you get by default is usually recency, which is a worse ordering than almost anything you would design on purpose. Nobody decided that applying at 9am should beat applying at 6pm. It just falls out of the schema.

What that makes it

Once you’ve said it out loud, the thing you’re building has a different weight to it. This isn’t a recommender nudging somebody toward another article. A person applied for a job, and a model decided how far down the list they appear. If the score is wrong, somebody who should have had a conversation doesn’t get one, and nobody involved ever finds out. Not the candidate, not the recruiter, not us.

That asymmetry is the uncomfortable part. Bad recommendations on a content site announce themselves, because nobody clicks. A bad ranking here looks exactly like a good one from every angle you can observe.

So how would you know

You can’t evaluate this the way you evaluate search. There’s no click that means correct.

The only real signal is what the recruiter did next. Did they open the application, did they respond, did it go anywhere. So the signal we need is the accept or reject decision on each applicant, fed back as the label the score should have predicted. We don’t capture that yet, and until we do I can’t honestly say how well any of this works.

That’s also the honest reason the labelling work matters as much as the modelling work. A fitness score with no decision feedback is a number that cannot be wrong, because nothing ever contradicts it. That’s not a quality, it’s the absence of a test.

What I’d tell someone building one

Assume you will need it before you think you do. Volume arrives faster than capacity, and the ranking becomes load-bearing quietly.

Decide early what the score means, because a number on a screen acquires authority regardless of what you intended it to convey.

And build the feedback capture at the same time as the model, not afterwards. Retrofitting it is technically easy and organisationally hard, because by then everybody is already relying on the number and nobody wants to hear that you can’t tell whether it works.

LATER EDIT: The pandemic happened, and application counts went through the roof. People lost work, people who hadn’t been looking started looking, and every open role started collecting far more applications than before. The thing we built for ordinary arithmetic turned out to be the thing holding recruiters up, and it helped a great deal. I’d like to claim foresight. What actually happened is that we built something for unremarkable reasons a year earlier and then got very lucky about when it existed.

LATER EDIT: We built the feedback capture. Every applicant on a job can now be labelled accepted or rejected, which finally gives the score the ground truth it should have been predicting all along.