a biased LLM as a bias check in hiring

Much has been made about potential biases in LLMs - ones that emerge by default from their training content and ones that might be deliberately coded into model weights. I take that critique seriously, and as a result consider the LLM’s take as a perspective - like that of a person, one that is tilted, imperfect, and useful. 

In a recent hiring process, I interviewed a candidate 1:1 who made some strange choices in the conversation. I couldn’t tell if it was sheer nervousness or bad judgment on their part, but their performance landed badly with me. At the very least, the vibes were off. 

If I had simply trusted my gut, gone with my intuition, this would have been the end of the road for this candidate. We had a strong applicant pool and I was ready to screen them out at this early stage of the process. 

But I didn’t feel great about this call. So I typed up a summary of what I saw in the interview and how I interpreted it, then shared that with Claude and asked “what do you make of this?” 

Claude said: note it but don’t make too much of it. If you held aside this weirdness in the conversation, would you otherwise advance this person? For me, that was a yes. They had a record of achievement, had been a team leader in several phases of their career, and offered evidence of a growth mindset in practice. Good enough to move forward.

Interestingly, this candidate ended up making it to the final stage of a very competitive, rigorous process. Through subsequent rounds, my view of this person changed so much that I was ready to offer them the job if our top candidate didn’t accept. In other words, they proved themselves to be fully qualified for the job, despite their uneven interview performance.

I was delighted and shaken by this. It was a good reminder of how fickle a 1:1 interview can be. And of how useful it is to blunt the bias of a person or tool by adding more perspectives rather than striving to purify any one of them.

-eric