using LLMs in your hiring process
Around here, we’re believers in using a hiring process to give as much info as possible to the candidate. It’s really challenging to get good, credible signal from candidates in traditional 1:1 interviews. Smooth talkers outnumber smooth operators. And it’s easy to believe that all those cognitive biases that beset others don’t dog you, too. Ironically, you might be even more at the mercy of those biases when you’re aware of them.
I’ve found it helpful to use LLMs in hiring processes as one more “voice” and “perspective” that can help you check your own biases and develop a holistic view of candidates. Here are a few places in a process where you can bring them in:
Generating follow-up questions to ask in subsequent interviews.
Sample prompt: Upload the job description and interview notes from a screener or first interview then ask “Based on what you see here, what are 5 questions we should ask this candidate in the next round to give us the most complete understanding of their qualifications and mindset?”
Creating rubrics
This is one of those tasks that can take an hour or more for a person to do but maybe only requires 10 minutes for a person to edit when an LLM has done a rough draft.
Sample prompt: “Here’s the job description and list of interview questions we have for this role. Please create a rubric that will enable us to rate candidates on a scale of 1-7 in five different areas, based on their performance in interviews and a performance task. Please make the rubric a spreadsheet that we can easily copy into the workbook we’re using to track candidates.”
Design and editing of a performance task or work trial
One way to sort smooth talkers from smooth operators is a timed, scenario-based performance task that is then anonymized before grading. Ideally this presents a candidate with some true-to-life situations they have to address in writing.
Sample prompt: “Here’s the job description and some context on where we hope the team and organization are headed in the next year plus. Please generate three prompts for candidate’s to respond to in 90 minutes total that will a) give them a rich sense of what the job is really like and b) gives us a way to evaluate their skills, decision-making, and values”
Grading/reviewing performance tasks and work trials
As with interviews, there’s real value in having more than one perspective on the performance task. I like to have multiple human graders and an LLM grader as well. You can decide if you want the LLM’s scores to actually “count” in the evaluation of the candidate or if you just want them there for reference.
Sample prompt: “Here’s the rubric we’re using and here are the anonymized performance tasks of the candidates. Please grade each task according to the rubric and output a spreadsheet with the results.”
Checking your bias and intuition on things that don’t show up in the rubric
Sample prompt: “I saw a candidate do x during my interview with them and it made me wonder or suspect y. What do you make of this? How should I follow up on this intuition, if at all?”
Nb: Nothing here is legal advice and it’s important that you and your team are clear on the do’s and don’ts in the employment law of your state (especially if your organization is based in California :D
-eric
