Ultimate guide to doing reference checks

2 October 2020

Introduction
Reference checks are amongst the highest leverage activities when it comes to making an informed decision about whether to hire a candidate.

It’s been a year or so since we at TopHire came across a podcast by Tarun Davda, an MD at Matrix Partners. That served a solid starting point for us. The rest of this post hopefully complements and adds to his podcast well.

In the last 4 years, we've done reference checks for the majority of the non-freshers we've hired. I've personally conducted probably 80+ reference check calls. And Sameer Parwani my co-founder would have done several hundreds over the last 10+ years he's been managing companies.

Over time, I’ve developed a better sense of what to look out for, how to direct the conversation, what I take away from the reference call - so thought I'd share some of my experiences here.

Terminology
1. Hiring Manager - Person conducting the reference check on the candidate
2. Candidate - Candidate who you are trying to do a reference check for
3. Reference Contact - The person that you reach out to, whether it's a previous manager, subordinate or a peer.

Why is it important to do reference checks?
No matter how many interviews you do, you won’t able to accurately predict a candidate's on the job performance. A candidate who is good at interviewing will know what interview questions to prepare for and will be able to put forward the absolute best version of themselves while speaking to the hiring managers. And in assignments they might seek help from friends or seniors.

A reference check with someone who has spent years day in & day out with them is a high quality signal as to how they will be when working with you on your team.

Thumb Rules for how many reference checks to do

Skip if fresher. Not much to be gained from doing reference checks on college professors or peers. Professors especially are bound to give positive reviews because they want their students to get placed
Junior Position - 1-2 reference checks. With previous manager, and maybe 1 peer
Mid level Manager - 2-3 reference checks. With previous manager(s) and subordinate(s)
Senior and above manager - 3-5+ reference checks. Or as many until you are confident that you’ve milked all opportunities to get to know more about the candidate from the horse’s mouth. This means past managers, CEO/Founders of prev’s companies, subordinates they’ve manager. Even other functional team heads who may have worked with them. The cost of getting a senior manager hire wrong is especially high.
Executive - Upto 10 depending on seniority. Peers, other VPs, subordinates

Another way to do think about it is that keep doing ref checks until the feedback starts getting repetitive. Which means at least 3 people have shared a very similar set of +s and -s.

Reaching out to Reference Contacts
During the interview rounds with the candidates, I tend to directly ask them who their prev. manager is or to tell me about their peers and subordinates. I don’t tell them that i’m planning to reach out to them though. I just take a note of the name.

Somewhat off-topic but a useful question to ask candidates when interviewing them is what they believe are the strength / areas of improvements of their managers/subordinates/peers. In the process of the candidate answering this question, we'll know their names and be able to reach out to them.

If you don't know their names, reach out to the head of the relevant team on linkedin and work your way from there.

If you don’t manage to get a response on linkedin, try DM-ing them on twitter or use a tool like Snov.io, SalesQL, Hunter.io, RocketReach or Lusha to grab their email addresses.

Always push for a phone call or a video call. Reference contacts would hesitate sharing negative feedback about candidates in written communication. They’re far more likely to be open and candid over a phone call. As an added benefit, you get to build rapport much better over a phone call or a video call.

Do the prep work
We maintain detailed notes on every interview round we’ve had with the candidate. I gloss over them and pick up any points that I’d want to touch on with the Reference Contact.

Plus it obviously helps to have at the top of your mind the teams the candidate has worked with or the projects he’s been involved in so that you can ask specific questions. For e.g. a specific questions could be “Tanisha spent close to a year working with the CorpDev team as well while you were managing her. If I wanted to do another Reference Check with someone in CorpDev, who’d be a good person?” OR “I noticed she did some work with the CorpDev team while she was also working in your Operations team, did you feel she managed to juggle work at both teams well?”

I try not to surprise them with an unscheduled call
Unscheduled calls are quite common in India when it comes to a lot of other things. I’d much rather schedule a call so that the Reference contact is prepared for the call and so that they are in a conducive environment where they speak freely.

While I am on the call
I start with an introduction about myself and what we do as a company. I then go on to share some details about the candidate, where they are in the interview process with us and the details of the role we’re looking at them for.

Before they begin speaking, I reiterate how much importance we place on confidentiality. And that even if it came to a point where we wanted to not go ahead with making an offer to a candidate based on the feedback I’m about to receive from them, that I’d be extra careful and go out of my way to make sure that the candidate doesn't feel that the reference contact jeopardized their interview process with us.

I also emphasize on how i’m speaking to 5 other people apart from them so that they feel “safety in numbers”.

Be vulnerable
Tell them that how important it is for you to find the right person for the role and how are you worried about making the wrong decision. And that a lot hinges of their feedback and their candidness. Helps them open up too.

Questions to ask the Reference Contact
I’ve compiled here[scroll down to appendix] a list of questions we like asking. The compilation of questions serve as a good starting point.

But as Tarun Davda points out in his podcast, the best ones are questions that naturally come up when you go with the flow of the conversation and dig into specific threads.

Pick up on even the smallest of red flags and ask them to elaborate on it
A standard question to ask is “what areas did you wish they’d improved on”. Most ref checks always have substantial points to say here but by default they sugarcoat (or avoid altogether) it to a point where you need to be sensitive to the smallest of red flags they point out.

For example we’ve had a Reference Contact tell me that “well this person tends to get moody at times but yeah it’s fine” and tried to brush it aside and move on to the next point. When it was my turn to speak, i asked if we could rewind and share an instance where they got “moody” and asked about how often it happens. And kept the conversation going by asking questions like how does the “moodiness” manifest itself and what happens after they get moody. Do they become rude? Do they lose productivity for days on end? Do they get moody when you try giving them pointed feedback? As I started digging deeper, it became increasingly clear that the candidate we were interviewing, may have soft skills issues which would have been hard for us to surface in the interview rounds we had with them.

The above pattern has repeated itself several times. 1) Pick up on a subtle red flag they are hinting at. 2) Ask the Reference Contact to share an example incident so i’d get a better idea, 3) Ask more questions around it to get a good picture of how severe this red flag is.

Put another way, you start with broad questions, then keep narrowing down into specific threads as they point them out.

It’s hard for them to lie when you ask for specifics when they gave some positive endorsement
If a reference person is giving high level, unhelpful positive statements about a candidate. Ask them to share a specific example that made them think that way. If their feedback was authentic they should be able to find some specific incident or the other. If they aren’t being authentic, they might get agitated easily when you dig on specific threads. And agitation itself is a bad signal for me that the ref check isn’t enthusiastic enough about the candidate to try to make him look good

Did the Reference Contact seem like someone who is Competent
At times the person you are speaking to may come across as someone who is incompetent themselves. An incompetent person will have a low standard for the people around them. I tend to disregard the positive feedback if i feel this way about someone and try to figure out who else i can speak to, to get deeper insights.

Did the Reference Contact seem like someone who is Authentic
When I think of the most insightful reference checks I've done, they are ones where the Reference Contact kept their feedback balanced. In other words, they were able to objectively point out what is good about the candidate and also what they needed to work on.

What kind of a relationship the Reference Contact and Candidate share?
Does the reference contact seem to go out of their way to make the candidate look good? Then we have a bias problem.

The average candidate has a tendency to exaggerate their achievements
This has been tricky for me because i’m not sure what to make of it at times.

On a handful of occasions, the candidates we speak to over emphasize the role they had to play in a certain project they were involved in. From one angle, it's natural to want to slightly mislead an interviewer into thinking you had a bigger role than you actually did. Or sometimes the candidate has a distorted view of the role they had to play in driving / executing / owning a project.

If i’m skeptical of certain claims, I tend to bring up the project and enquire with the Reference contact what the candidate’s involvement was like. In case there is a huge disparity with the candidate’s claims and what the Reference Contact says, then I know it’s worth digging more.

Gather as much key information as you can in the first 20 minutes of the call
It’s only the first 15-20 mins or so where the Reference Contact will cheerfully offer up insights into what it’s like working with the Candidate in question. And you’ll slowly start seeing the enthusiasm levels dip, signalling that they are fatigued and want the call to end soon.

Takeaway here being you want to ask all the high value/impact questions as early as you can into the conversation while they still feel obliged to offer insights.

Write down detailed call notes for yourself immediately after you are done
The average reference check call I get on lasts 25 minutes. The range would be 15 minutes for a conversation that isn’t flowing well and upto 40 minutes for calls that are flowing well.

I share detailed notes with only Sameer Parwani(my cofounder) on specific things that were brought up the Reference Contact, the way in which they said it(did they seem uninterested during the entire conversation, did i have to really push them to share specific negative instances, etc.), my commentary and takeaways from the conversation and so on.

These calls notes become really useful over time because it serves as a solid feedback loop after you end up hiring the person. At times we look at past reference call notes and think to ourselves “the reference correctly identified the that they are strong in these aspects”.

Or in some rare cases we may look back at our notes and think to ourselves “the ref check did try to subtly hint that this may be a weakness but we didn’t pay attention at that point”

A hugely negative reference check may need to be taken with a pinch of salt.
A very negative reference is rare, but we’ve had a couple of them in the past. Sometimes ref checks have an agenda and want to jeopardize the chances of the candidate for a reason that's driven by resent. So as soon I believe they are sour about something, I begin to question the authenticity of their feedback.

There is one person we hired despite their reference check sharing terrible feedback about the person that ended up eventually joining us. And in hindsight we made the right call by disregarding the feedback that the Ref Contact gave. Because they were disgruntled about something and failed to be objective.

Don't reach out to their current employers unless you have their permission
It's unethical for obvious reasons.

Reaching out to the current manager is acceptable only if the candidate confirm that they are fine with it.

Bonus
Ask the Reference Contact to compare the Candidate to the candidate’s peers. When they talk about the best people, take note of any details they mention about them. For example, they could say something like “this lady who has been with us for 3.5 years is no.1, and {Candidate} would be 3rd after her”. Thats someone you could try to headhunt separately by searching for them on linkedin

Appendix

General questions to ask for all roles:

Is {CandidateName} someone you worked closely with? [This is a good question to start the conversation off]
What did you think about her work ethics?
How come she left? Did you try to retain her?
Would you say she is "not too reliable", "somewhat reliable" or "very reliable"
Do you think she sometimes requires a little bit of a push to get things done?
Do you think she is someone who would go above what is expected of her?
What would you say this person has to work on to become really effective?
Can you specifically tell me about a time when {{CandidateName}} disappointed you or didn’t meet your expectations.
What are the first few keywords that come to your mind when you think about {{CandidateName}}
Was she someone who gave her everything to the company? In other words was she someone who believed work / life balance was important.
If I call you in a year and tell you that candidate X didn't work out, what is your best guess as to why?
Is {{CandidateName}} the type of person who would take up additional responsibilities / the type who would go out of her way to do things even if it's not in her purview or KRA?
Of the people in your team, would you rate them as the top 10%, top 20% or what bracket would you put them under?
Rate this person out of 10. But you can't use 7. [This forces the Ref Contact to not play it safe]
How would you describe {{CandidateName}} personality?
Would you want to work with her again?
Do you think {{CandidateName}} handles pressure well?
Do you have anyone else that you think I should talk to, to understand more about {{CandidateName}}? Someone who has worked closely with them. I will reach out to them discreetly on linkedin.
If I were going to hire a partner for this person to complement their strengths and weaknesses, what would that person be good at?

Jason Fried's questions from here:

What's something that would surprise us about them?
Specifically, any areas where you were surprised they weren't as good as you expected with A, B, or C? Or much better than expected with D, E, and F?
What's the difference between how they interview and how they deliver on the job?
Is there a difference between how a boss, a peer, or a direct report would describe them? If so, what's the difference?
If you were at another company, would you absolutely hire this person again for a similar role?
Who do they naturally gravitate to inside an organization? Or naturally avoid?
What are they better at than they think, and, on the flip side, worse at than they think?
What sort of things do they do that often go unnoticed or are under-appreciated?
What don't they get enough credit for?
Can you tell me about the kind of people they've hired?
Do they leave disagreements on good terms?
Are they more curious or critical about what they don't understand?
What's the one thing nearly everyone would say about them?
What kind of company feels like a natural fit? And which kind would be a challenge?
Can you describe a time when they changed their mind? From what to what, and what caused the change?
What's the best thing about working with them? And the hardest?
If you could change something about them, what would it be?
Are they better working with what they have, or working with what they want?
When have you seen them get in over their head? And how did that turn out?
Have you seen them get better at something? Worse?
Do they make other people better? How?
Are they better at taking credit or giving credit?
Are they more likely to adjust to something, or try to adjust the thing?
Primary blindspot? And bright spot?
As well as you know this person, what do you think their secret career ambition is?
If they hadn't been at your company, how would your company have been different?
Can you remember a time you wished you had their advice on a decision, but you didn't?
Have they ever changed your mind?
What's the easiest thing for them to communicate? And the hardest?
How have they changed during the time you knew them?
Do you still keep in touch even though you don't work together anymore?
What do they need to be successful?
Why do you think we'd be a better company with them on board?
Who else should I talk to that would have something to say about them?

SWE specific questions:

Have you every made him re-write code that he has written because you thought it was not up to the mark. if yes then what was it and why you made him do it?*
What are his debugging skills like? Does he go deep into figuring out what’s wrong? Or does he give up after a certain point and call for help from you.
Conflict resolutions. I asked him, sometimes engineers like to do certain things in a certain “interesting way”. Like “hey lets use a new programming language to code this microservice.” and you have to tell them "no that's a bad idea". How would {candidate} deal with that kind of stuff? Does he take it well or does he resent.
Is {candidate} an engineer with a good product sense? Does he think about how to solve the company's users' problem himself. Or is he a "Here's the requirement, let me code it and move on to next requirement” type of guy.
Does his code/thinking take into consideration how things can change/evolve in future? and does the current version of what he written/thinking will be able to easily accomodate those changes in future?
Has he ever said i am not able to do it and gave up on something when it comes to coding. If yes, what was it?
What’s he like as a coworker

https://twitter.com/asanwal/status/1474407648267313175?s=20

https://x.com/siddharthgopi/status/1703319494066888867?s=20

https://x.com/Altimor/status/1703198296255471948?s=20

https://www.matrixpartners.in/matrixmoments/extreme-referencing

Ultimate guide to doing reference checks

Appendix

Related reading: