One of the challenges faced as a hiring manager is how to decide if the candidate you just interviewed is really the best candidate you’re going to find.  Whatever you do, don’t pick the first few candidates you interview.  Even if they seem awesome, don’t do it!

For those who love statistics and are unfamiliar with optimal stopping theory, I recommend reading what’s commonly called the “Secretary Problem”.  The summary for those who dislike math is that if the number of candidates (N) that will apply for this role is known, you should interview and then immediately reject the first N/e candidates (where the mathematical constant e = 2.71828).  So statistically speaking if you expect to get 50 candidates, you should reject the first 18.394 outright, and then you should hire the next candidate you find that is better than anyone else you’re interviewed so far.  Even if candidate 1 or 2 seems great, the odds are that some candidate between interviews 19 and 50 will be even better.

“But how do I know how many people will apply” you may rightly ask?  As it turns out, if you don’t know N but you do have a defined time period (T) in which you must hire, there’s a similar “1/e-law of best choice” which applies.  In that case, you should interview and then reject all applicants which apply during the first T * (1/e) period.  So for instance if you must fill the role within 30 days (perhaps for the holiday shopping season), you shouldn’t make an offer to any candidate that applies during the first 11 days (since 30 * (1/2.71828) = 11.036). but starting on day 12 you should hire the next candidate that comes along who is better than any prior interviewee.

“But what if I don’t know the time period (T) or number of candidates (N)?”  As it turns out, so long as you have a reasonable enough time period to find a sufficient minimum number of candidates, you can probably apply some estimates (perhaps T=90 or N=50) and divide by 3 to come up with a results that reasonably approximates the mathematical version (see Heuristic Performance of the Secretary Problem).

Of course like most managers I don’t completely trust statistics in the real world.  So while I don’t completely subscribe to the above (though it’s been shown statistically to select the best candidate 37% of the time), I do believe it provides two good rules of thumb that every hiring manager should use:

1. Always interview several candidates before even considering one, and
2. Always pass on a candidate that isn’t a slam dunk – don’t keep them “on the back burner” in case you can’t find someone better

It is critical that teams have a defined set of interview questions that can clearly and objectively rate the skill set of any candidate.  And if you’re managing a team of engineers, chances are they’re doing this wrong.

In many interviews, engineers on the team pepper the candidate with a series of questions in an attempt to gauge whether the candidate is strong enough for the role.  Often this comes in the form of the “interview question sheet”, often with upwards of a hundred questions of various difficulties and domains.  There are several problems with this method:

1. Engineers equate trivia with useful knowledge.  Ask yourself how valuable is it really for someone know from memory specific details like how many LUNs your fibre channel adapter supports or what obscure flag you have to pass on the command line to an application.  And even if the candidate does know the answer, does that mean they actually understand how to solve a day-to-day problem?
2. Engineers equate trivia with depth of knowledge.  While knowledge of trivial details can be a proxy for depth or experience, the candidate might also just be lucky.  What if the the really obscure question you asked just happens to be a problem they recently stumbled upon?
3. Engineers like to select questions in which they already know the answer, and they especially like to pick questions based on the candidate’s resume.  This means that not all candidates will get the same questions, and that the questions asked may be very focused on one or two technologies.

In my experience a much better way to interview is to choose a few progressively difficult questions ahead of time and step through them from simple through arcane until the candidate gets stuck.  If you do a few of those in different knowledge domains, you will gain confidence that the candidate understands the technology at a particular level, and didn’t just get lucky or happen to know the answer.  For instance:

You’re trying to log into a remote server with SSH and your connection fails.  Walk me through how you debug this.

• A level 1 engineer should ask “is my login failing or does it never connect?”  If you say “you never get a login prompt” and ask what port SSH uses they should know 22/tcp.
• If you ask them to debug the connection a level 2 engineer should know the 3 way TCP handshake and either be able to debug where the problem lies using ‘ssh –v’ or since it’s TCP just trying a ‘telnet 22’ to see if it hangs on the SYN
• If you ask for a more detailed analysis of the connection problem a level 3 engineer should be able to run tcpdump or wireshark and tell you if the SYN is being dropped or rejected
• If you tell them it’s being rejected, a senior engineer should be able to explain why (host firewall set to REJECT instead of DROP) and how the REJECT is delivered (e.g. via an ICMP “destination port unreachable” message)
• A principal engineer should be able to tell you the ICMP message type is 3 and will likely know some of the ICMP codes (e.g. ICMP code 4 is “fragmentation needed but the DF bit was set”).

It’s possible not all candidates will be able to dive deep, but if you ask several questions like this across different knowledge domains (operating system internals, networking, security, etc) you should start to feel comfortable with their depth of knowledge.  Once you’re sure that the candidate has sufficient technical skills, the team should ask some general problem solving questions to evaluate how the candidate approaches problems like:

You’re faced with the task of deploying 500 new and identical servers this week and you have no existing tools to do so.  How would you approach this problem, what tools or methods would you use, what if anything would you ask me for, and what trade-offs or design decisions would you have to make in order to meet your deadline?  If I gave you 4 weeks to do this, what would you do differently?  What about 24 hours?  What if you had to do 20 different configurations (with 25 servers each)?

If you have time at the end of the interview and want to ask some questions that are highly relevant to their work experience you can do that, but your primary goal should be to determine their general technical depth as well as their ability to think through problems and work in a team before spending valuable time asking the candidate about the fusion reactor or flux capacity they built in their garage.  (Unless you’re a startup building a time machine, in which case that should probably be the first question you ask).