The Pointy Haired Boss

It is critical that teams have a defined set of interview questions that can clearly and objectively rate the skill set of any candidate.  And if you’re managing a team of engineers, chances are they’re doing this wrong.

In many interviews, engineers on the team pepper the candidate with a series of questions in an attempt to gauge whether the candidate is strong enough for the role.  Often this comes in the form of the “interview question sheet”, often with upwards of a hundred questions of various difficulties and domains.  There are several problems with this method:

1. Engineers equate trivia with useful knowledge.  Ask yourself how valuable is it really for someone know from memory specific details like how many LUNs your fibre channel adapter supports or what obscure flag you have to pass on the command line to an application.  And even if the candidate does know the answer, does that mean they actually understand how to solve a day-to-day problem?
2. Engineers equate trivia with depth of knowledge.  While knowledge of trivial details can be a proxy for depth or experience, the candidate might also just be lucky.  What if the the really obscure question you asked just happens to be a problem they recently stumbled upon?
3. Engineers like to select questions in which they already know the answer, and they especially like to pick questions based on the candidate’s resume.  This means that not all candidates will get the same questions, and that the questions asked may be very focused on one or two technologies.

In my experience a much better way to interview is to choose a few progressively difficult questions ahead of time and step through them from simple through arcane until the candidate gets stuck.  If you do a few of those in different knowledge domains, you will gain confidence that the candidate understands the technology at a particular level, and didn’t just get lucky or happen to know the answer.  For instance:

You’re trying to log into a remote server with SSH and your connection fails.  Walk me through how you debug this.

• A level 1 engineer should ask “is my login failing or does it never connect?”  If you say “you never get a login prompt” and ask what port SSH uses they should know 22/tcp.
• If you ask them to debug the connection a level 2 engineer should know the 3 way TCP handshake and either be able to debug where the problem lies using ‘ssh –v’ or since it’s TCP just trying a ‘telnet 22’ to see if it hangs on the SYN
• If you ask for a more detailed analysis of the connection problem a level 3 engineer should be able to run tcpdump or wireshark and tell you if the SYN is being dropped or rejected
• If you tell them it’s being rejected, a senior engineer should be able to explain why (host firewall set to REJECT instead of DROP) and how the REJECT is delivered (e.g. via an ICMP “destination port unreachable” message)
• A principal engineer should be able to tell you the ICMP message type is 3 and will likely know some of the ICMP codes (e.g. ICMP code 4 is “fragmentation needed but the DF bit was set”).

It’s possible not all candidates will be able to dive deep, but if you ask several questions like this across different knowledge domains (operating system internals, networking, security, etc) you should start to feel comfortable with their depth of knowledge.  Once you’re sure that the candidate has sufficient technical skills, the team should ask some general problem solving questions to evaluate how the candidate approaches problems like:

You’re faced with the task of deploying 500 new and identical servers this week and you have no existing tools to do so.  How would you approach this problem, what tools or methods would you use, what if anything would you ask me for, and what trade-offs or design decisions would you have to make in order to meet your deadline?  If I gave you 4 weeks to do this, what would you do differently?  What about 24 hours?  What if you had to do 20 different configurations (with 25 servers each)?

If you have time at the end of the interview and want to ask some questions that are highly relevant to their work experience you can do that, but your primary goal should be to determine their general technical depth as well as their ability to think through problems and work in a team before spending valuable time asking the candidate about the fusion reactor or flux capacity they built in their garage.  (Unless you’re a startup building a time machine, in which case that should probably be the first question you ask).