The best interview's I've done (very senior developer here) have been pair programming interviews. We have a suite of several problems and rotated through them to our candidates. The "problems" are a list of acceptance criteria for a program that we give to the interviewee a criteria at a time and they finish making the program up to the criteria they've seen so far. There usually a couple of difficult changes to the program as new criteria get revealed that need refactoring to achieve or at least careful thought. We sit with the developer and use a style called ping pong pair programming where we go back and forth as driver and navigator every test. The problems are about 5 AC each and can be done it about an hour and a half. We do it tdd style. It doesn't matter if they've never done tdd. It's easy with a guide to do it. Our company actually doesn't do tdd, but we do a fair amount of pair programming. The point of the TDD is to allow collaboration to switch back and forth as driver navigator and have a natural switch between the two. What this interview does is show a lot from entry level to senior level, how do they work, think, collaborate, problem solve as an actual developer. They're encouraged to work as you would work. You can use help/google/talking to the co programmer (the interviewer), just as you would while really working. The interviewer can help interviewees who need it in the form of collaboration. This help gets a lot of nervous people over their nervousness. This interview also reveals quickly if they have no ability to develop (i.e. they can't solve the problems or use the ide). I've had nearly half of the interviewees tell me this was the best interview they've ever been on. After the pair programming exercise we (the interviewer and interviewee) reflect and talk about it. This is where you actually do the senior developer interviewing. Talking about performance, repurcussions, re-use, refactorings, scaling, parallelisms. As you're doing the pair programming exercise, you get an idea and impression of what it would be like to work with the person and what their personality & work habits might be. From the hiring side. Every person I've decided to recommend hiring from this process has turned out to be a great addition to the company. I've never recommended somebody from this interview who quickly left or was a bad fit. I've also only ever heard of one other company doing this kind of interview, which is a shame, because the interview is basically work with me, solve a problem with me, do I, as the interviewer, like working with you and think you can work well in the company. We generally had one interviewer as a pair programmer and one other dev sitting and mostly observing and occasionally helping during the programming part of the interview and all three talked during the reflection part of the interview.
I think treating it like a pair programming session is the best way, whether doing algorithms or real-world problems. Even if the candidate hasn’t done pair programming before, it’s more like a condensed plan/code/review loop that people do on a daily basis.
Of course I think real world problems give better signal and opportunity for the candidate to demonstrate a broader range of skills.
Your method is similar to how I’ve led interviews. It’s also how the best interviews I’ve been a candidate for were done.
It felt like my regular work, the thing I do every day, instead of feeling like a school exam with a teacher that has to stay late and just wants you to hurry up so they can go home.
I love pair programming. I think not only that two heads solve problems better, reduce time and frustration during PR, exchange experience, but, which I think is very important, they also help build relationships in a team, which is what is most lacking in remote work.
The problem I see with pair programming on interview however is that you already know the solution. The candidate is still solving the problem and you are rating a solution. You are not actually working together.
Because it's an interview, it's ok if the interviewer knows some solutions to the problem, already. With the ping pong programming, where you switch every test and TDD, the person who knows what to do, does it. That still leaves the interaction that happens when he does it, and who's to say the person leaving you a non working test, gave you a direction you expected. It's only partially about solving the problem (albeit, a large part). It's also about working with the person and seeing how they interact and think (and tear problems down) while doing the programming. The talking that goes on while you do it, reveals a lot of what you want to know. Gauging when to help/prod a candidate to keep the programming going, is part of what you learn as you do more of these. It has a lot to do with the seniority level of the candidate or sometimes their nervousness/anxiety level or sometimes their problem solving ability. When helping them, approach it with a question that leads them to the answer in a helpful manner. Try to not make the person feel bad about not getting it. Many times it's writers block/brain freeze from the interview situation. Try and teach them while working with them, the same as if you were working with a new junior dev. If you can help them get past a block in a non adversarial way, often the rest of the interview goes well. Using this approach, even people who failed the interview have said to me, that was the funnest/best interview I've ever been on.
I might suggest the interviewer use a problem they’ve never seen before. But, I’ve tried pairing with people on Leetcode problems that I haven’t seen before in non-interview situations.
It was much harder for me to focus because I was part trying to understand the problem and part trying to help the other person. I don’t think the results were as good as if I already knew the solution. I’d need a lot more practice with it to be comfortable doing that as an interviewer.
Even then, I’d eventually run out of fresh problems.
When you know the solution, as an interviewer, you have to hold back and let the candidate solve it. You see their mistakes, but you have to let them try it out first before saying anything. You have to gently nudge them when they need it, but otherwise observe and wait. It takes balance.
It’s a skill in itself, and I think that’s why so many interviewers are bad at it. At the moment it’s a “I know it when I see it” situation. I can’t yet clearly define it, so I can’t clearly coach others how to do it.
What really happens is you get a ticket that says "Our largest customer wants a report that shows XYZ sorted by the third letter of each users middle name by EOD. Freaking do it now!" This is the algorithm for "Don't waste your time with Big-O nonsense and code something already". This is the real-world scenario you will see most often and the business analysts in your company think of it as O(1).
I came up with a solution of the ”hard” interview problem in Python in 5 minutes. The tests you suggested I would have flunked. So you are definitely right.
I picked a random LeetCode “hard” as an example of what their problems are. Maybe I should pick one that is more objectively hard 😅. But difficulty of the problem itself isn’t the problem with these types of interviews as the article explains.
Why do we technical interviews at all? Do other jobs work this way?
Every technical interview I've ever had is like two car mechanics sitting across from each other with the following exchange:
- What does a windshield do? Keep the wind out
- What does the vent do? Let the wind in
- What does the heat seater do? Heat the seat
- What does the stereo do? Play the beat
The questions I get asked are insipidly stupid, because the interviewer doesn't give a damn, because interviewing candidates is not a developers job, it's a managers job. And because he doesn't give a damn, I get ghosted as a moron for answering the questions correctly, just like I get ghosted for providing a working code test.
The only good interviews I've had in the biz were with managers only (no dev present), with only one exception - I once had a dev interviewer who actually had a paper printout of questions to ask, which were actually relevant to the job posting.
So as far as I'm concerned, we need to reject technical interviews, and being interviewed by devs. We should only accept interviews by managers, so we can be treated just like everyone else. Because just like everyone else, we have to be reasonably capable at our job to remain employed.
Leading interviews is a skill. If as a candidate you’ve only had bad interviewers, you don’t know what good looks like so are likely to be bad at giving interviews.
Companies that have standardized interview methods and training have devs that do better at leading interviews.
A tech interview should feel like working on a regular problem the company deals with on a day-to-day basis. It’s an opportunity for the interviewer to see what it’s like to work with the candidate. It’s an opportunity for the candidate to see what it would be like working with a member of the team.
Interviewing with a dev gives you a better feel for the job, the culture, and the team. If interviews with devs have been bad, seems like they need training, better interview challenges, or that working with that person will be bad.
The best interview's I've done (very senior developer here) have been pair programming interviews. We have a suite of several problems and rotated through them to our candidates. The "problems" are a list of acceptance criteria for a program that we give to the interviewee a criteria at a time and they finish making the program up to the criteria they've seen so far. There usually a couple of difficult changes to the program as new criteria get revealed that need refactoring to achieve or at least careful thought. We sit with the developer and use a style called ping pong pair programming where we go back and forth as driver and navigator every test. The problems are about 5 AC each and can be done it about an hour and a half. We do it tdd style. It doesn't matter if they've never done tdd. It's easy with a guide to do it. Our company actually doesn't do tdd, but we do a fair amount of pair programming. The point of the TDD is to allow collaboration to switch back and forth as driver navigator and have a natural switch between the two. What this interview does is show a lot from entry level to senior level, how do they work, think, collaborate, problem solve as an actual developer. They're encouraged to work as you would work. You can use help/google/talking to the co programmer (the interviewer), just as you would while really working. The interviewer can help interviewees who need it in the form of collaboration. This help gets a lot of nervous people over their nervousness. This interview also reveals quickly if they have no ability to develop (i.e. they can't solve the problems or use the ide). I've had nearly half of the interviewees tell me this was the best interview they've ever been on. After the pair programming exercise we (the interviewer and interviewee) reflect and talk about it. This is where you actually do the senior developer interviewing. Talking about performance, repurcussions, re-use, refactorings, scaling, parallelisms. As you're doing the pair programming exercise, you get an idea and impression of what it would be like to work with the person and what their personality & work habits might be. From the hiring side. Every person I've decided to recommend hiring from this process has turned out to be a great addition to the company. I've never recommended somebody from this interview who quickly left or was a bad fit. I've also only ever heard of one other company doing this kind of interview, which is a shame, because the interview is basically work with me, solve a problem with me, do I, as the interviewer, like working with you and think you can work well in the company. We generally had one interviewer as a pair programmer and one other dev sitting and mostly observing and occasionally helping during the programming part of the interview and all three talked during the reflection part of the interview.
I think treating it like a pair programming session is the best way, whether doing algorithms or real-world problems. Even if the candidate hasn’t done pair programming before, it’s more like a condensed plan/code/review loop that people do on a daily basis.
Of course I think real world problems give better signal and opportunity for the candidate to demonstrate a broader range of skills.
Your method is similar to how I’ve led interviews. It’s also how the best interviews I’ve been a candidate for were done.
It felt like my regular work, the thing I do every day, instead of feeling like a school exam with a teacher that has to stay late and just wants you to hurry up so they can go home.
I love pair programming. I think not only that two heads solve problems better, reduce time and frustration during PR, exchange experience, but, which I think is very important, they also help build relationships in a team, which is what is most lacking in remote work.
The problem I see with pair programming on interview however is that you already know the solution. The candidate is still solving the problem and you are rating a solution. You are not actually working together.
How do you solve that problem?
Because it's an interview, it's ok if the interviewer knows some solutions to the problem, already. With the ping pong programming, where you switch every test and TDD, the person who knows what to do, does it. That still leaves the interaction that happens when he does it, and who's to say the person leaving you a non working test, gave you a direction you expected. It's only partially about solving the problem (albeit, a large part). It's also about working with the person and seeing how they interact and think (and tear problems down) while doing the programming. The talking that goes on while you do it, reveals a lot of what you want to know. Gauging when to help/prod a candidate to keep the programming going, is part of what you learn as you do more of these. It has a lot to do with the seniority level of the candidate or sometimes their nervousness/anxiety level or sometimes their problem solving ability. When helping them, approach it with a question that leads them to the answer in a helpful manner. Try to not make the person feel bad about not getting it. Many times it's writers block/brain freeze from the interview situation. Try and teach them while working with them, the same as if you were working with a new junior dev. If you can help them get past a block in a non adversarial way, often the rest of the interview goes well. Using this approach, even people who failed the interview have said to me, that was the funnest/best interview I've ever been on.
Sounds like you’re doing it well.
Yeah, it’s a tough dichotomy.
I might suggest the interviewer use a problem they’ve never seen before. But, I’ve tried pairing with people on Leetcode problems that I haven’t seen before in non-interview situations.
It was much harder for me to focus because I was part trying to understand the problem and part trying to help the other person. I don’t think the results were as good as if I already knew the solution. I’d need a lot more practice with it to be comfortable doing that as an interviewer.
Even then, I’d eventually run out of fresh problems.
When you know the solution, as an interviewer, you have to hold back and let the candidate solve it. You see their mistakes, but you have to let them try it out first before saying anything. You have to gently nudge them when they need it, but otherwise observe and wait. It takes balance.
It’s a skill in itself, and I think that’s why so many interviewers are bad at it. At the moment it’s a “I know it when I see it” situation. I can’t yet clearly define it, so I can’t clearly coach others how to do it.
Unfortunately, I don’t have a solution.
What really happens is you get a ticket that says "Our largest customer wants a report that shows XYZ sorted by the third letter of each users middle name by EOD. Freaking do it now!" This is the algorithm for "Don't waste your time with Big-O nonsense and code something already". This is the real-world scenario you will see most often and the business analysts in your company think of it as O(1).
I came up with a solution of the ”hard” interview problem in Python in 5 minutes. The tests you suggested I would have flunked. So you are definitely right.
I picked a random LeetCode “hard” as an example of what their problems are. Maybe I should pick one that is more objectively hard 😅. But difficulty of the problem itself isn’t the problem with these types of interviews as the article explains.
Why do we technical interviews at all? Do other jobs work this way?
Every technical interview I've ever had is like two car mechanics sitting across from each other with the following exchange:
- What does a windshield do? Keep the wind out
- What does the vent do? Let the wind in
- What does the heat seater do? Heat the seat
- What does the stereo do? Play the beat
The questions I get asked are insipidly stupid, because the interviewer doesn't give a damn, because interviewing candidates is not a developers job, it's a managers job. And because he doesn't give a damn, I get ghosted as a moron for answering the questions correctly, just like I get ghosted for providing a working code test.
The only good interviews I've had in the biz were with managers only (no dev present), with only one exception - I once had a dev interviewer who actually had a paper printout of questions to ask, which were actually relevant to the job posting.
So as far as I'm concerned, we need to reject technical interviews, and being interviewed by devs. We should only accept interviews by managers, so we can be treated just like everyone else. Because just like everyone else, we have to be reasonably capable at our job to remain employed.
Leading interviews is a skill. If as a candidate you’ve only had bad interviewers, you don’t know what good looks like so are likely to be bad at giving interviews.
Companies that have standardized interview methods and training have devs that do better at leading interviews.
A tech interview should feel like working on a regular problem the company deals with on a day-to-day basis. It’s an opportunity for the interviewer to see what it’s like to work with the candidate. It’s an opportunity for the candidate to see what it would be like working with a member of the team.
Interviewing with a dev gives you a better feel for the job, the culture, and the team. If interviews with devs have been bad, seems like they need training, better interview challenges, or that working with that person will be bad.