Interviewing from the other side of the table, part 2: designing interview questions


Hiring new people is a risky activity for me as a manager. It takes time for screening and interviews, and once someone new starts, it will take the time of my team to train them before they are productive. There is risk in selecting the wrong person for a job, be it that they don’t have the right skills, or that they will be a poor fit for our team or culture. In this three part series, I will share how I screen resumes, how I design questions to learn about candidates and understand their skills, and what my team looks for in an interview. Team and culture is very important to me, so I am very selective about who joins our team; fairness and consistency are important as well, because for every candidate in the process, it is an incredibly personal journey to find a new job.

In the past year, I’ve been in some interviews where the interviewer talked the whole time, asked rambling questions that had no real answers, and asked questions that were actually illegal. Designing and planning interview questions is important because it provides consistency by which to judge each candidate’s skills, as well as formality and structure for the interview. Each interview for a given position should be more or less the same, allowing each candidate’s experience and personality to show.

Everyone is a customer, even job-seekers

I think it is important to give candidates a great experience with my company. From the moment an applicant is selected for an interview, to the time I walk them back to the lobby afterward, each interaction is a chance to share our company values—whether they are selected for a position or not. I don’t want anyone leaving with a poor experience with our company or the people on our team—each candidate could be a customer, and at the very least should want to apply for another position with us.

First, what are the key qualities and skills?

As I plan the hiring process, I have thought about what I want to find in the new team member. Usually, these qualities include both hard skills and soft skills. Hard skills include the ability to perform the tasks that make up the job, be it writing code, designing test scripts, or eliciting requirements. Soft skills include intangibles like negotiation, executive presence, and the ability to communicate clearly.

As a hiring manager, both when I am writing the job description, and designing interview questions, I rank the qualities that are most important for this hire. I think about the specific position, as well as a the mix of skills within my current team. For a recent position where I hired a Test Lead for an agile team, I included the following:

  1. Team culture (engaging, collaborating, action oriented) – soft skills
  2. Leadership and influence – soft skills
  3. Quality assurance principles and continuous improvement for the team’s testing process – hard and soft skills
  4. Test design and traceability – hard skills
  5. Test automation using Ruby and Cucumber – hard skills
  6. Agile mindset – hard and soft skills

Each quality is important to me, but some are more important for this role than others. Generally, I value soft skills over hard skills, since I put so much effort into building strong Scrum teams. I want team members to be able to work closely together, and be accountable to each other—I find it harder to find awesome team members than Test Leads, or Analysts, or Developers.

Interview Questions

Interviews are stressful. Its hard to get a feel for skills and team fit in an hour with someone new. Just like I design a meeting to achieve a specific goal, I design interviews that allow my interview team to get a feel for the candidate’s skills, experience, and personality. Each interview has a beginning, middle, and end; and follows a familiar structure that my team understands. Structure helps to lead the candidate though the process, and gives them a chance to feel more calm.

I have a philosophy about interview questions. Each question asked should have diagnostic value, and clearly elicit an actual answer. Questions should not be designed to stump the candidate or make them look bad. Every question should give the candidate a chance to share their best qualities, clearly articulate what they bring to the table, and feel like they had a chance to shine. Questions about hard skills tend to have correct and incorrect answers, but soft skill questions should be designed to understand the candidate’s thoughts and views on a specific topic. All this means that we should plan our questions beforehand.

Team culture questions

Since we can’t just ask, “Will you fit in with our team?,” we need to design questions that get at what qualities are are most important for successful team members. We want to find candidates with teamwork and collaboration, tact and finesse, engagement and openness. These should be questions that everyone has an experience they can share, as well as to provide clear direction as to what the interview team wants to know. These questions generally have no right or wrong answers, but I’ll get into that in part 3 of this series: What my team looks for in an interview.

  • Tell me about a time when you were a team member, rather than just a Tester on your team.
  • Tell me about a time when you took ownership, and removed a roadblock for your team.
  • Tell me about a time when you had to work with a difficult team member or stakeholder.
  • What did you bring up in your last team retrospective?

Leadership and influence

In a team lead role, it is important the candidate can share some past experiences where they showed leadership qualities. Just saying that “I led three other testers on my last project” doesn’t give us insight as to how well they performed, what they learned from the experience, or how they might approach it again. Questions about style, coaching, and resource management include:

  • Describe your leadership style.
  • Tell me of a time when you provided coaching or critical feedback to a team member so they could improve.
  • Imagine the time you have for system testing is cut in half, and you know this at the beginning of the project. How do you lead your team’s work differently?
  • A team member isn’t meeting your expectations. How do you handle this?

Quality assurance and continuous improvement

In our team culture, quality is everyone’s role, and we value the continuous improvement of our process. This tends to be a mix of both hard and soft skills, emphasizing taking action to try something new:

  • What are some quality assurance activities that have been successful for your current team?
  • What should your current team do differently? Why? What have you done about it so far?
  • Tell me about a time when you introduced a change to improve your team’s process.

Test Design

For every position, I include questions to elicit hard skills and experience required for the job.

  • How do you design tests for user stories? How do you ensure traceability to requirements?
  • What do you know about acceptance test driven development?

Test automation

I’m not going to go into audition interviews here, but for test engineers, we request some code examples on the whiteboard. For a Test Lead position, understanding their views on test automation strategy was key.

  • Describe how you decide what tests should be automated and what tests to perform manually.
  • What is the value of test automation?
  • What are some practices to ensure high quality test automation?

Agile mindset

An understanding of agile and Scrum is important to our team.

  • Why agile over waterfall? What value does it provide for our stakeholders?
  • What is a Tester’s role on an agile team? Is it different than on a waterfall team?

Questions I always ask

There are a few questions I ask in every interview—sometimes just to get the ball rolling, and sometimes to close out the process.

  • Tell me about yourself. Why did you choose this position?
  • Tell me about a time that you had to quickly become a subject matter expert for your team.
  • Share some critical feedback you have received recently.
  • Do you have any questions? Was there anything you wish we had covered that you want to share?

Questions I never ask

There are two groups of questions that I never ask—questions with little diagnostic value, and questions that I am legally not allowed to ask.

Questions with little diagnostic value

I think we may have all been asked questions in interviews that have questionable value with regard to understanding how one would perform a specific job. I’m sure there were reasons why those questions were asked, but I really want to trace our questions back to skills that are needed for the position.

  • Sell me this pen.
    I can’t imagine a position where this is relevant. Wait, maybe a pen salesman?
  • What are your top three strengths and weaknesses?
    Every candidate just presents strengths as weaknesses: “I am too dedicated, too driven, or too amazing.” Asking behavioral and team fit questions where the candidate can show their strengths are more useful.
  • If you were a kitchen appliance, what would you be?
    “That guy said he was a blender. They will never fit in here, we only want microwaves and toasters on our team.”
  • What is the best color?
    Obviously the answer is green. End of discussion.

Additionally, we’ve heard about the impossible brainteaser questions that some companies ask. My favorite has always been “How many basketballs can you fit in this room?” A few years ago, there were some articles about why Google decided this might not be the best approach to predict future performance.

Illegal questions

Any questions that can elicit information that could be used to discriminate against a protected class cannot be asked during an interview or included in a hiring decision. These questions also have no value in determining if the candidate has the skills or experience to perform the job. These include questions like:

  • Where are you from?
    Opportunity to discriminate based on national origin.
  • How old are you?
    Opportunity to discriminate based on age.
  • Are you pregnant?
    Do I really need to explain this one?

Its important to educate the interview team as to what questions should be avoided and why—I’ve always found it curious that companies would allow interviewers to have little to no training for something so potentially risky.

Choosing an interview team

I’d once heard that an interview team should be like a good luck bride on her wedding day, and include someone old, someone new, someone borrowed, and someone blue. As in, the interview team should include an experienced team member, a new team member, someone from another team, and someone critical of the candidate.

Generally, I like to include three to four people—team members from other teams, but all in the same role, with at least one person from the team that will include the future hire. I feel this is an engaging activity for associates, and my department uses it for almost every hire, to get as many perspectives as possible.

Before I became a hiring manager, I had never been involved in an interview other than my own. I’ve been amazed at what I’ve learned over the course of hundreds of interviews—what works well and what doesn’t, including what questions get the best responses. I think it can be a powerful tool to help interview candidates for your own team, and should give any associate better preparation for when they will interview for their next position.

One thought on “Interviewing from the other side of the table, part 2: designing interview questions

Comments are closed.