Technical Interviewing – surely we can do better

Introduction

Ted Tencza discusses the technical interviewing process, drawing from 25 years of experience. He emphasizes the importance of improving interviewing practices and highlights the need for takeaways applicable to general interviewing for tech leaders and managers.

Problems with Technical Interviews

Ted outlines common problems, including gatekeeping (degrees, prior companies, time spent at companies), bias (especially against women), and bad coding tests. He discusses the "thou shalt not pass" syndrome and the tendency of companies to copy-paste interview practices from other companies without considering their own context.

Bad Coding Tests

Ted delves into the reasons behind bad coding tests. He points out the irrelevance and unreliability of many tests, particularly those with convoluted questions or unclear expectations. He also criticizes the variability in test formats and the focus on memory rather than practical development skills.

Poor Interviewing Skills and Overwhelm

Beyond technical tests, Ted addresses the issue of interviewers lacking interviewing skills. He highlights the problem of overwhelming both candidates and interviewers with excessive interviews, especially in companies with high hiring needs.

Candidate Impact and Asymmetrical Effort

Ted discusses the negative impact on candidates, particularly the asymmetrical value and effort. He argues against requiring excessive effort from candidates early in the process, emphasizing the importance of providing value to candidates upfront.

Proposed Solutions and Their Issues

Ted examines several proposed solutions to improve technical interviews, including scrapping coding tests, using GitHub repos, pre-packaged tests, take-home projects, and one-day trials. He analyzes the shortcomings of each approach, highlighting their potential for exclusion and reliance on unreliable data.

Keys to Good Solutions

Ted presents key principles for effective technical interviews, including candidate care, clear expectations, avoiding assumptions about personal time and resources, making tests relevant and iterative, and helping candidates when they get stuck.

Standardization and Interviewer Training

Ted stresses the importance of standardizing expectations, both for candidates and interviewers. He recommends using a question library with exemplar answers and training interviewers effectively, potentially using a tiered system to assess interviewer skills.

Candidate Briefing and Realistic Conditions

Ted advocates for a candidate briefing package to reduce stress and provide information about the interview process. He also emphasizes setting up realistic conditions for coding tests, allowing access to resources that would be available in a real-world development environment.

Homework and Closing

Ted encourages listeners to audit their own interviewing processes and offers his assistance. He reiterates the importance of candidate care and shared expectations.

Okay, so I want to talk to you today about, the technical interviewing process, but it's not going to be just limited to that.

There's also going to be, hopefully, some takeaways for general interviewing around, maybe your tech leaders or managers or whomever.

And I've been in this industry for, 25 years and, I've seen a lot of really, horrendous interviewing processes, as Gretchen was, talking about.

And I think that it's incumbent upon us as leaders to do a better, job at it.

What we'll cover today is, we'll talk about what are some of the problems, what are some solutions that are out there, everybody's got a blog about, how to do better hiring, and getting white people pissed off about it, we've got some keys to a good solution, so I'm not going to tell you exactly how your interview process should go, because that's going to be a lot based on context of your organization, but I will give you some ideas on what you can do to ensure that it is good, and that and then some next steps.

Once again, more homework.

So what are some of the problems that we see?

First off is there's a lot of gatekeeping in the industry.

Now what is gatekeeping?

Gatekeeping is basically making sure that you're focusing on things that aren't important for the role and using them as exclusions so that you can prevent certain people from even, maybe even getting into the interview process.

One of the ones that a lot of people use are degrees.

Every job that I've had since I've moved to Australia has said bachelor's in computer science required, master's preferred.

I've never taken a computer science course in my life.

Not one.

I've got a master's degree, but it's in history.

But I applied, because I look like me, and so I can get away with it.

There's also gatekeeping around prior companies.

You look at somebody's resume, and they are, they've been at one of the really cool ones.

I worked, like I said, 25 years.

Of that 25 years, four of them were at Atlassian, and I've been trading off that ever since.

It's just a really good way to get your foot in the door.

And everybody, or most everybody at Atlassian will tell you the same thing, same with Google or Facebook or any of these other companies.

And it doesn't matter, because, trust me, at Atlassian I met some people that really weren't good coders.

It's not like just being there automatically made you a good developer.

And then also there's a real bias around time that you spend at certain companies.

Either it's too little, so somebody's like a job hopper because they've only spent, two years or, one year or two years at several companies, and so you think, oh, they're gonna, they're gonna, they're flighty and they're gonna leave quickly.

Or somebody spent, 12 years at a company, oh, they don't like change, so we're not going to do it.

So you're unless you have that perfect kind of, okay, three to five years at every company in your entire career, some people are going to go, oh, let's, go ahead and toss this, CV out.

There's also a problem with bias.

Gretchen was talking about, when you get, especially females coming into the thing going, oh, are you technical enough?

And having these unconscious bias that really impacts it.

And if you take nothing else away from this talk, please go back to your organizations and remove not technical enough from any feedback form on any interview.

And anybody that does it, just retrain them.

Because it's just not what you want to, say as just standalone comment.

And finally, specifically with the technical interviews, there's a lot of really, bad coding tests out there.

And I'll go through a lot of the reasons why.

So here are the common problems.

You also have what I call thou shalt not pass syndrome, where somebody feels like, okay, they've been entrusted with this interview.

They have to protect the company, they have to protect the culture.

And so they go into it with a mindset of, I'm gonna make sure I fail that this candidate, because we don't want to, we don't wanna ruin it.

And then the candidate has to be above and beyond what it is, and this idea that, 80 percent of companies believe they have the top 10 percent of the market.

You do the math.

I'm a history major, I don't do that.

But you can see where the problem is.

So they're not setting up, correct expectations, and they've got this really standing there going, I'm going to protect this organization at all costs, and then losing out on really good developers because they don't have a degree, or they only spent two years at a company, or whatever they've done.

' .

You have a lot of companies that do copy and paste solutions, and what

I mean by this is they say, okay, this company is successful, and I heard this rumor that this is how they do the interviews, so we'll add that to our interviewing process.

Famously back in the sort of early days of Google, they had the brain teasers, how many light bulbs does it take to fill a marble filled pool in Los Angeles, or something like that.

And it's like it made absolutely no sense to what the what the job was.

And then Jared made a mention about Shopify, and the Spotify model and how it was sort of cargo culted.

And I was at a talk once where there was a engineer from Spotify who was like, at Spotify we don't use the Spotify model.

Never have.

It was a marketing blog post and everything.

And yet there's all kinds of people that are now copying and pasting that, that solution into organizations.

And that's one of the reasons why I'm not going to tell you exactly how your interview process should be structured down to how many interviews and how long they are and things like that, because it depends on your context.

Are you a startup?

Are you a really established company?

It makes a difference on how you want to go through your interview process.

And if anybody wants to take a picture There's the picture slide.

So if we talk about the bad coding tests, why, are they bad?

First off, you have a lot of stuff around, it's irrelevant, and it's unreliable.

You have tests that have absolutely no relation to the work that's being done, and it's very much a, CS algorithm that's really gonna, to benefit a very specific cohort of candidates who are recently graduated or.

Spend a lot of time really working on this thing without necessarily being a good software developer and being able to understand the business and the problems and solving problems.

Instead, it's, reverse the string in place.

Never had to do that in production, ever.

But I have had to do it in coding tests.

A second problem with a lot of it is when you're dealing with people who maybe don't have English as their, primary language, their first language, and I've seen these questions on technical tests that are so convoluted as a native English speaker, I couldn't understand what they were asking for.

You've got a, two different arrays of nodes.

One's a parent node, one's a child node, put them in order of the child nodes.

If unless it's a grandparent node, in which case, then you have to reverse it.

And I'm like, I have no idea what you're asking for.

And it's just absolutely impossible.

And that's coming from somebody who knows how to speak English as the only language I speak.

You have unclear expectations a lot of times.

At a previous company, somebody would take a technical test and get done and even the interviewer wouldn't be sure if that was a pass or a fail.

Because they didn't have a standard of, if they get through three questions and the code is good, then it's a pass, and if they can only do one, then it's not a pass.

And there's also very unclear expectations for the candidate.

They don't know, am I supposed to be working, making sure that it's super long and everything's commented and I've got, documentation, or am I just supposed to, meant to get it working, because I've only got forty five minutes or whatever.

Okay and then you have a bunch of variability.

So when I say you're coming in for a technical test, everybody here now has a different imagination of what that's going to look like.

Is it going to be writing code from scratch?

Is it going to be doing a code review?

Is it going to be doing, here's some code that doesn't work, fix the bug and make the test pass?

Is it going to be pseudocode on a whiteboard?

Which, please ban.

I, in 25 years I've never shipped a whiteboard to production.

It's really not what you want to do.

You want to focus on things that are actually relevant.

And then you also have a take home test a lot of times.

And, in talking to people, I was hearing about one company that was doing sort of a take home test, and they said they time boxed it to 16 hours.

And I was like, that's not a time box, that's like a time shipping container.

And so it's, you really, that's not what you want to do, and again, it favors a very specific cohort of people.

You're not going to get a diverse candidate pool and a diverse workforce, you're going to get people without any care responsibilities, you're going to get people that don't have to drive their kids to sport on Saturday.

You're going to get a very limited cohort if you're going to require them to spend their entire weekend coding.

You also have, interview lottery.

If you don't take the time to make sure everybody's on the same page, you get stuff where people focus on irrelevant criteria.

Previous company that I worked at, one of the interviewers, if they were doing a technical test, would automatically fail a candidate if they used their mouse.

Because they believe that, good developers use keyboard shortcuts.

I've been on projects that have lasted a long time, some of them did come in a little late, but it was never because somebody was using a mouse and not keyboard shortcuts, trust me.

That's not where the delays were.

You also have style bias.

Some people like to just get in there and just start writing code.

And just trying to get the quickest thing to work and then iterate on it and make it better.

Other people like to sit and think and go, okay, I'm going to figure out how to do this and try to get a more complete solution on the first attempt.

Now, if you're a, I'm going to just do what I can right away type of person, and the interviewer is the other style, they may not recognize that both of those are legitimate ways to do that.

And you end up, again, with just a lottery of did you get the same type of person working the same way that you happened to work.

A lot of times technical interviews can, turn into syntax bingo.

Where, you're asking about, okay, tell me about the difference between this particular syntax and that syntax.

Without anything in front of you, and all of a sudden under pressure, you're like, oh, I can't remember, I sometimes forget how to, properly structure a for loop.

It just depends.

And nobody in the real world has to sit there and do everything from memory.

They invented stack overflow for a reason.

And so you get, unreliable results, because you're, testing memory rather than the ability to develop.

And then you have, sometimes you have multiple solutions, but you only accept one.

And this goes back to interview lottery.

Does your thing, actually work with the person that's scoring it?

So we had a, at a previous company, we had a take home code review.

That could be solved either by just throwing the entire code away and rewriting it as like a two line function, because that actually would have worked, or going through and really explaining why all these things are wrong.

Both of those are legitimate ways, but if you're were a, I'm gonna do the really quick short term answer, and then you got scored by the somebody who wanted to see all of the comments, then you were out.

Even though both are legitimate.

So this is why I think that there's a lot of irrelevance and unreliability within the technical interviewing process.

Another problem, and this goes to not just technical interviews, but a lot of interviews in our industry, and that is, one, is that you have a lot of people with exceptionally poor interviewing skills.

And this goes back to what Andrew was talking about earlier, where it's like, OK, you've done coding for a while, now you're thinking about all these other skills, whether it's going into management, but it's also for leadership.

OK, you've coded for five years, you're now an interviewer.

Without any training on how to actually do an interview, and how to look to assume that, the person can actually evaluate what, not just good code, but what a good developer would look like.

What kind of questions do you ask?

You also have the tendency to overwhelm the candidate with interviews.

I felt like sometimes, this was my interview here, and this was the interview panel.

Like, all of that, literally up against, six or seven different people, all sitting there asking you questions.

And it's absolutely ridiculous.

You can also have a tendency to overwhelm the interviewers.

I was just speaking to somebody the other day, and their company, they're looking to hire probably 30 to 35 more developers within the next, 6 to 8 months.

And they have, according to their CTO, about 4 good interviewers.

Now, if you've only got 4 people trying to interview for 35 roles, which means you're going to have to interview probably 100 something people in order to get it… and then, what's going to happen?

That interviewer is just going to, never be able to do anything other than interview, because they're not spreading the workload around.

And then when it comes time for the performance review, I guarantee you someone will go, Oh, you only did 4 Jira tickets this quarter, why weren't you doing your job?

Because I was interviewing the entire time So you want to make sure that you have a good pool of interviewers, and it is a skill, and more people can do the interviewing, and so you don't want to sit there and have everyone, for all of the ability, fall on a very small cohort of developers.

You wanna make sure that you spread that around and make sure everybody is doing it.

Again, you wanna make sure that you've got people with interviewing skills, you wanna make sure that you don't overwhelm your candidates and you wanna make sure that the candidates themselves aren't, or the interviews themselves aren't overloaded.

There's also a lot of candidate impact with, problems.

And the first thing I like to talk about is this idea of asymmetrical value and effort.

And that is one side has to do all the effort while the other side has all the value.

And so it's a very asymmetrical trade off.

And a lot of companies will say, okay, we want the candidate to put the effort in so that we can get the value.

So very first step in the process is a 16 hour take home test.

Now, as a candidate, I've got to do all of the effort, and I get no value out of this.

I don't know if, even if this is a great company, I don't know what their, culture is like, nothing like that, I just, looked at the job description, sent in my application, and now I've got two days of homework.

And honestly, a lot of times you'll get people to just drop out, especially really good developers who have options will just drop out of the process right then.

When you're doing this, swap that around.

Do make the company put in the effort.

So that the candidate gets value.

And then by the time the candidate has that value, then you can start to make the candidate put in effort, because they've already seen the value and why they should be putting in this effort, and why they should be working towards it.

So you definitely want to make sure that you don't have that.

And this also comes back a little bit to the copy and paste mode, in that really, well known companies can have a lot more effort on the candidates part because the value is that oh I get to go work for Atlassian or Google or someplace like that but if you're just a regular startup or just an average thing and you're not super well known companies or candidates aren't going to put in that effort and you're going to lose out on good candidates And, honestly, even the companies like Google and Atlassian have started to come around to this and they're trying to put a little bit more value for the candidate in early, even though those kind of companies can get away with it.

And the reason is, again, skilled candidates will have other options.

As somebody mentioned earlier, if the culture's bad and somebody can, have a job the next day day, they're going to walk.

The same happens with interviews.

If you sit there and your interviewing process is long and doesn't have a lot of good feedback and is putting all of the effort onto the candidate, a good candidate will just drop out of the process and will go to the next company on the line and they can have a job right away.

And so you are losing out on potentially really, good candidates.

So you definitely want to make sure that look at what your effort is, how are you treating your candidates, and making sure that they're not just dropping out of the process.

So here are some solutions, most of these have been proposed by other people and I found them in as I was looking into this talk, and I want to go through them and some of the, issues I see with these particular solutions.

People have said just scrap a coding test altogether.

Just don't code it.

Assume that somebody with, three or four or ten years of coding experience knows how to code.

Or, don't do coding.

Just have them give you their, personal GitHub repo, so that you can look at their code online.

Or do pre packaged coding tests.

Something like the HackerRank kind of thing, where, they've got all of these pre made questions, and can give you a really detailed answer to, how the candidate went about doing the work.

Doing a take home project, so you just sit there and say, okay, here, take this away, and do it in your own time and get back to us in, a certain amount of time.

Or, even, I've seen some people talk about, just don't even interview them, just hire them for a day and work with, work next to them for, a day.

And all of these have some good points and some bad points, I'm going to talk a little bit more about the bad points because I want to talk about ways to improve.

And here are some of the problems.

If you scrap the coding test or you outsource it to things, you're going to start to lose some of that data that you really want to have to make a hiring decision.

How does the person work?

How do they interact with the interviewers?

How do they, comport themselves?

It's also extremely exclusionary.

If you want a diverse workforce, you can't use these kind of things that all skew towards single, young, developers who have no other responsibilities outside of, of work.

I've, I know a lot of people, it's give me your private, personal GitHub account.

And it's I don't code on the weekends, I do family stuff on the weekends.

I don't have a GitHub account that I can show you, for this thing.

Again, it's a lot of the value asymmetry.

When you're getting people to do take home tests, when you're getting them to do these hacker rank things, You're making them do all the work where you're getting the value for it.

Once you start doing stuff like a take home test or prepackaged coding test, now the answers become available online.

Somebody will post the question to Glassdoor, or somebody will, have taken that exact same question out of the question bank from HackerRank or whatever.

It also, A lot of these things, they make assumptions not based on actual evidence, and what I mean by this is when you take like a hacker rank test, it'll say, the person took 45 minutes to, to finish this thing and the average is 12 minutes, therefore they're a very slow developer.

No, what happened was they started the thing and the kid started crying, they had to go, the tradie showed up and they had to go let them in, or, who knows what happened.

What you don't know is you don't know that it took this candidate 45 minutes of constant work to try to solve it.

You only know that it took 45 minutes for them to hit the submit button.

And they have things where it's how many times they tabbed off the screen.

And maybe they were trying to get this done while they were trying to do something else.

Again, it just makes a whole lot of assumptions that you don't want to factor into your hiring decision.

And a lot of it's irrelevant.

Again, if I'm working for a company that does, a comparison website, for instance, and the test is all about how to reverse a string in place, I'm never going to do that at work.

What I want to do is I want to find out, can this person solve the problems that we're facing as part of our company's work?

So again, I can't give you exactly what you should do for your interview, but there are a couple of keys that I want to give you to good solutions.

First, make sure that you have candidate care at the forefront, right?

If, what you really want to do is you want to make sure that you're taking care of these candidates.

So even candidates that have a, a no at the end and don't get hired, don't walk away going, wow, that company is filled with a bunch of jerks.

Because if you don't take care of the candidate, when you do get somebody who's successful, they're going to say, I don't want to work with these people.

They were, awful to me.

And so you want to make sure you're doing that right up at the front, and it should be a driving factor for all of the interviewing process that you do.

Have clear expectations for the candidate.

Make sure that you know when you come in for the technical test, it was going to be some pre written code that you have to find the bug in, and then, solve it.

Telling them that ahead of time does not give anyone an unfair advantage, it doesn't skew the results, they still have to write the code, or solve the code, but it lowers the stress level and allows for them to have a more realistic, outcome.

Don't assume personal time.

There's a lot of people who are really good candidates, who will do a lot of work for you during work hours, and then when they're out of work, they turn off.

And that's just how some people are.

And that doesn't make them a bad candidate, doesn't make them a bad developer, it just makes them not a 23 year old computer science major graduate from last week that has no other life to worry about.

Don't assume prosperity.

And what I mean by this is if you sit there and have somebody doing like a take home test that requires a super high end spec laptop.

Maybe they have one for work, but they might not have one for home.

Especially if you're talking about, people in lower cost of living areas.

Where, if you're doing like a distributed team.

And they can't use their own work laptop for interviewing for a different company, because that's unethical.

And if they're using their own, it's too slow to actually solve the problem.

So don't assume that people are just going to have, just because we're developers doesn't necessarily mean everyone is high paid and it does mean that everyone has, access to super high end machines.

Make it relevant.

And this, is vital.

Don't ask questions that have absolutely nothing to do with what their role is going to be because then you'll get information that's not going to help you decide whether or not this person is going to be good at the role.

Makes sense.

Make it iterative.

And what I mean by this is constantly look at your candidates, See how they're going, look at the tests, make sure that the tests are asking the right questions, have a, my previous company I was at, we set up a monthly review with the senior leaders, and every month we talked about the questions on the test and said, have we learned anything?

Is there a way we can do it better?

And one of the improvements we made was, originally, you had to solve four or five different bugs before the test started to pass, and we realized that was making people really nervous, because they'd solve it, and none of the tests would pass.

And then they'd start going, oh, what am I doing wrong?

What am I doing wrong?

And getting jittery and everything like that.

So we switched it around and made it so every single time you fixed something, a test would pass.

So you get that positive feedback loop going, which made for a much better experience for the candidate, and it also made for a more realistic, sort of thing.

If you're writing code, you don't want to have a test that requires, 18 different bugs to get fixed before you, before the tests pass.

Make your tasks clear, right?

Explain exactly what you want to do, not sort of Google's old, trick question kind of thing.

Make a very, specific, very clear test.

Focus on problem solving, right?

We don't need to solve necessarily, depending on sort of your industry, you don't necessarily need to solve super hard computer science algorithm type questions.

Make it relevant and focus on solving a problem, especially a problem that you might face in your company.

And help candidates to get stuck.

I still remember a company where they put the candidate in the room for a coding test and then left the candidate in there by themselves.

For an hour.

And, if they had, even a little problem where, you can't get your display to go onto the right screen, you've lost an hour of the person's time.

So again, make sure that you keep candidate care at the forefront.

It will help you, and you'll get especially good word of mouth, even if they aren't necessarily a successful candidate.

And then standardize the expectations.

Have a question library and have some of what are the, exemplar answers so that everybody understands what you're doing.

Don't have the code review that has two completely different answers and just depend on luck on whether or not you get the right person scoring it.

Say this is what we accept, this is not what we want, and then, make sure that you're explaining it to the candidate too so that the way you ask the question gives them a hint as to what you're expecting from, the answers.

And make sure that the interviewers, the entire people that are all doing the interviews, have that shared expectation of what the answer is.

And then clear expectations on what is passing.

Is solving three of the four problems passing?

Is solving two of the four problems passing?

What is technical enough?

And make sure that nobody is saying not technical enough without explaining why they weren't technical enough.

What did they fail?

Avoid Blocking questions, leading questions, and closed questions.

And by this, blocking questions are questions where you basically cut out potential answers.

So if I were to say to a candidate, which step would you do first if you were creating a CI/CD pipeline?

Would you set up, your Bamboo instance first or would you set up something else first?

Now all I've done is I've given them, I haven't let them explain their process.

I've, limited their, answers.

And so you don't get a good answer from, the candidate.

Or leading.

So did you learn something from that, didn't you?

Yes, , it, doesn't tell you anything.

The only time I would say that you can use these kind of questions is you've got a candidate that's really nervous and stuck and you can use it to, to, free them up and, have them get on a roll and get a couple wins under their belt as they go through the interview.

And set up realistic conditions, right?

As, as we do our jobs, we have access to the internet, we have access to an IDE.

Don't sit there and criticize syntax as if you're writing it on a whiteboard.

That's just not how development works.

And so that shouldn't be how a technical interview works.

Standardize the experience for all the candidates and for the interviewers so that you're getting a consistent experience.

We also to do is a candidate briefing package.

Send the candidate the information beforehand, which gives, how many tests, how many interviewers, how many interviews, how long are they going to be, etc.

It reduces the stress because it reduces the unknown.

Making sure that they have, the appropriate equipment.

If they need to have, if they need to come into the office in order to have it, make sure that they've got an IDE that they're familiar with.

If you're doing something like, for instance, you're having them implement an API, make sure that they have that ahead of time.

And if you need to do any sort of pre work, again, based on not wanting to do asymmetrical by an effort, I encourage you to very, much minimize the amount of pre work that you're asking for.

And then train your interviewers.

It's a skill, right?

Get, recognize it as a skill.

Actually, in my previous company, we set it up and we actually had levels of interviewers.

We had junior interviewers, mid level interviewers, and senior interviewers that had nothing to do with their title as junior developer, mid level developer, or senior developer.

It was how good they were at interviewing.

So you could have a senior interviewer, Developer who is a junior interviewer, and then your talent p and c team can then match up when you have a panel to say, okay, look, don't put two juniors on the same interview panel.

Put a junior on with a senior interviewer so that you get a better interviewing experience review.

Your processes, make sure that it's still relevant, and have somebody who monitors all the feedback so that you can see if some of your people on your interview panels are biased against certain types or if the feedback is, so minimal that you can't determine why they made the decision.

If you get really good feedback, it should be like you were in the interview yourself.

The homework, go back and please, everybody, audit your interviewing process.

Take a look at it.

Do you have candidate care at the front?

Is everybody got a shared understanding of what's expected of these things?

And little plug, if you, do run into roadblocks, feel free to reach out to me.

I can help you with this.

I've done it at a lot of different companies.

Thank you very much.

Web Direction Code Leaders 2024

Technical Interview Processes—Surely We Can Do Better

By Ted Tencza

Abstract, purple gradient on the right side.

What We'll Cover

01 Problems

02 Some Solutions

03 Keys to Good Solutions

04 Next Steps

Problems

Common Problems

Gatekeeping

+ Degrees + Prior companies + Time at prior companies

Common Problems

Bias

Common Problems

“Thou shall not pass” syndrome

Common Problems

Bad Coding Tests

Common Problems

Key Points

  • Gatekeeping
    • Degrees
    • Prior companies
    • Time at prior companies
  • Bias
  • "Thou shall not pass" syndrome
  • Copy / Paste solutions
  • Bad Coding Tests

Irrelevance / Unreliability

Tests/tasks have no relation to work that will be done

+ CS Questions / tasks

Irrelevance / Unreliability

ESL Skills

Irrelevance / Unreliability

Unclear expectations (on both sides)

Irrelevance / Unreliability

Variability (From Scratch, Code Review, Bug Fix, Whiteboard, Take Home)

Irrelevance / Unreliability

Interviewer Lottery

  • Focusing on irrelevant criteria
  • Style bias (Just get it working vs. trying for optimal solution from start)

Tests with multiple solutions

—but only accepting one

  • Take home code review - one line correction or long essay both possible interpretations

Irrelevance / Unreliability

Tests with multiple solutions — but only accepting one

+ Take home code review - one line correction or long essay both possible interpretations

Irrelevance / Unreliability

  • Tests/tasks have no relation to work that will be done
    • CS Questions / tasks
  • ESL skills
  • Unclear expectations (on both sides)
  • Variability (From Scratch, Code Review, Bug Fix, Whiteboard, Take Home)
  • Interviewer lottery
    • Focusing on irrelevant criteria
    • Style bias (Just get it working vs. trying for optimal solution from start)
  • Syntax Bingo - memory under pressure
  • Tests with multiple solutions - but only accepting one
    • Take home code review - one line correction or long essay both possible interpretations

Other Issues

Poor Interviewer skills

+ Assuming if someone can code, they can interview developers

+ Assuming they can properly evaluate performance

Other Issues

Overwhelming interviewer with candidates

Common Problems

Key Points
  • Poor Interviewer skills
    • Assuming if someone can code, they can interview developers
    • Assuming they can properly evaluate performance
  • Overwhelming candidate with interviewers
  • Overwhelming interviewer with candidates

Candidate Impact

Asymmetrical value/effort

Candidate Impact

Skilled candidates will have other options

Candidate Impact

Key Points

  • Asymmetrical value/effort
  • Skilled candidates will have other options

Some Solutions

Solutions proposed by others

  • Scrap coding tests altogether
  • GitHub project reviews
  • Prepackaged coding tests—e.g., Hackerrank
  • Take home projects
  • Day contracts (or longer)

QR code in the bottom right corner.

Issues with those Solutions

  • Lack of data to make hiring decision
  • Exclusionary—biased towards a specific cohort of developers
  • Value Asymmetry
  • Answers might be available online
  • Assumptions not based on evidence
  • Irrelevancy—especially pre-packaged tests

Keys to Good Solutions

Candidate Care At The Forefront

Should be driving factor in everything else that follows

Candidate Care At The Forefront

Clear expectations—and no trick questions

Candidate Care At The Forefront

Don’t assume free time (Take Home, Personal GitHub, side projects)

Candidate Care At The Forefront

Don't assume prosperity (Personal super high spec laptop)

Candidate Care At The Forefront

Make it relevant

Candidate Care At The Forefront

Make it iterative

Candidate Care At The Forefront

Make tasks clear

Candidate Care At The Forefront

Focus on problem solving

Candidate Care At The Forefront

Help candidates that are stuck

Candidate Care At The Forefront

  • Should be driving factor in everything else that follows
  • Clear expectations—and no trick questions
  • Don’t assume free time (Take Home, Personal GitHub, side projects)
  • Don’t assume prosperity (Personal super high spec laptop)
  • Make it relevant
  • Make it iterative
  • Make tasks clear
  • Focus on problem solving
  • Help candidates that are stuck

Standardisation of Experience

Question library with exemplar answers

+ Shared understanding amongst interviewers

Standardisation of Experience

Clear expectations on what is passing

Standardisation of Experience

Questions to avoid (blocking, leading, closed)

Standardisation of Experience

Realistic conditions (IDE, Google/Stack Overflow, clarifying questions)

Standardisation of Experience

  • Question library with exemplar answers
    • Shared understanding amongst interviewers
  • Clear expectations on what is passing
  • Questions to avoid (blocking, leading, closed)
  • Realistic conditions (IDE, Google/Stack Overflow, clarifying questions)

Candidate Briefing Package

  • Reduces the unknown and therefore reduces stress
  • Equipment / IDE / APIs etc.
  • Any pre-work needed (Minimise this)

Next Steps

Next Step

Audit your entire interview process

If you hit any roadblocks, I'm here to help.

The slide contains two QR codes. The first is labeled "Message me on LinkedIn" and the second is labeled "Book a quick call directly."