The Age Old Tech Interview Argument

You go and find any online discussion about tech interviews and you will find two almost opposing views that cannot be reconciled easily. On the one extreme, the recruiting company run some kind of coding or aptitude test. It might be one large test or exercise or a series of smaller ones. It might take a day or an hour. It is likely to involve something that some people cannot work out and they fail the interview claiming it is not fair. On the other extreme you get self-entitled Developers who think they are basically top 1% and if you want to recruit them, you should not test them, or maybe set a really low bar so they won’t get caught out but don’t worry, we will be just fine at your organisation!

Clearly being too strict with technical testing might mean missing out on someone who would be really good for your business (although you cannot know that in advance). On the other hand, if you don’t test enough, you get people who win jobs because of over-confidence or even dishonesty and who end up providing limited value for probably a lot of salary.

Taking the Pieces Apart

Something which is surprisingly lacking in many of these heated debates is the acknowledgement that there are many variables involved in recruitment and you cannot simplify it down to a single formula.

For example, I suspect that Google has many 1000s of applications per day for their Development roles and many of these probably have computer science degrees, great skills and possibly some good experience. On paper, they all look good but Google only have space to recruit, say, 100 engineers per month. What do they do? Whatever they can do to filter out candidates. Messy or incomplete CVs. Badly answered filtering questions. Lack of desire to attend interviews or whatever. Taken in isolation, these are not good ways of filtering bad people out but from Google’s perspective, it is better than attempting to interview 1000s of people per day just in case one of those bad CVs with lots of typos and badly answered questions might be the next Guido van Rossum or Linus Torvalds.

But that doesn’t matter, there is no point discussing these things if you are not in the same position as Google/Meta/Amazon etc. who are offering top 0.05% salaries to the very best engineers in the world.

The second variable to understand is the disconnect between what you know as the candidate and what the recruiter knows about both you and the job. You cannot expect the recruiter to invest a significant amount of time in you just because you know that you are worth employing. All they know about is what you have told them and what they have read in perhaps 5 minutes. That’s it. That thing you mentioned on your CV means nothing to them, they don’t know how hard it was to produce, how many hours you put in and, more importantly, how many hours a good engineer would have needed to produce the same thing. When you tell them you are at Senior level, they don’t know that and, to be honest, you probably don’t either. I have been programming for 30+ years but where am I in the world of programmers between 1 and 100? No idea! I’ve been able to accomplish many things and have never been unable to produce what I have been asked but on the other hand, maybe my tasks have never been hard enough!? The recruiter has to do something to close the gap between what you might know and what they need to be sure of.

A third variable is the knowledge that not all people are honest. Some are deliberately dishonest and will say anything for the role. Some are deluded and simply think they are really good. Some people genuinely think they are better than they are and simply lack the awareness to know that. I spoke to one guy who was 2 years out of college and who was applying for a Senior Developer role. I asked him some of the simple “warm up” questions that help people feel comfortable answering questions like, “What is the difference between overloading and overriding?” and “What is the difference between a class and a struct in C#?”. These aren’t noob questions but they are certainly something a Senior Developer would know or at least have a good stab at. This guy didn’t know the answer to any of them! I don’t usually finish interviews early but I did in this case and tried to politely point out that I thought he was being unrealistic to consider himself a Senior Developer when so young and inexperienced and most importantly, not actually knowing anything. He argued with me about that :-)

A fourth variable is that roles are very different and have different needs. If I need someone to work in the Cloudflare networking team, they are clearly going to need to be very familiar with the Linux networking stack and it shouldn’t come as a surprise if they are tested specifically on that. They might also be applying for a high paid role,one with good perks, so they might accept the effort required to do so. Another business is paying bottom dollar for people working on simple internal CRUD applications, as long as you can spell your name and know Visual Studio, you are in! Clearly, the approach for this company is likely to be very different. Likewise if you are going to part of a large team, the risk is much lower if you cannot make the grade and have to be let-go after 2 months, it won’t be the end of the world.

Getting to the Sweet Spot

So, where is the sweet spot in effective interviews? How do we balance the need to verify what people claim about their abilities but also not putting too many high barriers between a potential engineer and the role we want them to fill?

Firstly, screening is really key to avoid taking up your previous time as a Developer or Manager in recruitment. It is depressing interviewing completely the wrong person because you want to be polite and not end the call after 5 minutes even if you already know they are entirely unsuitable. We advertised a Development role once and someone who applied worked in a mobile phone shop but was interested in becoming a programmer despite the job role clearly saying that we needed experience!

There are a number of screening techniques that depend on the platform you are using. The simplest is just filtering questions like, “Do you have a minimum of 5 years commercial Development experience”. If they say no to any of the questions (or yes if they are worded the other way round!) then you automatically tell them that they are not suitable and they have only wasted their own time. What if they lie? Well you can’t stop people from lying but there are still checks you can make, for example, their CV might only go back 3 years. It might be a mistake on their part but if I saw that, immediately rejected, assuming they are not being truthful. If they say, “yes” and then in their covering letter say, “I have worked for 3 years in a company but before that I used to write my own software and sell it to my friends”, great! You can take a view on that but they have earned your trust.

Talking of which, a covering letter, to me, is a must. Firstly, I want people to realise they need to work. If someone can’t be bothered to spend 5 minutes writing me a letter about why they want to work at my company, do they really have a good work ethic? If on the other hand, they are not doing it because they are just blanket applying for jobs, I am also not interested. My company has a culture, a passion, a motivation etc. and I don’t want someone working here just because I am paying them. I want them to work here because they want to work here (and they add value). It doesn’t have to be long but it does give a chance for someone to highlight anything that is not clear from the CV e.g. “The 6 month gap was when I returned to India to look after my dad” and actually gives them an advantage over people who don’t want to write a letter.

You can get AI screening but to be honest, I don’t think CVs are good enough most of the time for AI to be effective so maybe use it to highlight stuff quickly like, “Shows significant experience with C#” but don’t use it as an automatic means to rejection.

You can then use in-house or third-party recruiters to do other types of screening, again, based no your company’s needs. Some businesses need a specific mindset and require a psychological test. I did one of these once and then didn’t get the job. I don’t know why and I think they should have given me some feedback even if it was something I didn’t want to hear like, “your profile has you as a risk taker and we felt this wasn’t the right fit for this role” but don’t use these for the sake of it. They are subtle and require training to understand but if you need to, you can.

Other types of screening might be as simple as, can they communicate clearly. Did they dress respectfully and turn up on time (or at least ask what they should wear). Do they know anything about the company whose time they are now taking up? When people say, “no”, they get pushed very close to the exit in my mind, again, because if you are offering yourself for a role with all of the cost and time but don’t know anything about the business, do you even know if you want to work here? These types of screenings can be done by anyone, and ideally can be done by different types of people. Why not have someone who is not a Developer seeing whether they like a candidate and can have a conversation with them.

Technical tests

I have previously used technical screening tests. These are simple online tests that the candidate does in their own time but take up to 1 hour. It might be something like “write some code that counts the number of even numbers in this array”. You might think these would be pointless but I have had people who either cannot do this or inexplicably take like 10 minutes to do something simple like count the number of words in this string (literally “return myString.Split(‘ ‘).Length”). This can tell you a lot. The good people finish them in 3 minutes. Could they cheat? Sure but you can play back the recording of them interacting with it and see them possibly fiddling constantly with code and not getting it. This puts some people off but you know what? A lot of what we do is online and a lot of what we do involves written instructions. If you cannot handle that, sorry but no thanks.

The real scret of the effective technical test is to ask about 5 quite high-level and open-ended questions about 5 different areas of the work that person will be doing. If they are genuinely experienced, they already know what these things mean so you don’t need to explain them, just to see how deep their understanding goes, how quickly they can recall what they know and how clearly they can articulate it.

When I do these interviews, it only takes me 30 minutes and I know whether they are the right level or not. I am not looking to tell a 98 from a 99 out of 100. Just to know that you have said you are experienced and you understand stuff and you have proved it.

Example Technical Question

So for example: “Imagine you are tasked with finding out why a database query is running slowly. What is your approach to working this out and what tools do you use to identify the problem?”. Obviously you might make allowances for the fact they have only used Postgresql and not SQL Server so the exact answer might vary but still, you can tell how much they know from their answer.

Bad Answers (I have heard these!)

  • “I don’t know, I never worked on databases”. Yes you did because you are a web developer and you should have learned about them to be effective.
  • “I would look at the code and try and run through it and verify”. No. I already told you it was the query that was slow. If that wasn’t clear, you should have asked me whether you can assume it is definitely the query
  • “It is probably slow because of indexes”. Nope. That is both a massive presumption and by not having any process to speak of, you clearly do not understand how to debug a slow query.
  • “Entity Framework could be pulling back related tables”. Yes I know. How does that answer the question?
  • “If it was slow I would just add some indexes”. This is sometimes the solution but how would you know whether these are needed and how would you know which ones to add? Would you just eat up disk space in the hope that you might just do something that works - for now anyway?

Minimum knowledge required

  • “I would find out the actual query being run”. Great, that is indeed the first step. Although if you said this, I would ask how!
  • “I would use the query analyzer to see what is slow”. Also important and correct. The follow-up here would be, what would you actually see in the query analyzer that would highlight something is slow?

The Full Answer

  • “I would use the SQL profiler to isolate the specific query that is slow. This will also confirm that is indeed one query and not possibly a number of queries that we might not be aware of that are all a bit slow and which add up to the overall performance issue”
  • “I would then run the query in the query analyzer where it will show a percentage cost on each element of the query, enabling me to identify which part of the query is likely causing the problem. This will also show which columns are being filtered on and which are being selected from which might give me a clue as to whether there is an index problem”

Extra credit

  • “The SQL profiler shows levels of IO so if these are high, it is likely that a table scan(s) is being carried out”
  • “The SQL profiler sometimes shows that we are selecting more columns than it appears from the Linq-to-SQL/ActiveRecord query, which could highlight why it won’t use an index that is already present”
  • “In most cases, a slow part of a query is where a scan is taking place instead of a seek or where a lot of key lookups are being performed. Using the filtered and selected columns, I would evaluate existing indexes to see whether a small modification could provide what we need without using up more disk space and performance or whether another index might be needed”
  • “It is sometimes possible to use a filtered index to significantly reduce disk space required but still get the benefits of an index for query speed”
  • “Table statistics can sometimes affect the ability of the query optimizer to select the correct execution plan so we might need to look at adding additional statistics or being more clever with data distribution, adding redundant where terms etc. to help it out”

So what starts as a simple open-ended technical question can easily determine whether the applicant really knows their stuff, or whether they are reaching for terms they might have heard but don’t really understand.

If you are worried about answering questions with AI, it is harder than you think for people to do that naturally at the right speed and without making their face look like they are reading something or typing.

This one question wouldn’t determine whether someone gets though this stage but there might be a question on .Net or on Visual Studio or on your approach to work etc. each of them open-ended, each of them an opportunity to demonstrate a lack of deep expertise or someone who really knows what they are talking about, who can ask sensible follow-up questions and who knows when they have been talking too long and asks, “is that enough or do you want me to explain any more”!

Self Awareness

Something I also used to do before my HR team harmonized the process was to ask the candidate before they arrived to the interview to rate themselves from 0 to 5 across a number of relevant technologies like Cloud, Databases, .Net, HTML, Vue JS etc. I tell them what 0 to 5 mean. This serves 2 purposes.

Firstly, if they rate themselves as e.g. a 1 in databases, what’s the point in asking about them in the interview?

Secondly, I am less worried about whether they are a 4 or a 5 in something but if they say they are a 5 (knows the whole domain i.e. an expert) and I ask them a tough but not impossible question, they should be able to come out with a pretty-good answer even if not perfect. If I ask them the question above and they can only answer it in a really basic way, I can immediately know that a) they are not much of an expert and b) they think they are!

When using this method, which I didn’t do for Juniors, I would have a rough expectation that Seniors might have at up to 6 ‘4’s and ‘5’s but likely no more than 2 ‘5’s. If they thought they had a lot more than that, it is also a signal. I have had people just assume that 20 years experience = a 5 in CSS and when you ask them about flexbox, they don’t really know anything about it. That puts you right down in the 2 bracket but you assumed you were a 5?

What I like about this system, and I will try and bring it back, is that it places some responsibility on the candidate to propose who they are and then the interview is really just confirming that. If they can’t answer the questions then you are rejecting them on the basis of, “well you said you were a 5 and couldn’t answer these questions confidently”, which seems fairer than the alternative of, “I asked you some random questions and you didn’t happen to know the answers so you don’t get the job”, which is how people often relate the experience when they don’t make the grade in their Google application.

It is not an exact science of course and I would never look for perfect knowledge but most of the time people are either clearly in the right ballpark (i.e. they are self-aware) or obviously nowhere near it.

The Take Home Test

I have only done this once for a specific candidate who had been a Development Manager, hadn’t enjoyed it that much and wanted to go back to being a Senior Developer. The Management experience can be really helpful because it shows business awareness, probably organisation skills, self-motivating etc. but I wasn’t convinced that he still had the coding chops because he had not been programming for about 5 years and his answers to my technical questions were not massively convincing. I proposed the take-home test.

This is a reasonable piece of work based on something they are likely to do at your company. I created a separate backlog and git repo, which I gave him access to, I set him the task telling him it should take no longer than 1 day to code but he had a week to complete it.

We also paid him, I can’t remember how much but a normal kind of contract rate. Why? Because why should he do it otherwise for free?

I was able to judge not only the final output in terms of code quality etc. (it wasn’t great) but also, did he ask me any questions (he did not!), did he make small commits and ask me to review them (he also did not). So I was able to determine that not only was his code not of a suitable level but he didn’t make up for it in his ability to collaborate.

I had a friendly conversation afterwards and pointed out the ways in which the code did not meet the standards I was looking for. How he had missed that some of the functionality already existed in libraries he could have used, how he lacked Unit Tests and possibly because of that, there were a number of edge cases that were missing.

He thanked me for the opportunity and for paying him for it and for the chance to get an objective feel for how rusty he was!

Conclusion

So the secret?

  • Screening effectively
  • Asking a handful of different open-ended technical questions that allow someone to demonstrate their expertise and communication skills
  • (Optional) Get them to rank themselves both as a skills indeicator and also to find out self-awareness
  • (Optional) Take home test if you really want to see what they will actually do on the job.

Good luck. Recruitment is hard!