How 1 Engineer Hired 6 Top Programmers With Minimum Effort

This is a story how we at Toggl are using automated skill-based testing to hire top developers with little effort. Our mobile technical lead writes about the key elements in his process and gives useful tips on how to build workable tests with Hundred5.

Hiring Top Programmers

Usually I like to write about technical topics, about cool upcoming new features or entirely new apps I am working on. But today I would like take to focus on my second big responsibility here at Toggl: building a team of amazing programmers to make all the other great things possible.

Specifically, I would like to talk about how over the last year I almost singlehandedly hired six great developers with very little effort.

I started out with very little experience in hiring and while I still would not consider myself an expert, I learned a lot over the months that I would like to share here.

The process

While we are very flexible at how we get things done at Toggl, most of our hiring processes look something like the following:

  1. Our initial selection is made using the automated online testing tool Hundred5
  2. Candidates passing the online test get invited to one or two video call interviews
  3. The most promising interviewees are invited to work with us for three to five days, just as if they were part of the team
  4. After a short final interview, those that did well in their test-week get an offer and often start working with us within a few weeks

Depending on your background, this process might on the surface seem very straightforward. Or maybe you think we are crazy for trusting automated tools, never even asking for a candidate’s CV and not bothering to meet people face-to-face before hiring them.

I would like to convince you that this process is not only viable, but very efficient, if done right.

Automated testing with Hundred5

While not the only or maybe even the most important part of our hiring process, Hundred5‘s automated skill-based testing is the key to the efficiency or Toggl’s hiring.

It is the first filter applied to the large pool of potential candidates, and being almost entirely hands-off it can do work equivalent to checking hundreds or thousands of CVs in the blink of an eye.

In fact I think that skill-based testing such as Hundred5 allows us to do is even more valuable than reading CVs for two reasons:

1. CVs are written to present the applicant in the most positive light possible

They focus on the strengths of the candidate, and might leave out important weaknesses. They also require you to trust in the honesty of the applicant, which at this stage in the process is not a given.

An online skill test, evaluated automatically, however, is much harder to fool. You cannot simply exaggerate your skills and experience. Either you get the answers right or you do not. In addition, since you are the one creating the test, you can make sure to ask questions testing the skills and knowledge important to you and the position.

2. You will find candidates who may not have even been looking for a new job

Writing and submitting a CV can be a significant hurdle when applying for a position. Certainly those not looking for a new job will never even bother applying for your position.

By having an online skill-test, we do not only advertise the availability of a position but also tempt people’s curiosity with a quick challenge, and a free T-Shirt when doing well on the test.

This significantly improves your reach by including excellent candidates that might not realise that they want to work for you.

Certainly most of the people I hired took the initial test for fun, for the free shirt or simply out of curiosity.

For these reasons, when it comes to efficiency, I consider automated skill-based tests our most powerful hiring tool. In fact, given the above, Hundred5 replaces not only CVs, but even initial phone screenings, all while finding more candidates than a regular application form ever could.

The hiccup of automated testing

However, all the above hinges on one thing: the quality of your test.

An automated test will always be exactly as good as its questions. Get the questions right, and you will find who you are looking for.

Unfortunately, writing a good test is not easy and requires an initial investment before you can reap the benefits. I hope the following tips, learned over many months while continuously refining and updating our tests to the best of our understanding, will help you get a head start.

How to create a good automated skill-based test

The key aspect of a good skill-based test is in the name: You want to test skills, and you want to be sure you are testing the right ones, which is not always as easy as it seems.

Automated Pre-employment Testing With Hundred5

1. Know what you are testing for and make sure it’s what you need

It is easy to come up with a bunch of questions related to, for example, programming. Just because the question relates however, does not mean it is testing the applicant’s skill as a programmer.

You need to think through the questions, and how you would or could solve them. If it requires the experience and skills you want in a new hire, your question is good.

2. Avoid trivia questions

How tall is the Eiffel Tower? What does KPI mean? What version of iOS runs on the iPhone 5?

What these questions have in common is that the only skill they require is to open a second tab and copying them into Google. Or alternatively, Siri gives straight up answers for the first two, and a useful link for the third.

Unless “looking up things” is a key skill for the position, this kind of question may not be what you want.

3. Ask questions that test understanding

With the availability of search engines, testing pure knowledge is almost impossible. What is possible however is to test understanding.

It is likely that the position you are hiring for requires dealing with and understanding some complex system or another. For programmers these are pieces of code, larger scale architecture, and the ecosystem an app might be part of. For someone in marketing, this might be a basic understanding of psychology, or knowing how different marketing channels can interact or counteract each other.

As an example, I ask candidates to give me the results of several pieces of code, usually just a few lines each. This tests that they can read code and step through it in their head, while keeping in their head (or written out) the various values of variables. This tests exactly the kind of skill that is one of the requirements for any programming position.

4. Avoid “gotcha questions” and riddles

You want to test the skills the candidate will have to use on a regular basis.

Most gotcha questions and riddles don’t test analytical thinking, but much more whether the candidate has heard the riddle before, or happens to intuitively guess the answer to the question.

Instead you want to ask questions that require basic knowledge in the relevant field, and manners of thinking things through that will actually be required on the job.

5. Ask questions that test analytical thinking

While the importance of soft skills can differ between different positions, analytical thinking and general intelligence is a boon to any line of work.

You could give the candidate a small logical puzzle with a clear solution that can be arrived at by thinking it through attentively, or ask them to choose which of a given list of arguments or facts contradicts a stated position or line of reasoning.

No matter how you do it, you want to _pick the brain_ of the applicants to see if they can understand and solve problems and arrive at the correct or best solution.

Hiring Top Developers With Hundred5

6. Keep it short

A guideline I follow myself is that each question should be answerable in less than two minutes, and that the whole test does not take longer than 20, and I ere on the side of shorter tests.

One reason for this is that a shorter test will be taken by more people, especially if you advertise the short duration ahead of time.

Secondly, I rather ask a few well thought-out questions that tell me a lot about a candidate, than casting my net too wide, and asking too many questions, or questions at which people can get stuck for a long time.

7. Think carefully about the test’s passing grade

I have both seen and tried several approaches when determining a passing percentage, or the number of correct answers to advance to the next round.

The first approach is to set the passing grade at 100%, only allowing those that answer all questions correctly to pass on. While this may seem harsh and unreasonable it can work well for shorter tests that test basic understanding and experience that any candidate needs to have to even be considered. Generally, those answering all such questions correctly are often good candidates.

The second approach is to set a relatively low passing grade (I have gone as low as 50%). To make this approach work, the questions have to be harder however. Applicants are not expected to answer all questions correctly. Instead it is more a question of how many they manage to answer well in the limited time given.

If your test does not quite achieve the right outcome, whether it lets through too many or too few candidates, consider adjusting the adjusting the difficulty and passing grade of the test. But always do so with an understanding of which of the two approaches above you are going for, or come up with your own.

8. Open-ended questions

While much of the above applies generally, there is an inherent problem with open ended questions: they cannot be graded automatically.

For my case of looking for developers this is not a big problem, since the technical skills required here can be tested by objective questions.

However, when looking for a designer, marketing stunt creator or HR ninja the relevant skills may be harder to test for. Even in those cases however, I would encourage you to still test for critical thinking and general understanding. You could also test their views on aspects of work culture to find someone with similar values as you or your company, though you may lose out on otherwise good candidates because of small differences in opinion.

Even for more technical positions, however, there can be a lot of value in the right kind of open question. For example, I ask every applicant to solve a relatively easy programming problem that requires just a couple of lines of code but shows whether the candidate is comfortable writing code. A good candidate is usually able to implement a decent solution in just a minute or two.

While that kind of question cannot be graded automatically, I use it as another manual filtering step: only those that pass the other questions and also submit good code – something I can usually determine at a glance – will receive an interview invitation.

I also used to leave an open question where someone could leave a link to their website, portfolio, GitHub account or similar place to show of their experience and previous work, but I find that this information does not really help me make a decision while taking a lot more time than quickly scanning a piece of code.

However, I can see more value in such open questions for less technical positions, like designers where a good portfolio is key.

9. Test your test

Lastly, but maybe most importantly: make sure you test your test. Show it to people in similar positions to the one you are trying to fill, see how well they do on it, and ask them what they think.

I even ask everyone I interview how they liked the online test, and incorporate their feedback into future versions to make the test and Hundred5 do as much of my work for me as possible.

How to get the most out of automated testing

They real key to unlocking the vast potential of automated skill-based testing, like we do with Hundred5, is to ask the right questions.

Investing a few hours to create a good Hundred5 test can save hundreds of hours of looking for and sorting through candidates.

Using Hundred5 and the processes and tips above I was able to single-handedly and by investing just the bare minimum of time find an entire team of great developers that I am happy to now work with every day. And we continue to hire using the same process with confidence and no intention of making big changes anytime soon.

It took me months before I developed my current understanding of what makes for a good skill-based test. I hope that with this post you can do the same in weeks or days.

If you have questions, other ideas on how to create great automatic skill-based tests, please feel free top get in touch or drop a comment below.


Until then, happy hiring!

October 23, 2017

Related Posts


How We’re Hiring Without Reading a Single CV

This is a simple story of how we made hiring easy using Hundred5.   We killed the CV as a hiring tool in 2013 because after spending a ridiculous amount of time sifting through piles of résumés, we regularly ended up with a lot of hard to compare CVS and no way to make well-reasoned decisions. 


How We Killed the CV as a Hiring Tool and Why We’re Never Going Back

This is a simple story of how we made hiring easy using Hundred5.   Recruitment solutions have not kept pace with the developments in the rest of the modern working culture, and the way companies approach talent search and hiring processes is too inefficient. We know, because we’ve been there.


5 Tips to Personalize One-Way Video Interviews

One-way video interviews save time and energy but can be impersonal and discourage potential candidates if not handled correctly. Illustration: Annelise Capossela Typically screening interviews, used to filter out potential employees, are 15 to 30 minutes long phone calls with qualified applicants. By screening candidates, you can quickly sort through applications and efficiently identify the