Strategic Hiring: How To Build a Screening Test for a Conversion Rate Optimizer Position

Illustration of a character holding a UI component

This is a story of how we built a screening test for the applicants of the Conversion Rate Optimizer position with Hundred5.


There’s a lot to consider when hiring someone to work on improving your product.

A good conversion rate optimizer is a lot of things. They’ll question the status quo and push your limitations. They understand the meaning behind what’s being said. They’re strategic yet detail oriented; creative and gets things done. Oh, and they are also analytical while being good with words.

Basically, you’re looking for a unicorn.

Here’s the million dollar question, though: How do you separate the unicorns from the crowd? Let me show you how I did it.

Step 1: Figure out who you’re looking for

It might seem simple enough, but clarifying this is the single most important step to start off on the right foot. These are some of the things I considered before taking any further steps:

  • Which skills do I need in my team in order to reach our goals?
  • Which skills are currently missing in my team?
  • What type of personality would best compliment my existing team?

Step 2: Prioritize

Once you’ve got your initial parameters in place, it’s time to prioritize them. What’s absolutely non-negotiable? What are you willing to compromise on?

In an ideal world, we’d get all that we’re asking for with sprinkles on top, but the reality might not be quite as cut and dry.

A few things to consider.

The mindset and personality is usually not something that will change. Skills, however, can be learned. The question is whether you have the time and resources needed to train someone, and how much effort you’re willing to put in.

In my experience: experience is good, but the right mindset is golden. There’s no limit to what someone with the right mentality and work ethic can accomplish, given the right support, environment and opportunities.

Step 3: Come up with a strategy

Now that you’ve figured out who you’re looking for, it’s time to put together a plan on how to recognize that perfect candidate.

This is where Hundred5 comes in. Simply put, it allows you to test candidates and eliminate anyone who doesn’t fit the bill. It automates the initial screening process, saving you an infinite amount of time.

The key at this stage is being strategic about which questions to include. In other words: It’s time to brainstorm.

Try to come up with as many ways as you can to test for each specific requirement you listed. I ended up with three main categories I needed my candidate to be good at:

  1. Knowledge of and experience in conversion rate optimization (terminology, methodologies, understanding basic user psychology, metrics)
  2. Copywriting
  3. Critical and analytical thinking

Think what you could ask that would show whether or not someone fits the bill. For example, to know if someone has experience in CRO, I asked this questions:

Which of the following would tell you if you have a problem with getting users to stick around?

  • LTV
  • Retention
  • Cohort analysis
  • CAC
  • Avg. MRR
  • NPS

Here I used the abbreviations as a secondary filter. If a candidate knows the meaning of the terms, but is not familiar with the abbreviations, they can look these up easily. However, if the applicant is not familiar with either of them, it’ll likely take them too long to find the right answers.

When coming up with these questions, try to think outside of the box.

For example, if you’re looking for someone who is detail oriented, you might add a harder-to-notice detail or a twist to one of the other questions to see if the candidates catch it.

If working well under pressure is important, make sure that the time limit for finishing the test is tight.

If you’re looking for an out-of-box thinker, give them an opportunity to do so: open-ended questions are good for that.


Read on: 100 Soft Skills Questions To Help You Hire Top Talent


Step 4: Narrow it down

Now that you’ve got a bunch of questions, start selecting the strongest contenders from each category. This might be a good time to bring in a fresh pair of eyes.

These are some of the questions I ended up with.

To test for critical thinking skills, I used a simple riddle:

Which of these statements is true?

  • The number of false statements here is one
  • The number of false statements here is two
  • The number of false statements here is three
  • The number of false statements here is four

One of the important things for me was making sure that the general approach and priorities of the candidate match those in our team. Here’s an example of what I used to determine that:

What’s the single most important thing to communicate to a new user trialling a web service?

  • All the possibilities the service has to offer
  • How the service will help them achieve their goals
  • The most popular features
  • How others are using the product

There are no definitive answers here. You could probably argue your case for most of these, depending on the circumstance. However, the 2nd answer is the best match to our approach to onboarding and would tell me if the applicant has a similar mindset.

I also wanted to see the candidates’ reasoning skills, so I added this question:

Which part of the conversion funnel has most potential to attain quick growth? Why?

I didn’t have a “correct answer” in mind for this question. I simply wanted to see the candidate’s thought process and how they would back it up.

One of the things I’ve found out during this process is that testing copywriting skills with this type of test is challenging. I’ve used mostly free form questions to assess writing skills, for example:

Create a short copy for a landing page with a CTA.

Purpose: Convince people to sign up for an app’s beta release
The app: GoExplore, an imaginatory fitness app that counts your steps.
Length: up to 165 characters

I was hoping to see if the candidate had any knowledge of persuasive copywriting, but mostly the results were pretty average.

It did, however, show if a candidate wasn’t fluent in English as well as whether they were able to incorporate all the details that were mentioned: it being a beta release; the copy going on a landing page, not inside a mobile app, etc.

Surprisingly, some candidates even flunked out because they went over the character limit – an indicator that they’re either not attentive enough or simply bad at following instructions.

I also tried a multiple choice question to see how strategically the applicants would approach copywriting:

Which one of these CTAs would you use in this pop-up?

  • Keep me informed
  • Continue
  • Get started now
  • Yes, I want to become a marketing expert

I was looking for a candidate that would have a similar approach to copywriting that we do in our team. It is not black and white, and there are no absolute right or wrong answers.

However, if someone chose the longest (although very specific) option, it would tell me that for them, the length isn’t important. Or, if they chose the generic “Continue”, being specific and purposeful with copy isn’t high on their list.


Read on: How to Hire Backend Developers

The last category of questions were those that covered skills and knowledge relating to the CRO field. The challenge was making sure that the answers couldn’t be googled easily.

I tried to make the questions rather challenging than easy to allow the strongest talents to shine. This was one of the questions I used:

When does it make sense to employ a multivariate instead of an A/B test?

  • When there are at least 5 different segments in both sample groups
  • When you’re dealing with a high traffic site
  • When you need test results faster
  • When testing to improve conversions on landing pages

Some of these answer options I literally thought up on the spot, using words that sounded like that sentence might be from the A/B testing vocabulary. The only goal being to confuse those that don’t know their stuff. It worked.

The same goes for the next question:

When do you stop an A/B test? (Select all that apply)

  • When the sample groups have reached the minimum predetermined size
  • When the test has reached statistical significance
  • When the test has run for at least one business cycle
  • When conversions reach your target goal
  • When the difference between the two sample groups in the end result is at least 5%
  • When the test has run for 3 full weeks

As the last question I asked to share any previous work. This proved to be great, giving me much more insight into the person’s interests and skills. Some would share blog posts or articles they’d put together, others their design portfolio or photo libraries.

It’s that additional bit of information that can help you make the choice between two great candidates.

Make sure that the test is on the shorter side – 10-15 questions is a good target. It’s better to have fewer questions, but ones that are really well thought out. You’ll likely get more interest with a shorter test, too.

Lastly, a few tips that I’ve found to be true

  • Be wary of know-it-all’s. Confidence is good, but ultimately you want someone you can work together, not butt heads with. A good CRO is always willing to learn and adjust their beliefs.
  • Don’t be afraid to hire someone smarter than you. In fact, you should be looking to hire someone smarter than you. This is what makes a strong team, and surrounding yourself with experts is a mark of a strong leader.
  • Look for a team player. Working on optimizing conversion rates and improving the product is never a solitary task. If the candidate is proving challenging to communicate with already in the hiring stages, you can bet it’ll translate to their work life as well.

If you’re looking for inspiration, this article helped me quite a bit. 

And if you’re interested in the whole test-first recruitment process, you can keep an eye on the Live Blog about how Hundred5 team is hiring an SEO Manager without a single CV.

January 16, 2018