Modern Work

Five Lessons I’ve Learned Doing User Onboarding

Image source: xkcd

The aim of user onboarding is simple – keep your new users coming back to your service. The art itself – not so simple. Here are five crucial lessons I’ve learned about the job while doing onboarding for the Toggl time tracking service.

My job is to make sure that new Toggl users get a good experience with the service when they sign up. It basically boils down to running A/B tests (control group versus experiment group), measuring their success, and running more tests. Successful experiments get launched to everyone, become a part of the control group, and ultimately get experimented on later.

I’ve always been used to being that straight-A student who always knows what t-s to cross and which i-s to dot in order to get my “A”. I was in for a rough surprise with user onboarding, though. Before long I found that:

  1. I had no idea what I had to do in order to get that “A”
  2. When I started trying, I just kept failing. A lot.

I have never failed an exam in my life, so you can imagine how frustrated I was by my predicament. This experience has pretty much turned everything I thought I knew about the world and the internet on its head. I wish I’d known when I started that it was all about the…

1. Metrics, metrics, metrics

The first wall I hit was what qualifies users as ‘active’? How do we measure this? What’s the percentage of users staying active now? What is my measure for success? When doing tests, how do we assign users to either experiment or control groups? What qualifies an experiment as a success?

I started with the simplest denominator – if a user has created a time entry, they’re active, i.e. I’ve succeeded. I soon realised though, that a whole bunch of users don’t ever make a time entry – they just use Toggl to run reports of their team’s time. So we needed to find a way to factor in that criteria for activity. And what if a manager doesn’t use the time tracker at all, but still actively encourages his staff to use it? Criteria are never as simple as they first seem, and defining them is absolutely crucial.

It took us over a year before we finally got to metrics that we can trust (well, with 90% certainty at least). While I can now trust my numbers to give me a good indication of a situation, I must always be ready to challenge them. Every time I run a new experiment, I have to check that the numbers are coming in and that they’re coming in the way they should. Things go wrong – randomly, for different reasons in different situations – but they regularly do. To cope with that, you have to know your metrics well enough to trust them, but also well enough to notice when they don’t seem quite right.

2. Be very, very careful with case studies

I regularly read e-mails from new users to get ideas for experiments. I also read articles on growth hacking and user onboarding (not that there’s too many of those around). New users often ask things like “How do others use Toggl?” or “How have companies similar to us set it up to work for them?” The articles in turn keep repeating one thing, over and over – “Use case studies!” So I thought I’d give them a go.

This was the test that followed. Half of the new users who signed up for Toggl, saw this for their first nine days using our service:

Case studies

I’d determined three main use cases for Toggl and wrote up a list of different features that I know have been very beneficial in those cases, and why. Each case was clickable on its own, or you could hit “Read more” and see them all. It took me a while to get all this down, and I was very proud and excited when the experiment launched. I set off tracking all the clicks and activity on the test.

The results were absolutely devastating. Firstly, users who saw this version of the timer page stayed less active than those who didn’t (i.e. the control group). And what’s worse – users who actually clicked on any of it, stayed considerably less active than those who just closed the whole thing (not to mention those who never even saw it).

In short – the case studies actually drove people away from our service. What may have happened was that people read about one way of setting Toggl up, saw that this way didn’t exactly match their unique needs and instead of trying a different setup, they simply went off looking for another service elsewhere. This experiment still ranks as the worst in all the 30 that have been completed so far.

The lesson is – be very, very careful with offering case studies as examples. People might think they want them, but my experience says that they can actually do more harm than good.

3. The craziest smallest things can have a huge effect

Some of the experiments I do require a lot of effort and time (both from me and the developers) to get launched. One such experiment was an importing wizard that we hoped would help users to add their teams to Toggl quickly and easily:

Import team

It was a 5-step wizard that took one of our developers three days to code (plus my testing time, changing stuff etc.). Within two days of going live, it was fairly evident that the test was doomed. The experiment ran for five days and in the end – in terms of results – it turned out to be my second biggest failure (right after that ill-fated case studies experiment above) ever. At that point, I’d been running A/B tests for about a year, and had had one semi-success during that time. I was getting really, really frustrated – all the effort we had put in those tests seemed wasted.

So we felt we had to take a break and just try something really small for a change – play around with button texts, change some placeholders, just turn the whole thing around and refresh. One of those small experiments included changing the Start button text to “Go!” for new users for their first 9 days. The change was literally just this:

Go button

… and it worked! I don’t have a convincing reason for why it worked – maybe the exclamation mark and short command were more exciting, maybe there was no rational explanation at all. But one thing that was clear from the start was that more users were staying active when they saw “Go!” than those who saw “Start”. I was completely baffled that such a small change could have such a palpable impact on activity, but the numbers were solid. I had over 2,700 users in both the experiment and control groups. And I trusted my numbers.

At long last, a year into working on customer onboarding, that moment came:

Hell Yeah!

4. Social media can work

Having played around with tiny changes for a while (and yes, there were failures there as well), it was time to go for something more complicated again. I decided (though somewhat hesitantly) to try what effect social media might have on user activity. We added a badge to the site that promised productivity-related content from us on Twitter. The badge led to a popup displaying Toggl’s real-time twitter feed. It was a gamble, because we can’t really control the feed entirely – our users determine most of its content. While they share a lot of joy about using Toggl, they also ask support-related questions (often when things are broken) and occasionally even express utter frustration. But I had planned on adding actual “#productivity” content there, so we decided to take the risk.

Half of our new users then saw a variation (depending on what was in the feed at the time) of this when they opened the “Follow for productivity” badge:

Twitter experiment

The experiment went live at the end of the week and I actually didn’t get the chance to schedule the “#productivity” content into our feed until the following Monday. It ended up being a good thing, though, because it made the metrics all the more interesting.

Before posting actual productivity-related content in our Twitter, we did get a couple more followers per day than usual, but user activity in the experiment group didn’t really change compared to the control group. However, once I’d scheduled 3-4 tweets per day containing productivity quotes, articles and pictures, things started happening – namely, the average for new followers per day more than doubled. Even more importantly, a significantly larger amount of users were staying active in the experiment group than in the control group. In the end, the overall change turned out to be the biggest positive change I’d seen in any of the experiments that we’d run so far.

In retrospect, it makes sense that when you deliver on your promise to provide your users with useful content, more of them will stay active. Plus, once they follow you on Twitter you get an extra way for reminding them of your existence (and they’re less likely to forget about you).

The main lesson for me personally though, was this – using social media channels doesn’t necessarily have to mean spamming your users. If you provide actual beneficial content through your social media, your followers won’t grow annoyed with you and will, in fact, be more likely to come back to your site later. Makes total sense, right? (So should the case studies of course, but go figure..)

5. Be prepared to kill your darlings

The most difficult (even painful) lesson for me has been accepting to let some things go. Understandably, the more time I’d spend on planning, developing and testing a feature (wizards and videos, for example, took a huge amount of time to create), the harder it was to cut it when the numbers went bad. I would think: “I worked so hard on it, maybe if I gave it just a little more time, it would improve. Surely it would?” It won’t.

Little Britain (

I usually run an experiment for a maximum of two weeks (that puts around 5,000 users into both the experiment and control groups), but with some it becomes evident in the first 4-5 days that it’s not going to work out. By that time, usually each group would have almost 2,000 users, so the numbers are pretty reliable. And if they say it’s not going to work, you just have to let it go.

The point is this – if the computer says “no”, kill your darling and do it quick. Because the next amazing thing could be right around the corner and the longer you let the failing experiment run, the more time you’ll lose in letting all these potential active users go.

The gist:

  1. Know your metrics, and always be ready to challenge them.
  2. Be very careful with case studies, your users don’t necessarily know what they want.
  3. Try small changes – small things can lead to big success.
  4. Don’t be afraid of social media – if you make it useful, your users will appreciate it.
  5. Learn to fail quickly.

If you want to learn how to be better at getting new users to stick around, Samuel Hulick does great teardowns on user onboarding. Also check out the Appcues Academy for more general information on what user onboarding is and how to define it for your company.

Have a tip for your fellow onboarding people? Give back, and share it in the comments!

November 25, 2014