Get in touchWhat to do when life gives you unsatisfactory participants. Maia explains in the third article of our Research Trail Guide series.
At Muir Wood & Co, we help product teams run research projects. Getting a duff participant can be a real bummer on an important study. So for our third edition of our Research Trail Guide, here’s Maia on how to spot 'em, deal with 'em and avoid 'em.
The interview began with an innocent bit of small talk. I was speaking to someone who had said on their screener that they lived in California and so I knew it would be pretty early in the morning for them. ‘What time is it where you are?’ His answer immediately raised a big internal eyebrow for me. He was off by 3 hours!
Things continued to unravel as I asked him the usual ‘getting to know you’ questions. Something wasn’t right, but I was really questioning my intuition at the same time. Next up he needed to share his screen to recreate a task and I was sure I was being scammed when his recent Google searches included ‘What time is it in New York?’ This was the time he’d given me, he was clearly a long way from his supposed home.
What followed was one of the more painful half hours of my life, during which he scrabbled around online trying to recreate a task that he’d definitely never done before. I tried to get him to give up the game, without straight up calling him out. He got more and more frustrated; the interview wasn’t the plain sail he’d been expecting either. I eventually managed to end the call, by which time we were both more than a bit sweaty.
This type of person is what we call a Lemon [def: an unsatisfactory participant]. They waste our time, client money and can skew or bias our data.
It was my first true lemon experience. I could tell it made me a little bit more wary in the interviews that followed and I made an effort to return to my prior open disposition.
Lemons are rare and it’s important to treat participants as the majority are - regular humans who need to be put at ease, not on trial, as they open up about their experiences. Also, who can really blame people for wanting to put their acting skills to use to make some money. It’s part of our job to make sure they never get the chance.
There are a few different types of lemons. Some are more damaging to your research than others and some might even be your fault…
An interviewee who will lie about who they are to fit your requirements and get that sweet, sweet cash. All their answers are vague, things just don't add up. They often start talking immediately about the topic of the interview.
They might not be lying, but you get the feeling they are churning through the interviews. Their browser has bookmarks for all the research platforms. We sometimes specifically ask for people who haven't participated in research in the last 6 months.
Monosyllabic answers from someone who’d really rather be anywhere else is quite different from the introvert who just needs some help warming up.
You’ll know you’ve got one when you get that mid-interview sinking feeling that you didn’t get the brief quite right. Just not the right fit, this one is probably on you for not being specific enough in your screening.
Well, would you look at that, you’ve answered my one question perfectly so you can go now!
When we’re interviewing remotely we ultimately have the power to end the call. So once you’ve explained the situation clearly and stated you’re going to end the call, do it. Nothing nice about it, but it’s better than spending half an hour perspiring like me.
This is inevitably trickier because you’re face to face with someone who has gone out of their way to be there. But remember, your time is valuable too.
Once you’ve decided you’re going to end the interview you need to clearly explain why - how they’re in breach of the agreement you had or why they’re not suitable, as well as what you’re going to do about payment. And then you need to stick to this, this isn’t the time for negotiations.
Most people aren’t actually that bothered about having an interview finish early, what they do care about is whether they’re going to get paid.
Our rule is that if the mistake was ours, in not being specific enough in screening, payment is absolutely still due. We don’t like to be taken for mugs, however, or have our clients taken for them. So if we’re being actively deceived it’s perfectly reasonable to withhold payment.
If the lemon was the result of a screening error it’s time to tweak your screener, or get in touch with upcoming recruits with the necessary follow up questions to check they’re still a good fit.
In terms of the data, our rule is that if you’ve paid for their time then use it if you can, it might help you define an anti-persona for example. If you didn’t pay however, or if they’re a scammer and everything they said was rubbish, it’s time to delete their data.
The best way to deal with lemons is to avoid ever having to speak to them. Here are a few ways to keep them out of your calendar.
But ultimately all sources are prone to scammers, this is why effective screening is so important.
Screening Research Trail Guide coming soon, but here are a few nuggets to help:
1. Filter out people who don't fit by cascading questions
‘Do you eat breakfast?’ should filter out the people who don’t break their fast til lunch and the people who never eat granola, before you ask ‘How often do you eat granola for breakfast?’
2. Don't give away what you're looking for
For example ‘What do you have for breakfast?’ instead of ‘Do you like granola?
3. Use open field questions (or screening calls) to find the best communicators
Not only will this put off the lazier scammers, but it’ll mean that if you need to end an interview you can refer to the expectations set out from the start. For example "You’ll need to show me your granola cupboard!"
Don't beat yourself up if one slips through! Even us wizened old pros still get them. Plan for it to happen and make sure everyone else understands this reality.
There you have it – some examples of the lemons we encounter and how we handle them. So, make sure your screening is clear and unambiguous, be firm and decisive when you think you're interviewing a lemon and don't beat yourself up if you get one!
Handling recruitment is one of the hardest parts of research, so if you’re finding it daunting we can help your team to think through recruitment and coach them on challenging scenarios like these.