As someone who designs
questions for research, that’s not really what I want to hear. But I do. And
sometimes expressed in more forceful terms than this too.
Most of the time, it’s in
surveys. I can chalk it off as the reaction of someone who is too short on time
to give a considered answer. But just occasionally, I’ve had it in focus groups
or interviews. And then you have to ask whether the question you’ve written is
as accessible as it should be? Does it make sense to the person you’re asking
it of? What help do they need to break down how they feel about their employer
into more manageable chunks?
In a focus group or
interview, you’ve got the chance to qualify the question and help people into
it. In a survey they’re on their own, so you have to be careful. So let’s look at a current
question from Linked In’s Inspiration Index:It’s a single question: “How
often do you feel inspired by your work?” And it has some issues, that I’m sure
they must have considered carefully:
- “Inspired”: is that the right word? We can see why they avoid “engaged”- that’s too hard to define. But why not “happy” or “satisfied”?
- “Inspired”: what do they mean? Those days when you genuinely feel the love for your boss, your clients and your work? Or just the days when you “get it”?
- “Inspired”: does it matter? Do you need to be inspired in your work, any more than you need to be inspired when you’re socialising, being a parent, or just doing the dishes?
- The rating system. There’s a sliding scale from “Never feel inspired” to “Always feel inspired”, with no points in between. How does my shifting the slider “a bit” compare to the next man’s?
- The rating system. There’s no control, or comparison. It’s an absolute scale, but no definition of what, say, 73% means.
- The timing. What’s happened that morning is likely to have an effect on my response. Will I be asked again?
But honestly, I don’t think
it’s a stupid question because, what they’re looking to find out is clear
enough. They’ve gone for one question to maximise response at the expense of detail,
but they know a lot about the people responding anyway. In aggregate - and there’s
over 100,000 responses already - the data will become fairly reliable. Potential
problems with the rating system and timing will likely be ironed out in the
bell-curve. And there’s standard practice to deal with the bulk response and
the outliers.
What I am saying is that
even to design a one-question survey, you need to consider a lot of things. So
get someone who does it a lot to help you.
