“That’s a stupid question”

As someone who designs questions for research, that’s not really what I want to hear. But I do. And sometimes expressed in more forceful terms than this too.

Most of the time, it’s in surveys. I can chalk it off as the reaction of someone who is too short on time to give a considered answer. But just occasionally, I’ve had it in focus groups or interviews. And then you have to ask whether the question you’ve written is as accessible as it should be? Does it make sense to the person you’re asking it of? What help do they need to break down how they feel about their employer into more manageable chunks?

In a focus group or interview, you’ve got the chance to qualify the question and help people into it. In a survey they’re on their own, so you have to be careful. So let’s look at a current question from Linked In’s Inspiration Index: http://lnkd.in/ii_nus. It’s a single question: “How often do you feel inspired by your work?” And it has some issues, that I’m sure they must have considered carefully:

• “Inspired”: is that the right word? We can see why they avoid “engaged”- that’s too hard to define. But why not “happy” or “satisfied”?
•  “Inspired”: what do they mean? Those days when you genuinely feel the love for your boss, your clients and your work? Or just the days when you “get it”?
• “Inspired”: does it matter? Do you need to be inspired in your work, any more than you need to be inspired when you’re socialising, being a parent, or just doing the dishes?
• The rating system. There’s a sliding scale from “Never feel inspired” to “Always feel inspired”, with no points in between. How does my shifting the slider “a bit” compare to the next man’s?
• The rating system. There’s no control, or comparison. It’s an absolute scale, but no definition of what, say, 73% means.
•  The timing. What’s happened that morning is likely to have an effect on my response. Will I be asked again?

But honestly, I don’t think it’s a stupid question because, what they’re looking to find out is clear enough. They’ve gone for one question to maximise response at the expense of detail, but they know a lot about the people responding anyway. In aggregate - and there’s over 100,000 responses already - the data will become fairly reliable. Potential problems with the rating system and timing will likely be ironed out in the bell-curve. And there’s standard practice to deal with the bulk response and the outliers. 

What I am saying is that even to design a one-question survey, you need to consider a lot of things. So get someone who does it a lot to help you.