“I Wonder About God” & Other Poorly-Designed Questions

Question design is difficult.  Anyone who has run cognitive interviews or simply conducted a focus group has discovered that even the most carefully designed question may be interpreted far afield from its intended meaning.  While qualitative methods give researchers insight on how interpretations of a question vary (and how to better design the question to come closer to the researcher’s objective), the reality is that question design is rarely put to the test and given the scrutiny it deserves.  Time and budget limitations as well as researchers’ overconfidence in their question-design skills typically lead to a hastily crafted and executed questionnaire.

This is a critical problem not only because it transcends mode – question design is an issue in off- and online modes as well as across quantitative and qualitative methods – but, more importantly, it has a direct, potentially negative impact on analysis, which in turn leads to wrong conclusions, which in turn leads end users along a path of misguided next steps.

Of course some poorly-designed questions are intentional, particularly in an election season when partisan politics triumph over sound research design.  A recent highly-public example is a survey conducted by the conservative-leaning Doctor Patient Medical Association among “random doctors.”  While the methodology alone raises concerns, of interest here is the question design and the reported conclusions from question data.  Of particular note is the unexacting question:  “How do current changes in the medical system affect your desire to practice medicine?”  In response, 82.6% of these doctors answered, “Makes me think about quitting.”  Wow, a strong agreement about “quitting,” but quitting because of what?  “Current changes in the medical system”?  That covers a wide territory.  Yet the reported conclusion to this survey result was stated in the media as:  “83% of American physicians have considered leaving their practices over President Barack Obama’s health care reform law” [emphasis added].

Another recent study concerning the use of scientific studies asked, “Overall, how often do you utilize information reported from scientific studies; or participate in activities that contribute to scientific studies?”  This double-barreled question is difficult enough but the answer options were equally confounding:

  • once daily or more
  • once weekly or more
  • once monthly or more
  • etc.

If I do something once a week or more, could I not also do it daily?  If I do something once a month or more, could I not also do it monthly, maybe even daily?

A technique I often use with clients who are insisting on an ill-conceived question wording is to ask them to think forward in time to when we are analyzing the data, to when we are attempting to draw relevant and worthwhile conclusions from survey results.  I simply ask, ‘Will we be able to honestly interpret the response to that question and clearly know what the data is telling us?’  For instance, how would we interpret the results from the survey question, “Do you access the Internet on at least an occasional basis?”  If everyone who answers “yes” is considered an “Internet user,” what does that really mean since we know nothing about how these people differ in terms of their interpretation of the word “occasional” and their meaning of “access” (e.g., I can access the Internet without being engaged by simply downloading email on my iPhone)?  So, can I really say anything about “Internet users” based on this survey question given that I really don’t know anything about their use or how segmented or unified the group is on key usage indicators?

In “The God Survey” from SurveyMonkey, the first question is, “I wonder about God…A lot, A little, Rarely, Never.”  While this is a potentially interesting question, it is difficult to analyze because it is impossible to know what the respondent was actually thinking about when answering.  As the respondent, I can only speculate what the researcher wants me to wonder about.  Do I wonder about the existence of God?  Do I wonder what God wants from me?  Do I wonder if God is all around me or just in certain aspects of my life?  Do I wonder if there is a universal God?  And yet, the reality is that it doesn’t really matter which interpretation of the question I choose because how I “wonder about God” will be ultimately swallowed up in an analytical black hole where the meanings respondents give to research questions are lost forever.

5 comments

  1. One of the most neglected areas of research, and why so many poll results on all subjects, reported in the media are purely bunk. One of the most valuable exercises I put research clients trough is a validity check…doing qualitative to make sure we know what meaning most potential respondents have for certain terms we are using. If we expect a high variance in meaning, we search for another way to ask the question. Takes time. Takes money. Worth every minute and penny.

    I hear too many non-researchers and psuedo-researchers talk about the reliability of a study, but no mention of validity. I tell them, if I give you get the wrong answer 9 out of 10 times…that’s very reliable garbage.

  2. Ohhh, the God questions… It seems like some of the best examples of poor question wording are from questions about God!

    I find that it is really difficult to convince people to think critically about question design. Oftentimes people believe (despite plenty of research evidence) that if they ask any question in the vicinity of the right one, respondents will just know what the underlying question is.

    Question design is the first, most important, crucial element in survey research. But it is so often just glossed over. It’s easy to see why this happens. The actual process of question writing can feel very arbitrary and unscientific, and the suggested methods for quality control can feel very foreign, expensive and unnecessary.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s