Question design is difficult.  Anyone who has run cognitive interviews or simply conducted a focus group has discovered that even the most carefully designed question may be interpreted far afield from its intended meaning.  While qualitative methods give researchers insight on how interpretations of a question vary (and how to better design the question to come closer to the researcher’s objective), the reality is that question design is rarely put to the test and given the scrutiny it deserves.  Time and budget limitations as well as researchers’ overconfidence in their question-design skills typically lead to a hastily crafted and executed questionnaire.

This is a critical problem not only because it transcends mode – question design is an issue in off- and online modes as well as across quantitative and qualitative methods – but, more importantly, it has a direct, potentially negative impact on analysis, which in turn leads to wrong conclusions, which in turn leads end users along a path of misguided next steps.

Of course some poorly-designed questions are intentional, particularly in an election season when partisan politics triumph over sound research design.  A recent highly-public example is a survey conducted by the conservative-leaning Doctor Patient Medical Association among “random doctors.”  While the methodology alone raises concerns, of interest here is the question design and the reported conclusions from question data.  Of particular note is the unexacting question:  “How do current changes in the medical system affect your desire to practice medicine?”  In response, 82.6% of these doctors answered, “Makes me think about quitting.”  Wow, a strong agreement about “quitting,” but quitting because of what?  “Current changes in the medical system”?  That covers a wide territory.  Yet the reported conclusion to this survey result was stated in the media as:  “83% of American physicians have considered leaving their practices over President Barack Obama’s health care reform law” [emphasis added].

Another recent study concerning the use of scientific studies asked, “Overall, how often do you utilize information reported from scientific studies; or participate in activities that contribute to scientific studies?”  This double-barreled question is difficult enough but the answer options were equally confounding:

• once daily or more
• once weekly or more
• once monthly or more
• etc.

If I do something once a week or more, could I not also do it daily?  If I do something once a month or more, could I not also do it monthly, maybe even daily?

A technique I often use with clients who are insisting on an ill-conceived question wording is to ask them to think forward in time to when we are analyzing the data, to when we are attempting to draw relevant and worthwhile conclusions from survey results.  I simply ask, ‘Will we be able to honestly interpret the response to that question and clearly know what the data is telling us?’  For instance, how would we interpret the results from the survey question, “Do you access the Internet on at least an occasional basis?”  If everyone who answers “yes” is considered an “Internet user,” what does that really mean since we know nothing about how these people differ in terms of their interpretation of the word “occasional” and their meaning of “access” (e.g., I can access the Internet without being engaged by simply downloading email on my iPhone)?  So, can I really say anything about “Internet users” based on this survey question given that I really don’t know anything about their use or how segmented or unified the group is on key usage indicators?

In “The God Survey” from SurveyMonkey, the first question is, “I wonder about God…A lot, A little, Rarely, Never.”  While this is a potentially interesting question, it is difficult to analyze because it is impossible to know what the respondent was actually thinking about when answering.  As the respondent, I can only speculate what the researcher wants me to wonder about.  Do I wonder about the existence of God?  Do I wonder what God wants from me?  Do I wonder if God is all around me or just in certain aspects of my life?  Do I wonder if there is a universal God?  And yet, the reality is that it doesn’t really matter which interpretation of the question I choose because how I “wonder about God” will be ultimately swallowed up in an analytical black hole where the meanings respondents give to research questions are lost forever.