In “‘I Wonder About God’ & Other Poorly-Designed Questions” (Research Design Review, July 25, 2012), it is argued that weak survey question design has a “potentially negative impact on analysis, which in turn leads to wrong conclusions, which in turn leads end users along a path of misguided next steps.” As one of several examples, this article highlights the ambiguity embedded in SurveyMonkey’s “The God Survey”; specifically, the problematic first question that asks how often “I wonder about God.” Poorly-designed questions raise serious concerns about how or if the researcher can legitimately analyze the resulting data (while also tackling issues of reliability and validity), a concern made more profound by the frequent failure to even consider the alternative interpretations respondents may give to survey questions. By failing to recognize the analytical limitations associated with “questionable questions,” the survey data “will be ultimately swallowed up in an analytical black hole where the meanings respondents give to research questions are lost forever.”

The October 2014 article – “Humanizing Survey Question Design with a Qualitative Touch” – promotes the idea of using qualitative methods “to create user-friendly survey questions that speak to respondents rather than at respondents.” The point being that “utilizing qualitative sensitivities to unwrap the true purpose of survey questions while replacing corporate jargon with the way real people talk and think, humanizes the research ‘instrument’ which is a win-win for researchers and respondents.”

Equally important to the notion of integrating qualitative sensibilities in the question-development phase is the additional step of inserting measures of clarification in the survey interview/questionnaire itself. To illustrate, it was suggested in this article that Pew Research’s 2013 study asking about government surveillance could have benefited from “a simple add-on question at the end of the survey interview – such as, ‘Were you thinking of anything in particular when I asked you about the government’s surveillance programs?’ [IF YES] What were you thinking?’” – in order to “shed some light on the extent to which respondents were in sync with the researcher’s meaning [of government surveillance programs].” Shedding light on what respondents are actually thinking when responding to survey questions goes a long way to increasing the credibility and quality of survey research.

Fortunately, there are researchers who have designed solutions and platforms specifically with the idea of marrying qualitative insight with quantitative survey design. 20|20 Research is just one example of a provider whose technology enables the researcher to incorporate qualitative IDIs or group discussions with an online quantitative study and thereby add depth and meaning to survey responses. Others, such as Knowledge Networks and Focus Pointe Global offer their own qualitative-quantitative integration solutions.

The question is: Why hasn’t the idea of incorporating qualitative techniques with the survey research interview received greater attention; and, indeed, why hasn’t a qualitative-quantitative approach become a best practice in survey research design? It does not take much looking around to find reports of survey findings that could use some clarification – some justification – to explain the purported conclusions from the data. It would be good, for instance, to understand why Hispanics in the 2014 Gallup-Lumina Poll were much more likely to state that “education beyond high school is affordable to anyone in this country who needs it” rather than, as Gallup concludes, attribute their relatively high agreement to “greater optimism.” Are Hispanic people more “optimistic” on the affordability of higher education? And, if so, what exactly does it mean to harbor “greater optimism” and how do Hispanic respondents operationally define that? And, if optimism has nothing to do with their responses to that survey question, what does explain why more Hispanics believe in the affordability of higher education? The integration of a qualitative approach – that sheds light on the context and meaning of survey responses – is a useful and necessary condition to a quality research design.

Image captured from: http://www.laboratoryequipment.com/news/2015/03/research-sheds-light-how-plants-control-themselves