If maximizing our understanding of how people think is fundamental to research design – a common theme throughout Research Design Review – then why is so little attention paid to the idea that thinking is not stagnant but something that is continually changing from moment to moment.  If I ask a survey respondent to name the primary reason she likes store A over store B, the response may be something entirely different than if I ask the same question the following day, or possibly even later the same day.  And if I ask how many miles you drive to the office each day, you might say 10 miles today but 15-20 miles if I ask the same question tomorrow.

Why is that?  In the February 2010 RDR post, “Qualitative Research & Thinking About How People Think,” I reference the four-step cognitive process (posited by others) considered necessary to respond “optimally” to research questions: interpretation, searching for relevant information, integration towards a judgment, and translation of a judgment into a response.   These four stages alone suggest that isolating the key reason for choosing store A over store B may be more complex, requiring more thoughtful contemplation than the quick response researchers typically encourage in order to keep the interview to a manageable length.  And that is why my judgment about how many miles I drive to work may change depending on whether I have included in my search for relevant information (the second stage of the cognitive process) the shortcut I take on certain days or whether I calculated the door-to-door distance versus some other parameter.  The judgments derived by research respondents can change from moment to moment for any number of reasons.

George Bishop and Stephen Mockabee in their December 2011 piece “Comparability of Measurement in Public Opinion Polls” discuss the “peril” of “ignoring the incomparable responses given to most survey questions,” and particularly “how the meaning-and-interpretation of survey questions can vary across respondents and over time even when the wording and context of the question itself remains identical.”  Meanings and interpretations (i.e., judgments) are continually changing within and across respondents depending on the social, economic, and political environment at any point in time.  As one solution, Bishop and Mockabee encourage the use of random probes to reveal (and track) the particular meanings respondents give to question wording.

In addition to social-economic-political contexts, the ever-changing landscape that gives birth to respondents’ judgments is equally impacted by the human condition.  Question interpretation, information retrieval, judgment formulation, and ultimate response may shift depending on a person’s general mood, physical condition, or surrounding stimulus and external cues.   All of these factors impinge on how we think and why identical questions at two different moments in time can elicit unequal responses.

All of this suggests that there is no truth to be gained from our respondents, just a judgment at one point in time.   Yet there is an important truth to be found in the meanings and interpretations that respondents give to our questions at the moment of asking.  Similar to Bishop and Mockabee, I believe that by building cognitive interviewing into our research designs – not just during the pre-field, testing phase but for the entirety of fielding – researchers will not only discover the basis from which responses are given but also gain the ability to segment respondents beyond the typical demographic and lifestyle characteristics to include how respondents group based on their interpretations of research questions.   And, as Bishop and Mockabee state, this would enable researchers to track fluctuations in meanings over time.