When I conduct a face-to-face qualitative study – whether it is a group discussion, in-depth interview, or in-situ ethnography – I am taking in much more than the behavior and attitudes of the research participants.  Like most researchers, my scope goes way beyond the most vocal response to my questions or the behavior of store shoppers, but incorporates much more detail including the nuanced comments, the facial and body gestures, as well as the surrounding environment that may be impacting his or her thoughts or movements.  So, while one of my face-to-face participants may tell me that he “just prefers” shopping at a competitor’s store for his hardware, I know from the entirety of clues throughout the interview that there is more to uncover which ultimately lands me on the real reason he avoids my client’s store – the unavailability of store credit.  Likewise, the mobile research participant shopping at Walmart for coffeemakers may share her shopping experience via video and/or text but unintentionally omit certain components – e.g., the impact of competitive displays, product packaging, store lighting, surrounding shoppers – that would have been discovered in an in-person ethnography and contribute important insights.

Selection bias is inherent in nearly all research designs.  At some level research participants are deciding what is important to communicate to the researcher and what is worthy of being ignored.  From deciding whether to participate in a study, to the granularity of details they are willing to share, the participant – not the researcher – controls some measure of the research input.  It is no wonder that many of the discussions concerning research design center on this issue, with survey researchers discussing at length the best method for sampling and selecting respondents (e.g., the next-birthday method in telephone studies), converting initial refusals, and effective probing techniques.

There is not much discussion on selection bias in qualitative research.  One exception is an article by David Collier and James Mahoney* that addresses how selection bias undermines the validity of qualitative research.  More focus on the issue of selection bias in qualitative research is warranted, particularly given the speed with which research designs today are evolving to keep up with new communication technology.

Mobile research is just one example of an increasingly popular qualitative research method.  Mobile research provides for the first time a viable way to reach consumers in their own environment and to gain a real-time view of their world.  At long last we have direct access to something that in the past has been elusive – reality, the connection between what people think, what they say, and what they actually do.  Mobile qualitative research is fueled by the notion that capturing people “in the moment” and allowing participants to drive what is or is not shared with the researcher results in a more real (i.e., accurate) accounting of some microcosm of a person’s life.  So, is it any wonder that mobile qualitative research was hailed as “more accurate” (compared to traditional modes) at the recent Market Research in a Mobile World 2011 conference in Atlanta (Kristin Schwitzer and Dana Slaughter’s, “Using Mobile Qualitatively to Transform Insight Generation”), and “closer to the truth” in Qualvu’s recent Webinar (“Mobile Research Gets Real”)?

And that brings me back to selection bias.  While the participant-driven model of mobile qualitative research may provide one perspective of human nature at a given point in time, we have to wonder how much of the whole story we are really getting.  As long as participants control the portal by which we judge their attitudes and behavior, we run the real risk of introducing selection error into our research designs.  Sound qualitative research, like any other research method, is built on a framework of design principles that ensure the integrity of our findings.  I look forward to future discussions of error-prone weaknesses in mobile and other qualitative research designs.

* Collier, David and James Mahoney, “Insights and Pitfalls: Selection Bias in Qualitative Research,” World Politics 49 (October 1996), 56-91.