Selection Bias & Mobile Qualitative Research

When I conduct a face-to-face qualitative study – whether it is a group discussion, in-depth interview, or in-situ ethnography – I am taking in much more than the behavior and attitudes of the research participants.  Like most researchers, my scope goes way beyond the most vocal response to my questions or the behavior of store shoppers, but incorporates much more detail including the nuanced comments, the facial and body gestures, as well as the surrounding environment that may be impacting his or her thoughts or movements.  So, while one of my face-to-face participants may tell me that he “just prefers” shopping at a competitor’s store for his hardware, I know from the entirety of clues throughout the interview that there is more to uncover which ultimately lands me on the real reason he avoids my client’s store – the unavailability of store credit.  Likewise, the mobile research participant shopping at Walmart for coffeemakers may share her shopping experience via video and/or text but unintentionally omit certain components – e.g., the impact of competitive displays, product packaging, store lighting, surrounding shoppers – that would have been discovered in an in-person ethnography and contribute important insights.

Selection bias is inherent in nearly all research designs.  At some level research participants are deciding what is important to communicate to the researcher and what is worthy of being ignored.  From deciding whether to participate in a study, to the granularity of details they are willing to share, the participant – not the researcher – controls some measure of the research input.  It is no wonder that many of the discussions concerning research design center on this issue, with survey researchers discussing at length the best method for sampling and selecting respondents (e.g., the next-birthday method in telephone studies), converting initial refusals, and effective probing techniques.

There is not much discussion on selection bias in qualitative research.  One exception is an article by David Collier and James Mahoney* that addresses how selection bias undermines the validity of qualitative research.  More focus on the issue of selection bias in qualitative research is warranted, particularly given the speed with which research designs today are evolving to keep up with new communication technology.

Mobile research is just one example of an increasingly popular qualitative research method.  Mobile research provides for the first time a viable way to Read Full Text

Visual Cues & Bias in Qualitative Research

The Darshan Mehta (iResearch) and Lynda Maddox article “Focus Groups: Traditional vs. Online” in the March 2011 issue of Survey Magazine reminded me of the “visual biases” moderators, clients, and participants bring to the face-to-face research discussion.  While there are downsides to opting for Internet-based qualitative research, the ability to actually control for potential error stemming from visual cues – ranging from demographic characteristics (e.g., age, race, ethnicity, gender) to “clothing and facial expressions” – is a clear advantage to the online (non-Webcam) environment.   Anyone who  has conducted, viewed, or participated in a face-to-face focus group can tell you that judgments are easily made without a word being spoken.

An understanding or at least an appreciation for this inherent bias in our in-person qualitative designs is important to the quality of the interviewing and subsequent analysis as well as the research environment itself.  How does the interviewer change his/her type and format of questioning from one interviewee to another based on nothing more than the differences or contrasts the interviewer perceives between the two of them?  How do the visual aspects of one or more group participants elicit more or less participation among the other members of the group?  How do group discussants and interviewees respond and comment differently depending on their vision of the moderator, other participants, and the research environment?

The potential negative effect from the unwitting bias moderators/interviewers absorb in the research experience has been addressed to some degree.  Mel Prince (along with others) has discussed the idea of “moderator teams” as well as the “serial moderating technique.”  And Sean Jordan states that “moderator bias” simply needs to be “controlled for by careful behavior.”

There is clearly much more effort that needs to be made on this issue.  Creating teams of interviewers may mitigate but may also exasperate the bias effect (e.g., How do we sort out the confounding impact of multiple prejudices from the team?), and instilling “careful behavior” can actually result in an unproductive research session (e.g., Does the controlled, unemotional, sterile behavior of the moderator/interviewer elicit unemotional, sterile, unreal responses from research participants?).

How we conduct and interpret our qualitative research – whether we (consciously or unconsciously) choose to impose barriers to our questioning and analysis in order to proceed with caution through the intersection of not knowing and insight, or go full steam ahead – rests in great measure with our ability to confront the potential prejudice in the researcher, the client, and our research participants.

Error in (Qualitative) Research

It should be pretty obvious from my earlier posts that I am a big believer in the idea that research design is governed by core principles that apply to everything we do.  I believe that it is not good enough to be a qualitative researcher or a quantitative researcher or an online researcher or an ethnographer or whatever.  That, regardless of our mode or technique, we are obligated as researchers to practice “good research” defined by adhering to basic tenets that we all should have learned in school.  Unfortunately, college marketing research courses may fuel silo thinking in research design by organizing in-class discussions around research “classifications” rather than Read Full Text