June 16, 2013
Storytelling is the ultimate goal of all research. In the end, researchers of all kinds are in the business of understanding how people think, and what better way than to hear their stories. Storytelling may sound like something only qualitative researchers should care about but survey researchers, knowingly or not, are equally concerned about the stories people have to tell. The recent brouhaha over Gallup’s failure to correctly predict the winner of the 2012 presidential election is a case in point. One of the fundamental weaknesses that contributed to the Gallup polls favoring a Romney win is how Gallup went about determining likely voters, including respondents’ past voting behavior and how much attention they were paying to the election. Like all pollsters, Gallup simply used the responses to these and other questions to calculate which respondents were most likely to vote in the national election. One of the problems that Gallup ran into, however, is that “many” of the Obama voters claimed not to be paying much attention to the election which, of course, disqualified them as likely voters. In essence, Gallup simply wanted to know each Read Full Text
May 31, 2013
Nonresponse and non-response error is more than a quantitative issue. While qualitative researchers may shudder at the thought, the typically-ignored impact of nonresponse is just as important in the qualitative realm. Why is nonresponse in qualitative research important? Because we are conducting qualitative research. Not qualitative let’s get a few warm bodies around the table for our face-to-face focus group, but actually research methods that, like all research, demand certain protocols that address potential biasing effects. One of these is nonresponse. The warm bodies in our group discussion may make the moderator and client observers feel great – Thank goodness, someone showed up! – but the uncomfortable reality is that the people who chose not to participate – or were never contacted by a recruiter and asked to participate in the first place – greatly affect our research outcomes. Indeed, the trajectory of a group discussion Read Full Text
May 18, 2013
If you are one of those researchers who work in both quantitative and qualitative design, something you are reminded of fairly quickly at the AAPOR annual conference – currently being held in Boston – is that there is really little separating the two genres. Survey researchers may deem the ‘hard facts’ of their quantitative data as a gold standard of sorts; and qualitative researchers may look questioningly at the righteousness of these ‘hard facts’, asking “Where’s the beef?” that explains the “why” behind the data, but there is no debate that we are all after the same thing. The following are just a few of the common areas of interest among quantitative and qualitative researchers:
- Question administration – What to ask & how to ask it
- Interviewer effect – Impact of interviewer’s behavior, appearance, & attitude on response
- Mode – Which mode for which population segment & its impact on response
- Cooperation – How to increase participation & decrease respondent/participant burden
- Analysis – How to organize data & develop coding schemes that accurately represent the data
- Cost – How to “do more” with smaller research budgets
And, interestingly, researchers of all stripes are addressing similar Read Full Text
April 30, 2013
An article posted on Research Design Review back in 2010 discussed the work of William James and, specifically, his concept that consciousness “flows” like a river or stream. The article goes on to say that James’ “stream of consciousness” is relevant to researchers of every stripe because we all share in the goal of designing research “to understand the subjective links within each individual.” Yet these subjective links come at a price, not the least of which is the “messiness” of the analysis as we work towards identifying these links and finding meaning that addresses our objectives.
Whether it is the verbatim comments from survey respondents to open-end questions or the transcripts from focus group discussions or ethnographic interviews, the researcher is faced with the daunting job of conducting a content analysis that reveals how people think while at the same time answers the research Read Full Text
“Keep it simple,” “keep it short,” and “make it fast.” These are the words that many qualitative researchers live by as they sit down to produce the final written report for their clients. The prevailing sense among some, particularly in the marketing research field, is that their all-too-busy clients don’t have the time, inclination, or research backgrounds to read lengthy reports detailing nuanced findings and method. Instead, clients want a brief summary of outcomes that are actionable in the short term. It is no wonder that PowerPoint reporting has become so popular. Who needs complete sentences when a key implication from the research can be reduced to a bullet list or an alluring infographic?
But what has become lost in the ever-increasingly-shrinking report is the discussion of research design. Where once at least cursory attention would be given to the basic design elements – this is what we did, this is when we did it, this is where we did it, and these are the demographics of the participants – in the first few pages of the report, this all-important information has been pushed to the back, sometimes to the appendix where it sits like frivolous or unwanted content begging to be ignored. Not only should the research design not be sequestered to the badlands of reporting but the discussion of research design in qualitative research should be expanded and enriched with details of the:
- qualitative method that was used (along with the rationale for using that method),
- target population,
- sample selection and composition of the participants,
- basis by which the interviewer’s/moderator’s guide was developed,
- reason that particular field sites and not others were chosen for the research,
- interviewer’s/moderator’s techniques for eliciting participants’ responses,
- measures that were taken to maximize the credibility and analyzability of the data, and
- coding and other analysis procedures that were used to arrive at the reported interpretations and implications from the outcomes.
The inclusion and elaboration of the research design in qualitative reports matters. It matters because qualitative research has a life, and it is only the researcher’s thick description of the paths and byways the research traveled that allows the life of qualitative research to thrive Read Full Text
Ethnography is a multi-method approach in qualitative research with observation at its core. Prolonged onsite observations in the participants’ natural material world are what make ethnography a unique and important research approach. Needless to say, the observer plays a central role in the success of an ethnographic study and there are few more important skills for the observer than those associated with the concept of analytical sensibility. It is the observer’s skills in sensibilities that can compensate for weaknesses in other aspects of the study design, such as the unavoidable pairing of an older male observer with a group of school-age girls. The observer’s analytical sensibilities include the capacity to be aware of and to reflect on his or her surroundings, the actions of the participants, and how the observer may be influencing the outcomes from the observation. This sensibility is analytical in nature because the focus is on the observer’s ability to apply analytical skills while in the field that deepen the researcher’s understanding of the culture and events from the participants’ point of view.
February 27, 2013
The RDR post on February 20, 2013 talked about focus group research and how it is anything but a “plain vanilla” research method in terms of design considerations. To illustrate, the post discussed the issue of group composition; specifically, the “homogeneity or heterogeneity the researcher wants represented by the group participants.” Another important design consideration in face-to-face group discussions centers on the social context and especially the impact that participants’ interactions have on the discussion and, consequently, the research outcomes. This is a pretty obvious facet of the focus group method yet, surprisingly, it is largely ignored in the analysis and reporting of group research. Researchers and non-researchers alike complain about the disruptive effect of “dominators” (outspoken group participants who assert their opinions without regard to others), the refusal of “passive” participants to speak their minds, and/or participants talking over each other (making it impossible to hear/follow the discussion) but focus group reports typically fail to discuss these interactions and the role they played in the final analysis.
The good news is that some researchers have given extensive thought to the interaction effect in focus group research and have promoted the idea that this effect needs to be a considered element in the study design. One example is Lehoux, Poland, and Daudelin (2006) who have proposed a “template” by which qualitative researchers can think about, not only how group interaction impacts the group process but also, how participants’ interaction dictates the learning or knowledge the researcher takes away from the discussion. The Lehoux, et al. template consists of specific questions the researcher should address during the analysis phase. For instance, group-process questions include ”What types of interactions occur among participants?”, “Which participants dominate the discussion?”, and “How does this affect the contribution of other participants?” The knowledge-content questions ask things like “What do dominant and passive positions reveal about the topic at hand?” and “What types of knowledge claims are endorsed and/or challenged by participants?”
The credibility and ultimate usefulness of our focus group research depends on a thorough and honest appreciation for what goes on in the field. The analysis and reporting of the “interactional events” that guided the discussions in our group research is the obligation of all researchers. Otherwise, what really went on in our discussions is some kind of dirty little secret that leaves the users of our research – and the researchers themselves – blinded to the true outcomes. Like a kaleidoscope, our understanding of what we “see” from our focus group research depends on how we account for the interactions taking place, and how each dominant and passive piece plays a role in creating the final effect.
February 20, 2013
Focus groups are ubiquitous to the point that, for some, they have become the plain vanilla choice in our ever-eclectic assortment of flavors in research methods. Yet, there are many (many) design considerations that complicate focus group research while directly impacting the credibility, analyzability, and, ultimately, usefulness of the outcomes. One such consideration is discussed here.
Fundamental to the design of a focus group study is group composition. More specifically, it must be determined the degree of homogeneity or heterogeneity the researcher wants represented by the group participants. There is any number of questions the researcher needs to contemplate, such as the participants’:
- Age range and/or stage of life.
- Income or socioeconomic level.
- Level of education.
- Profession or job (including, job title).
- Community of residence.
- Group or organization association.
- Involvement, experience, or knowledge with the research topic, e.g., product usage activity, purchase behavior, level of expertise using new technology.
Whether or not – or the degree to which – group participants should be homogeneous in some or all characteristics has been at the center of debate for some years. On the one hand, Grønkjaer, et al. (2011) claim that, “homogeneity in focus group construction is considered essential for group interaction and dynamics” and, in the same vein, Julius Sim has found Read Full Text
Many researchers have discussed Unilever’s accreditation program for qualitative research. Among others, the Market Research Society, ESOMAR’s Research World, and Kathryn Korostoff (Research Rockstar) have all outlined what led up to this program, the objectives of the program, and the accreditation process. In a nutshell, Unilever assessed the outcomes of their many qualitative studies around the globe and determined that the qualitative researchers Unilever has employed to conduct their qual studies have generally failed in providing management with a sufficient caliber of new ideas and insights that serve to move the company forward.
Manish Makhijani, a consumer insights director at Unilever, stated in an interview discussing the program that one of his top concerns with their qualitative research is the inconsistency in “the quality of insights and debriefs” among their qualitative researchers, emphasizing that “what matters in qual more than anything else is the quality of thinking that you put on the table.” And, indeed, Makhijani brought home this point at the November 2012 ESOMAR conference when he presented the notion that “good” qualitative research is derived Read Full Text