As Researchers, We’re All After the Same Thing
May 18, 2013
If you are one of those researchers who work in both quantitative and qualitative design, something you are reminded of fairly quickly at the AAPOR annual conference – currently being held in Boston – is that there is really little separating the two genres. Survey researchers may d
eem the ‘hard facts’ of their quantitative data as a gold standard of sorts; and qualitative researchers may look questioningly at the righteousness of these ‘hard facts’, asking “Where’s the beef?” that explains the “why” behind the data, but there is no debate that we are all after the same thing. The following are just a few of the common areas of interest among quantitative and qualitative researchers:
- Question administration – What to ask & how to ask it
- Interviewer effect – Impact of interviewer’s behavior, appearance, & attitude on response
- Mode – Which mode for which population segment & its impact on response
- Cooperation – How to increase participation & decrease respondent/participant burden
- Analysis – How to organize data & develop coding schemes that accurately represent the data
- Cost – How to “do more” with smaller research budgets
And, interestingly, researchers of all stripes are addressing similar Read Full Text
Content Analysis & Navigating the Stream of Consciousness
April 30, 2013
An article posted on Research Design Review back in 2010 discussed the work of William James and, specifically, his concept that consciousness “flows” like a river or stream. The article goes on to say that James’ “stream of consciousness” is relevant to researc
hers of every stripe because we all share in the goal of designing research “to understand the subjective links within each individual.” Yet these subjective links come at a price, not the least of which is the “messiness” of the analysis as we work towards identifying these links and finding meaning that addresses our objectives.
Whether it is the verbatim comments from survey respondents to open-end questions or the transcripts from focus group discussions or ethnographic interviews, the researcher is faced with the daunting job of conducting a content analysis that reveals how people think while at the same time answers the research Read Full Text
“Keep it simple,” “keep it short,” and “make it fast.” These are the words that many qualitative researchers live by as they sit down to produce the final written report for their clients. The prevailing sense among some, particularly in the 
marketing research field, is that their all-too-busy clients don’t have the time, inclination, or research backgrounds to read lengthy reports detailing nuanced findings and method. Instead, clients want a brief summary of outcomes that are actionable in the short term. It is no wonder that PowerPoint reporting has become so popular. Who needs complete sentences when a key implication from the research can be reduced to a bullet list or an alluring infographic?
But what has become lost in the ever-increasingly-shrinking report is the discussion of research design. Where once at least cursory attention would be given to the basic design elements – this is what we did, this is when we did it, this is where we did it, and these are the demographics of the participants – in the first few pages of the report, this all-important information has been pushed to the back, sometimes to the appendix where it sits like frivolous or unwanted content begging to be ignored. Not only should the research design not be sequestered to the badlands of reporting but the discussion of research design in qualitative research should be expanded and enriched with details of the:
- qualitative method that was used (along with the rationale for using that method),
- target population,
- sample selection and composition of the participants,
- basis by which the interviewer’s/moderator’s guide was developed,
- reason that particular field sites and not others were chosen for the research,
- interviewer’s/moderator’s techniques for eliciting participants’ responses,
- measures that were taken to maximize the credibility and analyzability of the data, and
- coding and other analysis procedures that were used to arrive at the reported interpretations and implications from the outcomes.
The inclusion and elaboration of the research design in qualitative reports matters. It matters because qualitative research has a life, and it is only the researcher’s thick description of the paths and byways the research traveled that allows the life of qualitative research to thrive Read Full Text
Ethnography is a multi-method approach in qualitative research with observation at its core. Prolonged onsite observations in the participants’ natural material world are what make ethnography a unique and important research approach. Needless to say, the 
observer plays a central role in the success of an ethnographic study and there are few more important skills for the observer than those associated with the concept of analytical sensibility. It is the observer’s skills in sensibilities that can compensate for weaknesses in other aspects of the study design, such as the unavoidable pairing of an older male observer with a group of school-age girls. The observer’s analytical sensibilities include the capacity to be aware of and to reflect on his or her surroundings, the actions of the participants, and how the observer may be influencing the outcomes from the observation. This sensibility is analytical in nature because the focus is on the observer’s ability to apply analytical skills while in the field that deepen the researcher’s understanding of the culture and events from the participants’ point of view.
The facet of sensibility that is imperative among all ethnographic observers is, what Stacey and Eckert called, “dual perspective” or the ability to derive meaning from participants’ Read Full Text
Striking a Balance in Research Design
March 18, 2013
One of the healthy outcomes from the rise of social media and mobile research is that it has brought to the forefront the issue of the balance of power – or control – in research 
design. Method specialists who are proponents of social media or mobile research often assert that a big advantage of these approaches is that the participant, not the researcher, controls what is shared or not shared. Qualitative researchers, for example, have discovered the value of Pinterest where, without any researcher involvement, they surmise the hobbies and characteristics of individuals that represent some segment of the population. And a mobile qualitative research study empowers the participant to select when, where, and how (in what format) information is provided to the researcher. The researcher may start with a few basic questions but it is the research participant (knowingly or not) who controls the input.
This participant-leaning balance of power is in contrast to other qualitative research – face-to-face focus groups and in-depth interviews – as well as quantitative survey research where the researcher drives the course for the research with carefully-considered Read Full Text
Accounting for Interactions in Focus Group Research
February 27, 2013
The RDR post on February 20, 2013 talked about focus group research and how it is anything but a “plain vanilla” research method in terms of design considerations. 
To illustrate, the post discussed the issue of group composition; specifically, the “homogeneity or heterogeneity the researcher wants represented by the group participants.” Another important design consideration in face-to-face group discussions centers on the social context and especially the impact that participants’ interactions have on the discussion and, consequently, the research outcomes. This is a pretty obvious facet of the focus group method yet, surprisingly, it is largely ignored in the analysis and reporting of group research. Researchers and non-researchers alike complain about the disruptive effect of “dominators” (outspoken group participants who assert their opinions without regard to others), the refusal of “passive” participants to speak their minds, and/or participants talking over each other (making it impossible to hear/follow the discussion) but focus group reports typically fail to discuss these interactions and the role they played in the final analysis.
The good news is that some researchers have given extensive thought to the interaction effect in focus group research and have promoted the idea that this effect needs to be a considered element in the study design. One example is Lehoux, Poland, and Daudelin (2006) who have proposed a “template” by which qualitative researchers can think about, not only how group interaction impacts the group process but also, how participants’ interaction dictates the learning or knowledge the researcher takes away from the discussion. The Lehoux, et al. template consists of specific questions the researcher should address during the analysis phase. For instance, group-process questions include ”What types of interactions occur among participants?”, “Which participants dominate the discussion?”, and “How does this affect the contribution of other participants?” The knowledge-content questions ask things like “What do dominant and passive positions reveal about the topic at hand?” and “What types of knowledge claims are endorsed and/or challenged by participants?”
The credibility and ultimate usefulness of our focus group research depends on a thorough and honest appreciation for what goes on in the field. The analysis and reporting of the “interactional events” that guided the discussions in our group research is the obligation of all researchers. Otherwise, what really went on in our discussions is some kind of dirty little secret that leaves the users of our research – and the researchers themselves – blinded to the true outcomes. Like a kaleidoscope, our understanding of what we “see” from our focus group research depends on how we account for the interactions taking place, and how each dominant and passive piece plays a role in creating the final effect.
Focus Groups: A Not-So-Plain Vanilla Choice in Research
February 20, 2013
Focus groups are ubiquitous to the point that, for some, they have become the plain vanilla choice in our ever-eclectic assortment of flavors in research methods. Yet, there 
are many (many) design considerations that complicate focus group research while directly impacting the credibility, analyzability, and, ultimately, usefulness of the outcomes. One such consideration is discussed here.
Fundamental to the design of a focus group study is group composition. More specifically, it must be determined the degree of homogeneity or heterogeneity the researcher wants represented by the group participants. There is any number of questions the researcher needs to contemplate, such as the participants’:
- Age range and/or stage of life.
- Race.
- Ethnicity.
- Income or socioeconomic level.
- Level of education.
- Profession or job (including, job title).
- Community of residence.
- Group or organization association.
- Involvement, experience, or knowledge with the research topic, e.g., product usage activity, purchase behavior, level of expertise using new technology.
Whether or not – or the degree to which – group participants should be homogeneous in some or all characteristics has been at the center of debate for some years. On the one hand, Grønkjaer, et al. (2011) claim that, “homogeneity in focus group construction is considered essential for group interaction and dynamics” and, in the same vein, Julius Sim has found Read Full Text
Many researchers have discussed Unilever’s accreditation program for qualitative research. Among others, the Market Research Society, ESOMAR’s Research World, and Kat
hryn Korostoff (Research Rockstar) have all outlined what led up to this program, the objectives of the program, and the accreditation process. In a nutshell, Unilever assessed the outcomes of their many qualitative studies around the globe and determined that the qualitative researchers Unilever has employed to conduct their qual studies have generally failed in providing management with a sufficient caliber of new ideas and insights that serve to move the company forward.
Manish Makhijani, a consumer insights director at Unilever, stated in an interview discussing the program that one of his top concerns with their qualitative research is the inconsistency in “the quality of insights and debriefs” among their qualitative researchers, emphasizing that “what matters in qual more than anything else is the quality of thinking that you put on the table.” And, indeed, Makhijani brought home this point at the November 2012 ESOMAR conference when he presented the notion that “good” qualitative research is derived Read Full Text
Top Two RDR Articles on Qualitative Research Design in 2012
January 8, 2013
In 2012, Research Design Review published 10 articles pertaining to qualitative research 
design. These 10 posts have been compiled into one volume titled, “Qualitative Research Design: Selected articles from Research Design Review published in 2012.” The most popular of these articles among RDR readers are “Designing a Quality In-depth Interview Study: How Many Interviews Are Enough?” published in September and “Insights vs. Metrics: Finding Meaning in Online Qualitative Research” published in June of 2012.
The first of these (i.e., regarding the optimal number of interviews) talks about the “two key moments” when a researcher needs to consider how many interviews to complete – once at the initial design phase and the other while in the field. Consideration at the initial stage of research design centers on very practical matters like the nature of the research topic and the heterogeneity of the target population. However, weighing whether “enough” IDIs have been completed while in the field – in the throes of actually completing interviews – is a more delicate and difficult matter. While the idea of “saturation” or the point in time when responses no longer reveal ‘fresh insights’ is well accepted particularly among researchers dedicated to grounded theory, it is not “good enough” from a quality design perspective. Rather than saturation, this article advises the qualitative researcher to review the IDI completions in the field and answer eight questions concerning their quality. Questions such as, Did every IDI cover every question or issue important to the research? and Can the researcher identify the sources of variations and contradictions in the data?
The second most-popular article – concerning online qualitative research – focused on the distinction between actually gaining new ideas or insights from online qualitative versus simply capturing metrics. The article promotes the belief that offline techniques (such as projective techniques) have their place online and that “the increasingly-loud buzz of social media metrics” or tracking shouldn’t distract qualitative researchers from the business of gaining true, meaningful insights. The article concludes by saying, “All of this tracking has the potential to provide marketers with some idea of what some portion of their target audience is saying or doing at a particular moment in time – insight with a small ‘i’. But let’s not confuse that with the ever-present need to understand how people think – Insight with a big ‘I’.”
These and eight other articles specific to qualitative research design can be found here.
Casting a Light Into the Inner Workings of Qualitative Research
December 17, 2012
Research Design Review has discussed the idea of transparency on several occasions. Last month’s post, titled “Designing Qualitative Research to Produce Outcomes You Can Use,” briefly mentioned the contribution transparency makes to the ultimate usefulness
of a qualitative research study emphasizing that full disclosure of the study’s details “empowers the reader of the research to make his or her own judgments as to the integrity of the research (Is it good research?) as well as its usefulness in furthering new ideas, next steps, and new applications.” The goal of transparency is to provide an audit trail in the final research document that allows the reader to duplicate the research (if that were possible), derive similar conclusions from the data as presented, or apply the research in other contexts. Transparency is important.
Transparency in the final document goes way beyond a simple account of the number and time frame when interviews, groups, or observations were conducted and a Read Full Text
