How does a client, an end-user, or buyer evaluate the quality of our qualitative research?  How does this person know with any degree of confidence that the qualitative end-product is legitimately useful?  We conduct the interviews and observations, deliver audio, video, and written analysis of our findings filled with implications and next steps, but how does anyone judge its efficacy as a piece of qualitative research?

We don’t seem to have this problem in survey research.  The countless discussions and experiments conducted on various aspects of survey design give ongoing support for a “quality framework” by which providers and users can gauge research results.  To this end, quantitative researchers often talk about “total survey error” and “fitness for use” referring to the variety of potential errors and “dimensions” that impact the survey quality framework.  By highlighting these errors, both researcher and end-user more fully appreciate research outcomes and understand what they have (or don’t have).  They understand, for instance, how the accuracy or projectable component of their research may have been sacrificed due to insurmountable budget or schedule constraints.

A quality framework is lacking in qualitative research.  Beyond the basic dos and don’ts, there are no tested dimensions we can use to compare one qualitative study from another.  While research-on-research is critical to improving the quality of what we do, the qualitative marketing research world has been blatantly absent from the investigative scene.  In their 2001 paper (“Why We Need to Reassess Focus Group Research”), Catterall and Clarke discuss the work that has been done in focus group research to better understand the effect of variables such as: the inclusion of professional participants, the presence of observers, and interviewer (moderator) effects.  Yet much of this work is done outside the practitioner arena and industry-wide discussions (dare I say, experimentation) on these and similar issues are, for all intents and purposes, nonexistent.

Back in 1944, Edwards Deming developed a classification of potential error in survey research, identifying 13 “factors affecting the ultimate usefulness of a survey.” These factors include “variability in response,” “bias and variation arising from the interviewer,” “imperfections in the design of the questionnaire,” among others.

So, where is our list of factors impacting the quality of qualitative research allowing us to judge the usefulness of our efforts?  One such classification scheme looks like this:

13 Factors Impacting the Quality of Qualitative Research

The Environment

Potential variability associated with the:

• Particular venue/setting (incl., face-to-face and online)
• Presence of observers/interviewers as well as other participants (e.g., groups vs. IDIs)
• Audio & video recording

The Dynamics

Potential variability associated with:

• Professional participants (“cheaters”)
• Participants’ cultural/social/economic/gender/age diversity
• Cognitive processes/constructs
• Geographic/regional differences
• Dominators, group vs. individual think

 

The Interviewer/Moderator

Potential variability associated with the:

• Personal/personality aspects of the interviewer/moderator
• “Best” techniques utilized for specific topics, type of participants, venue
• Question formatting
• Question sequencing
• Use of projective techniques (e.g., what to use when, impact on the discussion overall, analytical schemes)

 

Catterall, M., & Clarke, W. (2001). Why we need to reassess focus group research. ACR Asia-Pacific Advances.