13 Factors Impacting the Quality of Qualitative Research

February 28, 2011

How does a client, an end-user, or buyer evaluate the quality of our qualitative research?  How does this person know with any degree of confidence that the qualitative end-product is legitimately useful?  We conduct the interviews and observations, deliver audio, video, and written analysis of our findings filled with implications and next steps, but how does anyone judge its efficacy as a piece of qualitative research?

We don’t seem to have this problem in survey research.  The countless discussions and experiments conducted on various aspects of survey design give ongoing support for a “quality framework” by which providers and users can gauge research results.  To this end, quantitative researchers often talk about “total survey error” and “fitness for use” referring to the variety of potential errors and “dimensions” that impact the survey quality framework.  By highlighting these errors, both researcher and end-user more fully appreciate research outcomes and understand what they have (or don’t have).  They understand, for instance, how the accuracy or projectable component of their research may have been sacrificed due to insurmountable budget or schedule constraints.

A quality framework is lacking in qualitative research.  Beyond the basic dos and don’ts, there are no tested dimensions we can use to compare one qualitative study from another.  While research-on-research is critical to improving the quality of what we do, the qualitative marketing research world has been blatantly absent from the investigative scene.  In their 2001 paper (“Why We Need to Reassess Focus Group Research”), Catterall and Clarke discuss the work that has been done in focus group research to better understand the effect of variables such as: the inclusion of professional participants, the presence of observers, and interviewer (moderator) effects.  Yet much of this work is done outside the practitioner arena and industry-wide discussions (dare I say, experimentation) on these and similar issues are, for all intents and purposes, nonexistent.

Back in 1944, Edwards Deming developed a classification of potential error in survey research, identifying 13 “factors affecting the ultimate usefulness of a survey.” These factors include “variability in response,” “bias and variation arising from the interviewer,” “imperfections in the design of the questionnaire,” among others.  So, where is our list of factors impacting the quality of qualitative research allowing us to judge the usefulness of our efforts?  One such classification scheme looks like this:

13 Factors Impacting the Quality of Qualitative Research

The Environment

Potential variability associated with the:

  • Particular venue/setting (incl., face-to-face and online)
  • Presence of observers/interviewers as well as other participants (e.g., groups vs. IDIs)
  • Audio & video recording

The Dynamics

Potential variability associated with:

  • Professional participants (“cheaters”)
  • Participants’ cultural/social/economic/gender/age diversity
  • Cognitive processes/constructs
  • Geographic/regional differences
  • Dominators, group vs. individual think

The Interviewer/Moderator

Potential variability associated with the:

  • Personal/personality aspects of the interviewer/moderator
  • “Best” techniques utilized for specific topics, type of participants, venue
  • Question formatting
  • Question sequencing
  • Use of projective techniques (e.g., what to use when, impact on the discussion overall, analytical schemes)

6 Responses to “13 Factors Impacting the Quality of Qualitative Research”

  1. IMHO you forgot the main aspect: the recruitment quality. That’s the main factor for a good quality FG/IDI… and that’s the main problem for qualitative research, as well…

    • Thank you, Andrea. I totally agree. I began my list of factors with the assumption that all the quality factors imperative to the field and field process were in place. I continue to work on creating a quality framework for qualitative, and have already been working on the recruiting aspect.

  2. Very well written piece!!

  3. nation citisen said

    When i read this “product” and try to find the personal significance and meaning of this information i would tend to engage in some more reflections and seek clarifications without judgment or criticism. It seems to me that applying values from additional paradigms of thinking, may assist in evaluating the contributions of qualitative/ affective research.

  4. It feels as if this discussion is applying a quantitative mindset to qualitative research. In marketing research in particular, insight and understanding is the goal, and the variability of interviewers/ situations/ techniques can be seen as a benefit in bringing different perspectives to bear on the research problem. While it’s fundamentally important to have correctly recruited respondents and researchers who have good eliciting and listening skills, know how to manage interpersonal dynamics, build relationships,analyse and interpret(the foundations of quality), it is possible for their styles and approaches to vary dramatically. In qual its the researcher who is the research instrument and a genuine commitment to honing this instrument is probably the best guarantee of quality.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 204 other followers

%d bloggers like this: