saturation

Cognitive Interviewing: A Few Best Practices

Cognitive interviewing is a method used by survey researchers to investigate the integrity of their questionnaire designs prior to launching the field portion of the study. In the edited volume Cognitive Interviewing Methodology, Kristen Miller (2014) describes cognitive interviewing as “a qualitative method that examines the question-response process, specifically the processes and considerations used by respondents as they form answers to survey q4 attributes of the CI methoduestions,” further explaining that “through the interviewing process, various types of question-response problems that would not normally be identified in a traditional survey interview, such as interpretive errors and recall accuracy, are uncovered” (p. 2). In this way, survey researchers identify the users’ (i.e., survey respondents’) possible meaning and interpretation of survey questions – having to do with question structure or format and terminology – that may or may not deviate from the researcher’s intent. Importantly, the objective of the cognitive interview is not to simply determine whether a questionnaire item “makes sense” to an individual  but to go beyond that to explore the individual’s lived experience (personal context, attitudes, perceptions, behavior) in relationship to their interpretation and/or ability to answer a particular question.

Although not typically included under the “qualitative research” umbrella (with in-depth interviewing, focus group discussions, and observation), four of the 10 unique attributes associated with qualitative research are notably relevant to the cognitive interviewing method. They are the: importance of meaning, flexibility of design, participant-researcher relationship, and researcher skill set. These distinctive qualities of the cognitive interviewing method, and qualitative methods generally, define why researchers opt for Read Full Text

The Focus Group Method

In 2018, five articles were published pertaining to the focus group method. Two of these articles discuss the key differentiating attribute of focus groups, i.e., participant interaction and engagement, and the important role this attribute plays in the integrity of the research.

The interactive component of the focus group method also raises questions concerning mode, which is the subject of two other articles in this compilation. Specifically, these articles address the strengths and limitations of the in-person and online asynchronous focus group modes.

The fifth article in this paper discusses the concept of saturation in the context of determining the “right” number of focus groups to conduct for a particular study. Saturation has been discussed before in RDR, with the emphasis being on the idea that saturation alone is an inadequate measure by which to derive the number of events and, in fact, as a sole measure, saturation jeopardizes data quality, see “Designing a Quality In-depth Interview Study: How Many Interviews Are Enough?”

The focus group method has been discussed in RDR over the years, such as this article on mode differences, and this article (actually, slide show) on applying the Total Quality Framework to focus groups, and a discussion on the use of projective techniques, as well as an article on the many considerations in the design and implementation of the focus group method.

Beyond Saturation: Using Data Quality Indicators to Determine the Number of Focus Groups to Conduct

 The following is a modified excerpt from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 130-131).

Qualitative researchers are routinely faced with the decision of how many in-depth interviews (IDIs) orSaturation focus group discussions to conduct. This decision often revolves around time-cost-benefit trade-off considerations fueled by the tension between neither wanting to conduct too many nor too few IDIs or focus groups.

When it comes to the focus group method, the decision of how many group discussions to conduct is based on any number of factors and will vary depending on the situation for each study.  However, a few of the critical factors that the prudent researcher will think about when considering the number of discussions at the outset for any focus group study are the:

  • Geographic range of the target population, e.g., whether the target population for in-person groups is located in one city or spread across the U.S.
  • Depth of the discussions, i.e., the number of topics/issues and questions expected to be covered to satisfy research objectives. For example, fewer group discussions may be necessary if the primary research objective is to learn mothers’ preferences for shelf-stable baby food, while a greater number of groups may be needed if the objective is to understand mothers’ preferences across all types of baby food and, specifically, to investigate the priority they place on nutritional and organic foods.
  • Homogeneity or heterogeneity of the group participants. Using the example above, more groups will be required if the mothers of interest range in age from 25-40 years as well as in income level and if there is reason to believe that attitudes and behavior vary across these demographic characteristics.
  • Variation in results that is expected to occur across the different focus groups that will be conducted. If there is little variation expected from one group to another (e.g., if group participants are highly homogeneous, or the attitudes among participants in New York are not expected to be different than those in Dallas), then only a few focus groups may suffice.  If there is a great deal of variation expected, then many focus groups will be required to fully measure the range of experiences, attitudes, and knowledge the participants will have to impart in the discussions.
  • Project schedule and amount of available time to complete the study.
  • Research budget that is available to fund the study.

It is this assortment of factors that cause qualitative researchers to generally disagree on the optimal number of Read Full Text