Sample Size in Qualitative Research & the Risk of Relying on Saturation

Qualitative and quantitative research designs require the researcher to think carefully about how and how many to sample within the population segment(s) of interest related to the research objectives. In doing so, the researcher considers demographic and cultural diversity, as well as other distinguishing characteristics (e.g., usage of a particular service or product) and pragmatic issues Risk of Relying on Saturation(e.g., access and resources). In qualitative research, the number of events (i.e., the number of in-depth interviews, focus group discussions, or observations) and participants is often considered at the early design stage of the research and then again during the field stage (i.e., when the interviews, discussions, or observations are being conducted). This two-stage approach, however, can be problematic. One reason is that giving an accurate sample size prior to data collection can be difficult, particularly when the researcher expects the number to change as the result of in-the-field decisions.

Another potential problem arises when researchers rely solely on the concept of saturation to assess sample size when in the field. In grounded theory, theoretical saturation

“refers to the point at which gathering more data about a theoretical category reveals no new properties nor yields any further theoretical insights about the emerging grounded theory.” (Charmaz, 2014, p. 345)

In the broader sense, Morse (1995) defines saturation as “‘data adequacy’ [or] collecting data until no new information is obtained” (p. 147).

Reliance on the concept of saturation presents two overarching concerns: 1) As discussed in two earlier articles in Research Design ReviewBeyond Saturation: Using Data Quality Indicators to Determine the Number of Focus Groups to Conduct and Designing a Quality In-depth Interview Study: How Many Interviews Are Enough? – the emphasis on saturation has the potential to obscure other important considerations in qualitative research design such as data quality; and 2) Saturation as an assessment tool potentially leads the researcher to focus on the obvious “new information” obtained by each interview, group discussion, or observation rather than gaining a deeper sense of participants’ contextual meaning and more profound understanding of the research question. As Morse (1995) states,

“Richness of data is derived from detailed description, not the number of times something is stated…It is often the infrequent gem that puts other data into perspective, that becomes the central key to understanding the data and for developing the model. It is the implicit that is interesting.” (p. 148)

With this as a backdrop, a couple of recent articles on saturation come to mind. In “A Simple Method to Assess and Report Thematic Saturation in Qualitative Research” (Guest, Namey, & Chen, 2020), the authors present a novel approach to assessing sample size in the in-depth interview method that can be applied during or after data collection. This approach is born from Read Full Text

Cognitive Interviewing: A Few Best Practices

Cognitive interviewing is a method used by survey researchers to investigate the integrity of their questionnaire designs prior to launching the field portion of the study. In the edited volume Cognitive Interviewing Methodology, Kristen Miller (2014) describes cognitive interviewing as “a qualitative method that examines the question-response process, specifically the processes and considerations used by respondents as they form answers to survey q4 attributes of the CI methoduestions,” further explaining that “through the interviewing process, various types of question-response problems that would not normally be identified in a traditional survey interview, such as interpretive errors and recall accuracy, are uncovered” (p. 2). In this way, survey researchers identify the users’ (i.e., survey respondents’) possible meaning and interpretation of survey questions – having to do with question structure or format and terminology – that may or may not deviate from the researcher’s intent. Importantly, the objective of the cognitive interview is not to simply determine whether a questionnaire item “makes sense” to an individual  but to go beyond that to explore the individual’s lived experience (personal context, attitudes, perceptions, behavior) in relationship to their interpretation and/or ability to answer a particular question.

Although not typically included under the “qualitative research” umbrella (with in-depth interviewing, focus group discussions, and observation), four of the 10 unique attributes associated with qualitative research are notably relevant to the cognitive interviewing method. They are the: importance of meaning, flexibility of design, participant-researcher relationship, and researcher skill set. These distinctive qualities of the cognitive interviewing method, and qualitative methods generally, define why researchers opt for Read Full Text

The Focus Group Method

In 2018, five articles were published pertaining to the focus group method. Two of these articles discuss the key differentiating attribute of focus groups, i.e., participant interaction and engagement, and the important role this attribute plays in the integrity of the research.

The interactive component of the focus group method also raises questions concerning mode, which is the subject of two other articles in this compilation. Specifically, these articles address the strengths and limitations of the in-person and online asynchronous focus group modes.

The fifth article in this paper discusses the concept of saturation in the context of determining the “right” number of focus groups to conduct for a particular study. Saturation has been discussed before in RDR, with the emphasis being on the idea that saturation alone is an inadequate measure by which to derive the number of events and, in fact, as a sole measure, saturation jeopardizes data quality, see “Designing a Quality In-depth Interview Study: How Many Interviews Are Enough?”

The focus group method has been discussed in RDR over the years, such as this article on mode differences, and this article (actually, slide show) on applying the Total Quality Framework to focus groups, and a discussion on the use of projective techniques, as well as an article on the many considerations in the design and implementation of the focus group method.