Sample Size

A TQF Approach to Sample Size

Credibility TQF componentSample size and sampling in qualitative research design have been discussed elsewhere in Research Design Review, see “Sample Size in Qualitative Research & the Risk of Relying on Saturation” and “Shared Constructs in Research Design: Part 1 — Sampling.” In June 2022, “A TQF Approach to Choosing a Sample Design” was posted to RDR and considers ways to ensure that research participants are representative (share defining characteristics) of the population being studied.

The following is a modified excerpt from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 26-27) that briefly examines a Total Quality Framework (TQF) approach to another facet of sample design, i.e., sample size.

 


How large a sample to use is a decision that qualitative researchers need to make explicitly and carefully in order to increase the likelihood that their studies will generate credible data by well representing their population of interest. Unlike quantitative researchers who most often rely on statistical formulae to determine the sample sizes for their studies, qualitative researchers must rely on (a) past experience and knowledge of the subject matter; and (b) ongoing monitoring during the data-gathering period, which includes applying a set of decision rules, such as those listed in “Designing a Quality In-depth Interview Study: How Many Interviews Are Enough?” These decision rules consider (a) the complexity of the phenomena being studied, (b) the heterogeneity or homogeneity of the population being studied, (c) the level of analysis and interpretation that will be carried out, and (d) the finite resources available to support the study. These types of decision guidelines, along with past experience, should provide qualitative researchers with the considerations they need to carefully judge the amount of data necessary to meet their research objectives. (Of note, if a researcher does not have sufficient past personal experience, a literature review, or speaking directly with other researchers who do have such experience, should serve well.)

As importantly, during the period when data are being gathered, researchers should also closely monitor the amount of variability in the data, compared to the variability that was expected, for the key measures of the study. Based on this monitoring, researchers are responsible for making a “Goldilocks decision” about whether the sample size they originally decided was needed is too large, too small, or just about right. In making a decision to cut back on the amount of data to be gathered, because there is less variability in what is being measured than anticipated, the researcher needs to make certain that those cases that originally were sampled, but would be dropped, are not systematically different from the cases from which data will be gathered. In making a decision to increase the size of the sample, because there is more variability in what is being measured than anticipated, the researcher needs to make certain that the cases added to the sample are chosen in a way that is representative of the entire population (e.g., using the same orderly approach that was used to create the initial sample).

In all instances, and if the necessary resources (staff, time, budget) are available, it is prudent for a researcher to error on the side of having more rather than less data. Gathering too much data does no harm to the quality of the study’s findings and interpretations, but having too little data leaves the researcher in the untenable position of harming the quality of the study because the complexity of what was being studied will not be adequately represented in the available data. For example, case study research to investigate new public school policies related to the core science curriculum might include in-depth interviews with school principals and science teachers, observations of science classes in session, and a review of students’ test papers; however, as a complex subject matter, the research may be weakened by not including discussions with the students and their parents as well as by a failure to include all schools (or a representative sample of schools) in the research design.

Roller, M. R., & Lavrakas, P. J. (2015). Applied qualitative research design: A total quality framework approach. New York: Guilford Press.

Sample Size in Qualitative Research & the Risk of Relying on Saturation

Qualitative and quantitative research designs require the researcher to think carefully about how and how many to sample within the population segment(s) of interest related to the research objectives. In doing so, the researcher considers demographic and cultural diversity, as well as other distinguishing characteristics (e.g., usage of a particular service or product) and pragmatic issues Risk of relying on saturation(e.g., access and resources). In qualitative research, the number of events (i.e., the number of in-depth interviews, focus group discussions, or observations) and participants is often considered at the early design stage of the research and then again during the field stage (i.e., when the interviews, discussions, or observations are being conducted). This two-stage approach, however, can be problematic. One reason is that giving an accurate sample size prior to data collection can be difficult, particularly when the researcher expects the number to change as the result of in-the-field decisions.

Another potential problem arises when researchers rely solely on the concept of saturation to assess sample size when in the field. In grounded theory, theoretical saturation

“refers to the point at which gathering more data about a theoretical category reveals no new properties nor yields any further theoretical insights about the emerging grounded theory.” (Charmaz, 2014, p. 345)

In the broader sense, Morse (1995) defines saturation as “‘data adequacy’ [or] collecting data until no new information is obtained” (p. 147).

Reliance on the concept of saturation presents two overarching concerns: 1) As discussed in two earlier articles in Research Design ReviewBeyond Saturation: Using Data Quality Indicators to Determine the Number of Focus Groups to Conduct and Designing a Quality In-depth Interview Study: How Many Interviews Are Enough? – the emphasis on saturation has the potential to obscure other important considerations in qualitative research design such as data quality; and 2) Saturation as an assessment tool potentially leads the researcher to focus on the obvious “new information” obtained by each interview, group discussion, or observation rather than gaining a deeper sense of participants’ contextual meaning and more profound understanding of the research question. As Morse (1995) states,

“Richness of data is derived from detailed description, not the number of times something is stated…It is often the infrequent gem that puts other data into perspective, that becomes the central key to understanding the data and for developing the model. It is the implicit that is interesting.” (p. 148)

With this as a backdrop, a couple of recent articles on saturation come to mind. In “A Simple Method to Assess and Report Thematic Saturation in Qualitative Research” (Guest, Namey, & Chen, 2020), the authors present a novel approach to assessing sample size in the in-depth interview method that can be applied during or after data collection. This approach is born from Read Full Text