Research Design & Methods

A TQF Approach to Construct Validity

TQF approach to construct validity

Construct validity plays an important role in the design, implementation, analysis, and ultimate usefulness of qualitative research methods. The construct of validity itself in qualitative research is discussed in this article and cites qualitative researchers across disciplines who explore “unique dimensions” and other considerations  relating to validity in qualitative research.

The Total Quality Framework (TQF) relies heavily on construct validity in its quality approach to each phase of the qualitative research process. At each phase, the researcher must ask “Am I gaining real knowledge about the core concepts that are the focus of this research?” For example,

  • An important step when developing a research design is to identify the key constructs associated with the research objectives to investigate, and the particular attributes of each construct that the researcher wants to explore. So, for example, a researcher conducting a study on dietary behavior may have interest in “health consciousness,” including shopping behavior related to organic and fresh foods.
  • In the in-depth interview and focus group discussion methods, careful attention needs to be paid to guide development and the inclusion of questions relevant to the constructs of interest. When developing the guide, the researcher needs to ask “Is this [topic, question, technique] relevant to the construct we are investigating?”, and “Does this [topic, question, technique] provide us with knowledge about the aspect of the construct that we intended to explore in the interviews/discussions?”
  • In ethnography, the observation guide and observation grid are important tools. “The grid is similar to the guide in that it helps to remind the observer of the events and issues of most import; however, the observation grid is a spreadsheet or log of sorts that enables the observer to actually record and reflect on observable events in relationship to the research constructs of interest” (Roller & Lavrakas, 2015, p. 206).
  • The quality of qualitative data analysis hinges on the researcher’s ability to effectively identify, analyze, and develop valid interpretations of the data around the important constructs associated with the research objectives. To assist the researcher, a TQF approach to analysis recommends a codebook format and coding form (which is basically a reflexive journal for the coder[s] to record thoughts and justifications for their coding decisions) that highlights constructs of interest. For example,

TQF codebook and coding form

  • Construct validity also plays an important role in the transparency of the final research document. In the study report, the researcher can (and should) elaborate on the design, data gathering, and analysis decisions that were made pertaining to the key constructs, as well as the main themes that were derived from the data — i.e., the knowledge that was gained from the research — concerning these constructs.

Roller, M. R., & Lavrakas, P. J. (2015). Applied qualitative research design: A total quality framework approach. New York: Guilford Press.

 

Photo by Bon Vivant

A TQF Approach to Sample Size

Credibility TQF componentSample size and sampling in qualitative research design have been discussed elsewhere in Research Design Review, see “Sample Size in Qualitative Research & the Risk of Relying on Saturation” and “Shared Constructs in Research Design: Part 1 — Sampling.” In June 2022, “A TQF Approach to Choosing a Sample Design” was posted to RDR and considers ways to ensure that research participants are representative (share defining characteristics) of the population being studied.

The following is a modified excerpt from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 26-27) that briefly examines a Total Quality Framework (TQF) approach to another facet of sample design, i.e., sample size.

 


How large a sample to use is a decision that qualitative researchers need to make explicitly and carefully in order to increase the likelihood that their studies will generate credible data by well representing their population of interest. Unlike quantitative researchers who most often rely on statistical formulae to determine the sample sizes for their studies, qualitative researchers must rely on (a) past experience and knowledge of the subject matter; and (b) ongoing monitoring during the data-gathering period, which includes applying a set of decision rules, such as those listed in “Designing a Quality In-depth Interview Study: How Many Interviews Are Enough?” These decision rules consider (a) the complexity of the phenomena being studied, (b) the heterogeneity or homogeneity of the population being studied, (c) the level of analysis and interpretation that will be carried out, and (d) the finite resources available to support the study. These types of decision guidelines, along with past experience, should provide qualitative researchers with the considerations they need to carefully judge the amount of data necessary to meet their research objectives. (Of note, if a researcher does not have sufficient past personal experience, a literature review, or speaking directly with other researchers who do have such experience, should serve well.)

As importantly, during the period when data are being gathered, researchers should also closely monitor the amount of variability in the data, compared to the variability that was expected, for the key measures of the study. Based on this monitoring, researchers are responsible for making a “Goldilocks decision” about whether the sample size they originally decided was needed is too large, too small, or just about right. In making a decision to cut back on the amount of data to be gathered, because there is less variability in what is being measured than anticipated, the researcher needs to make certain that those cases that originally were sampled, but would be dropped, are not systematically different from the cases from which data will be gathered. In making a decision to increase the size of the sample, because there is more variability in what is being measured than anticipated, the researcher needs to make certain that the cases added to the sample are chosen in a way that is representative of the entire population (e.g., using the same orderly approach that was used to create the initial sample).

In all instances, and if the necessary resources (staff, time, budget) are available, it is prudent for a researcher to error on the side of having more rather than less data. Gathering too much data does no harm to the quality of the study’s findings and interpretations, but having too little data leaves the researcher in the untenable position of harming the quality of the study because the complexity of what was being studied will not be adequately represented in the available data. For example, case study research to investigate new public school policies related to the core science curriculum might include in-depth interviews with school principals and science teachers, observations of science classes in session, and a review of students’ test papers; however, as a complex subject matter, the research may be weakened by not including discussions with the students and their parents as well as by a failure to include all schools (or a representative sample of schools) in the research design.

Roller, M. R., & Lavrakas, P. J. (2015). Applied qualitative research design: A total quality framework approach. New York: Guilford Press.

A TQF Approach to Choosing a Sample Design

The Total Quality Framework (TQF) offers qualitative researchers a way to think critically about their research Credibility TQF componentdesigns and helps to guide their decision making. The TQF consists of four components, with each component devoted to the critical thinking considerations associated with a phase in the research process. The first component of the TQF is Credibility which is focused on data collection; specifically, Scope and Data Gathering. One of the many considerations related to Scope has to do with the sample design.

The following is a modified excerpt from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 25-26) on the different aspects of sampling that researchers might want to think about as they develop their qualitative research designs.


Once the researcher has identified the list (or lists) that will be used to select the sample, a decision must be made about which sampling approach will be used. If the decision is to gather data from each member of the population on the list (e.g., all 20 students enrolled in an honors science class), then there is nothing more for the researcher to consider. But for those studies where something less than the entire population will be chosen for study, additional Total Quality Framework (TQF) decisions need to be made about sampling.

Here, qualitative researchers may needlessly lessen the quality of their studies by not giving these decisions sufficient consideration. In fact, some qualitative researchers may think that how they create a sample of the population is unimportant. Qualitative researchers may proceed in this manner because they mistakenly believe that systematic sampling is too hard to carry out (i.e., too complex, too expensive, and too time-consuming) and that it is “too quantitative” a concern. Yet, in the vast majority of qualitative studies, systematic sampling is neither complex, expensive, nor time-consuming, and should not only be a quantitative issue. And by using an organized approach for choosing which members of their key population to study, as opposed to merely using a convenient and disorderly approach to sampling, qualitative researchers avoid a major threat to the credibility of the data they gather. That threat is the possibility that those from whom they gather data are not, in fact, representative (do not share defining characteristics) of the population being studied.

Take, for example, a focus group researcher that has a list of men and women who completed a cardiopulmonary resuscitation (CPR) training class in the past year. The researcher can choose one of two basic approaches to selecting those who will be invited to participate in a group discussion. The often-used but misguided approach is to start at the top of the list and contact people, one after another, until the focus groups have been filled with ostensibly willing attendees. The rigorous and correct approach is to use an organized scheme to sample CPR class graduates from across the entire list (i.e., stratifying the list and taking an ‘nth’ name approach). The second approach is preferred because it avoids the possible problem that the names on the list are ordered in a way that is not representative of the entire population of CPR graduates that the researcher wants to study.

A final TQF issue related to choosing a sample applies to qualitative studies that utilize observations of naturally occurring human behavior to gather data, such as in ethnographic research. In these studies, sampling considerations need to be applied to the times and the locations during which the behaviors of interest will be observed. By systematically choosing which locations and which times to conduct the observations—among all possible locations and times in which the behaviors of interest will be taking place—the qualitative researcher is greatly raising the likelihood that the observations included in the study are a representative subset of all the possible behaviors of interest to the study.

Roller, M. R., & Lavrakas, P. J. (2015). Applied qualitative research design: A total quality framework approach. New York: Guilford Press.