Total Quality Framework

Shared Constructs in Research Design: Part 3 — Validity

validity in research designNot unlike Part 1 (concerning sampling) and Part 2 (concerning bias) of the discussion that began earlier, the shared construct of validity in research design has also been an area of focus in several articles posted in Research Design Review. Most notable is “Quality Frameworks in Qualitative Research” posted in February 2021 in which validity is discussed within the context of the parameters or strategies various researchers use to define and think about the dimensions of rigor in qualitative research design. This article uses the Total Quality Framework (Roller & Lavrakas, 2015) and criteria of Lincoln and Guba (1985) to underscore the idea that quality approaches to design cuts across paradigm orientation, leading to robust and valid interpretations of the data.

Many other qualitative researchers, across disciplines, believe in the critical role that the shared construct of validity plays in research design. Joseph Maxwell, for example, discusses validity in association with his realism approach to casual explanation in qualitative research (Maxwell, 2004) and discusses in detail five unique dimensions of validity, including descriptive, interpretative, and theoretical validity (Maxwell, 1992). And of course, Miles & Huberman were promoting greater rigor by way of validity more than three decades ago (Miles & Huberman, 1984).

More recently, Koro-Ljungberg (2010) takes an in-depth look at validity in qualitative research and, with extensive literature as the backdrop, makes the case that “validity is in doing, as well as its (un)making, and it exhibits itself in the present paradox of knowing and unknowing, indecision, and border crossing” (p. 609). Matteson & Lincoln (2008) remind educational researchers that validity does not solely concern the analysis phase of research design but “the data collection method must also address validity” (p. 672). Creswell & Miller (2000) discuss different approaches to determine validity across three paradigm orientations — postpositivist, constructivist, and critical — and “lens” of the researcher, participants, and researchers external to the study.

Among qualitative health researchers, Morse (2020) emphasizes the potential weakness in validity when confusing the analysis of interpretative inquiry with that associated with “hard, descriptive data” (p. 4), and Morse et al. (2002) present five verification strategies and argue that validity (as well as reliability) is an “overarching” construct that “can be appropriately used in all scientific paradigms” (p. 19).

These researchers, and those discussed in Part 1 – Sampling and Part 2 – Bias, are admittedly a small share of those who have devoted a great deal of thought and writing concerning these shared constructs. The reader is encouraged to utilize these references to build on their understanding of these constructs in qualitative research and to grow their own library of knowledge.

 

Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into Practice, 39(3), 124–130.

Koro-Ljungberg, M. (2010). Validity, responsibility, and aporia. Qualitative Inquiry, 16(8), 603–610. https://doi.org/10.1177/1077800410374034

Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage Publications.

Matteson, S. M., & Lincoln, Y. S. (2008). Using multiple interviewers in qualitative research studies: The influence of ethic of care behaviors in research interview settings. Qualitative Inquiry, 15(4), 659–674. Retrieved from http://qix.sagepub.com/cgi/doi/10.1177/1077800408330233

Maxwell, J. A. (1992). Understanding and validity in qualitative research. Harvard Educational Review, 62(3), 279–300.

Maxwell, J. A. (2004). Casual explanation, qualitative research, and scientific inquiry in education. Educational Researcher, 33(2), 3–11.

Miles, M. B., & Huberman, A. M. (1984). Drawing valid meaning from qualitative data: Toward a shared craft. Educational Researcher, 13(5), 20–30. https://doi.org/10.3102/0013189X013005020

Morse, J. (2020). The changing face of qualitative inquiry. International Journal for Qualitative Methods, 19, 1–7. https://doi.org/10.1177/1609406920909938

Morse, J. M., Barrett, M., Mayan, M., Olson, K., & Spiers, J. (2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods, 1(2), 13–22.

Roller, M. R., & Lavrakas, P. J. (2015). Applied qualitative research design: A total quality framework approach. New York: Guilford Press.

Evaluating Proposals Using the Total Quality Framework

The following is a modified excerpt from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 345-346)

In addition to using the Total Quality Framework (TQF) to structure more rigorous and comprehensive research proposals, the TQF can be used by anyone who is evaluating a proposal for a research study that will use qualitative methods (e.g., members of a thesis or dissertation committee, funders at a granting agency or foundation, clients in the commercial sector). A TQF approach to evaluating research proposals effectively holds the proposal author(s) accountable for doing research that is likely to be accurate and, in the end, useful. The TQF provides a comprehensive system to methodically think about the strengths and limitations of the proposed study design and helps the reviewer ascertain whether there are outstanding threats to the quality of the proposed research that have been ignored or remain unanticipated by the researcher(s).

In essence, the TQF is a reminder to proposal evaluators that research integrity built around fundamental principles is equally important in qualitative as it is in quantitative research design.

The TQF criteria to be considered in the proposal review, within each of the four TQF components, are the following:

Credibility

• How the target population has been defined.
• How the list representing the target population will be created.
• How the sample of participants will be chosen from the list(s) that will be used.
• How many participants the researcher proposes to gather data from or about and the justification that is provided for this number, including its adequacy for the purposes of the study; a discussion of how the researcher will monitor and judge the adequacy of this number while in the field should also be included.
• How the researcher will gain cooperation from, and access to, the sampled participants.
• How the researcher will determine if those in the sample from whom data was not gathered differ in critical ways on the topics being studied from those participants who did provide data.
• What the researcher will do to account for the potential bias that may exist because not everyone in the sample participated in the research (i.e., no data was gathered from some individuals).
• The extent to which the relevant concepts that will be studied have been identified.
• How the researcher has operationalized these concepts in order to effectively collect data on them in the research approach.
• How the researcher has articulated and supported the research objectives and questions.
• How the data collection method(s) will be pilot-tested and revised as necessary.
• The precautions that will be taken to minimize (or at least better understand) the potential biases and inconsistencies that might be created in the data by those involved in data collection.
• The precautions that will be taken to assure high ethical standards throughout the entire study. Read Full Text

Quality Frameworks in Qualitative Research

The following is a modified excerpt from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 20-21)

Many researchers have advanced strategies, criteria, or frameworks for thinking about and promoting the importance of “the quality” of qualitative research at some stage in the research design. There are those who focus on quality as it relates to specific aspects—such as various validation and verification strategies or “checklists” (Barbour, 2001; Creswell, 2013; Brinkmann & Kvale, 2015; Maxwell, 2013; Morse et al., 2002), validity related to researcher decision making (Koro-Ljungberg, 2010) and subjectivity (Bradbury-Jones, 2007), or the specific role of transparency in assessing the quality of outcomes (Miles, Huberman, & Saldaña, 2014). There are others who prescribe particular approaches in the research process—such as consensual qualitative research (Hill et al., 2005), the use of triangulation (Tobin & Begley, 2004), or an audit procedure (Akkerman, Admiraal, Brekelmans, & Oost, 2006). And there are still others who take a broader, more general view that emphasizes the importance of “paying attention to the qualitative rigor and model of trustworthiness from the moment of conceptualization of the research” (Thomas & Magilvy, 2011, p. 154; see also, Bergman & Coxon, 2005; Whittemore et al., 2001).

The strategies or ways of thinking about quality in qualitative research that are most relevant to the Total Quality Framework (TQF) are those that are (a) paradigm neutral, (b) flexible (i.e., do not adhere to a defined method), and (c) applicable to all phases of the research process. Among these, the work of Lincoln and Guba (e.g., 1981, 1985, 1986, and 1995) is the most noteworthy. Although they profess a paradigm orientation “of the constructionist camp, loosely defined” (Lincoln et al., 2011, p. 116), the quality criteria Lincoln and Guba set forth more than 35 years ago is Read Full Text