Cognitive interviewing is a method used by survey researchers to investigate the integrity of their questionnaire designs prior to launching the field portion of the study. In the edited volume Cognitive Interviewing Methodology, Kristen Miller (2014) describes cognitive interviewing as “a qualitative method that examines the question-response process, specifically the processes and considerations used by respondents as they form answers to survey questions,” further explaining that “through the interviewing process, various types of question-response problems that would not normally be identified in a traditional survey interview, such as interpretive errors and recall accuracy, are uncovered” (p. 2). In this way, survey researchers identify the users’ (i.e., survey respondents’) possible meaning and interpretation of survey questions – having to do with question structure or format and terminology – that may or may not deviate from the researcher’s intent. Importantly, the objective of the cognitive interview is not to simply determine whether a questionnaire item “makes sense” to an individual  but to go beyond that to explore the individual’s lived experience (personal context, attitudes, perceptions, behavior) in relationship to their interpretation and/or ability to answer a particular question.

Although not typically included under the “qualitative research” umbrella (with in-depth interviewing, focus group discussions, and observation), four of the 10 unique attributes associated with qualitative research are notably relevant to the cognitive interviewing method. They are the: importance of meaning, flexibility of design, participant-researcher relationship, and researcher skill set. These distinctive qualities of the cognitive interviewing method, and qualitative methods generally, define why researchers opt for qualitative research, yet these same qualities pose a real challenge to researchers who seek to maintain the integrity of the method while also integrating best practices to arrive at valid data. It is by way of best practices and valid data that cognitive interviewing achieves its ultimate goal of mitigating survey measurement error.

The Total Quality Framework (TQF) is an approach to integrating best practices into qualitative research design and, by association  (as discussed above), the cognitive interviewing method. By way of its four interlocking components – Credibility, Analyzability, Transparency, and Usefulness – the framework offers researchers a way to think critically about each phase of the research process. Credibility, for instance, has to do with the data collection research phase; specifically, aspects of the research related to Scope and Data Gathering. One of the considerations related to the Scope of a cognitive interviewing study is the number of interviews to complete, which is often considered in terms of saturation or interviewing until no new insights are forthcoming. However, as discussed in this RDR article on the in-depth interview method, saturation, from a TQF perspective, doesn’t go far enough. As with in-depth interviews, there are quality aspects relevant to the conduct of cognitive interviews that need to be considered when determining sample size, such as the coverage of questionnaire items in each interview as well as the interviewers’ skills, objectivity, and consistency (e.g., the consistent manner in which interviewers introduce each interview). These and other considerations are listed below.

In addition to Scope, the Data Gathering component of the TQF offers researchers a way to think about how to mitigate researcher and participant effects in the data gathering process. Because the researcher’s qualitative skills are critical to the integrity of this process, it is imperative to the quality of cognitive interviews that they be conducted by a trained qualitative researcher. A qualitative interviewer is not only an expert in administering effective probing questions (i.e., questions that reveal the underlying meaning of each participant’s reactions or comments) but also astute to the potential for participant effects. The ability to weaken these effects relies partly on the interviewer’s skill in recognizing the ways participants may not respond honestly to the interviewer’s questions or respond in an unclear manner, such as participants who exhibit one or more of the following.

These and other considerations when applying best practices to the cognitive interviewing method can be found in the presentation “A Best Practices Approach to Cognitive Interviewing” available here. As another discussion of a rigorous approach to cognitive interviewing, the edited volume mentioned earlier — Cognitive Interviewing Methodology — is also recommended.

 

Miller, Kristen (2014). Introduction. In Kristen Miller, Stephanie Willson, Valerie Chepp, & José Luis Padilla (Eds.). Cognitive interviewing methodology (pp. 1-5). John Wiley & Sons.