In March 2018, Mario Luis Small gave a public lecture at Columbia University on “Rhetoric and Evidence in a Polarized Society.” In this terrific must-read speech, Small asserts that today’s public discourse concerning society’s most deserving issues – poverty, inequality, and economic opportunity – has been seriously weakened by the absence of “qualitative literacy.” Qualitative literacy has to do with “the ability to understand, handle, and properly interpret qualitative evidence” such as ethnographic and in-depth interview (IDI) data. Small contrasts the general lack of qualitative literacy with the “remarkable improvement” in “quantitative literacy” particularly among those in the media where data-driven journalism is on the rise, published stories are written with a greater knowledge of quantitative data and use of terminology (e.g., the inclusion of means and medians), and more care is given to the quantitative evidence cited in media commentary (i.e., op-eds).
Small explains that the extent to which a researcher (or journalist or anyone involved in the use of research) possesses qualitative literacy can be determined by looking at the person’s ability to “assess whether the ethnographer has collected and evaluated fieldnote data properly, or the interviewer has conducted interviews effectively and analyzed the transcripts properly.” This determination serves as the backbone of “basic qualitative literacy” which enables the research user to identify the difference between a rigorous qualitative study and Read Full Text
There are lots of articles discussing question design, focusing on such things as how to mitigate various forms of bias, clearly communicate the intended meaning of the question, and facilitate response. Survey question wording is discussed in this “tip sheet” from Harvard University as well as in “Questionnaire Design” from Pew Research Center, and a recent article in Research Design Review discussed the not-so-simple “why” question in qualitative research (see “Re-considering the Question of ‘Why’ in Qualitative Research”).
Getting the question “right” is a concern of all researchers, but qualitative researchers have to be particularly mindful of the responses they get in return. It is not good enough to use an interview guide to ask a question, get an answer, and move on to the next question. And, it is often not good enough to ask a question, get an answer, interject one or two probing questions, and move on to the next question. Indeed, one of the toughest skills a qualitative interviewer has to learn is how to evaluate a participant’s answer to any given question. This goes way beyond evaluating whether the participant responded in line with the intention of the question or the potential sources of bias. Rather, this broader, much-needed evaluation of a response requires a reflexive, introspective consideration on the part of the interviewer.
Observational research is “successful” to the extent that it satisfies the research objectives by capturing relevant events and participants along with the constructs of interest. Fortunately, there are two tools – the observation guide and the observation grid – that serve to keep the observer on track towards these objectives and generally facilitate the ethnographic data gathering process.
Not unlike the outlines interviewers and moderators use to help steer the course of their in-depth interviews and group discussions, the observation guide serves two important purposes: 1) It reminds the observer of the key points of observation as well as the topics of interest associated with each, and 2) It acts as the impetus for a reflexive exercise in which the observer can reflect on his/her own relationship and contribution to the observed at any moment in time (e.g., how the observer was affected by the observations). An observation guide is an important tool regardless of the observer’s role. For each of the five observer roles* – nonparticipant (off-site or on-site) and participant (passive, participant-observer, or complete) observation – the observation guide helps to maintain the observer’s focus while also giving the observer leeway to reflect on the particular context associated with each site.
As an adjunct to the observation guide, it is recommended that ethnographic researchers also utilize an observation grid. The grid is similar to the guide in that it helps remind the observer of the events and issues of most import; however, unlike the guide, the observation grid is a spreadsheet or log of sorts that enables the observer to actually record (and record his/her own reflections of) observable events in relationship to the constructs of interest. The grid might show, for instance, the relevant constructs or research issues as column headings and the specific foci of observation as rows. In an observational study of train travel, for example, the three key research issues related to activity at the train station might be: waiting for departures, delays in departures, and boarding; and the key areas of observation would pertain to behavior, conversations heard, and contextual information such as the weather and the general mood. Like the guide, the observation grid not only ensures that the principal issues and components are captured but also encourages the observer to reflect on each aspect of his/her observations and identify the particular ways the observer is influencing (or is being influenced by) the recorded observations.