Research Design & Methods

Research Quality & the Impact of Monetary Incentives

The following is adapted from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 78-79).

Monetary incentives & quality researchGaining cooperation from research participants and respondents is important to the quality of qualitative and survey research. A focus on gaining cooperation helps to mitigate potentially weakened data due to the possibility that the individuals who do not cooperate — do not participate in the research — differ in meaningful ways compared to those who do cooperate. As mentioned in an article posted earlier in Research Design Review, an effective component to the researcher’s strategy for gaining cooperation among participants is the offer of material incentives (e.g., cash, a gift card, prized tickets to a sporting event, donation to a favorite charity).

Although monetary incentives are routinely given to qualitative research participants to boost cooperation, the researcher needs to keep in mind that the offer of a cash (or equivalent) incentive may also jeopardize the quality of the actual focus group discussion, in-depth interview, or observation. The following is one example of how monetary incentives may have the unwanted effect of skewing participants’ responses in an in-depth interview (IDI) study.

Cook and Nunkoosing (2008) conducted an in-person IDI study with 12 “impoverished elders” in Melbourne, Australia to investigate community services for the poor among those “who are excluded or at risk of exclusion from their communities.” Research participants could participate in up to two interviews and were given $20 for each interview.

In reviewing the key findings, the researchers observed many “interview interactions that were atypical.” At least part of these irregularities was attributed to the monetary incentive which, according to Cook and Nunkoosing, helped to create an interview environment where interviewees were motivated “to manage the presentation of self, retain control over the exchange of information, and reduce the stigma of poverty by limiting disclosure and resisting researcher questioning” (p. 421).

The importance of the incentive in the interview process became clear when interviewees volunteered comments such as “I need the $20 . . . ” and critically compared the $20 to better (i.e., higher) cash incentives offered by other research studies. In this way interviewees were in effect “selling” their stories to the interviewer (and, some would say, at a bargain price) which, based on the researchers’ analyses, tainted interviewees’ responses with “stylized accounts” (or “rehearsed narratives”) as well as “minimal disclosure,” as seen in this excerpt from the transcripts (p. 424):
Participant: What did you want to know?
Interviewer: All about you.
Participant: That’s about it, like, there’s not too much.
Interviewer: Do you want to tell me a bit more? I don’t really know who you are yet.
Participant: You do.
Interviewer: Tell me a bit about who you are, what you like, what you don’t like.
Participant: I don’t like him [Gesturing toward the other agency client].

This dialog came towards the end of a 30-minute interview and helps to illustrate “the researcher’s frustration at [their] inability to engage the participant in in-depth discussion” (p. 423).

Research design is always a balancing act involving various trade-offs associated with meeting the key objectives, method(s) and strategy for engaging the target population(s), and the efficient use of available resources. An important researcher skill is understanding the implications of these trade-offs to the integrity of the final data and overall quality of the research investigation. A monetary incentive may be highly effective in securing participation in our research but what is its ultimate impact on data quality? This is the concern of a skilled researcher.

Cook, K., & Nunkoosing, K. (2008). Maintaining dignity and managing stigma in the interview encounter: The challenge of paid-for participation. Qualitative Health Research, 18(3), 418–427. https://doi.org/10.1177/1049732307311343

Qualitative Analysis: A Reflexive Exercise for Category Development

The second component of the Total Quality Framework (TQF) is Analyzability. This component provides researchers with critical thinking considerations relevant to the completeness and accuracy of their analyses and interpretations of the data. Analyzability consists of two fundamental elements — processing and verification — the first of which involves coding followed by deriving categories and themes from the data.

From a TQF perspective, a useful exercise for category development — particularly when the study entails multiple researchers and a large amount of data — is by way of the reflexive template. Although similar in spirit to the writing function in computer-assisted qualitative data analysis software programs, the primary purpose of this reflexive template is to encourage researchers to actively reflect as they go about developing categories or buckets from the underlying constructs gained from the data. By way of the template, the analyst can document the relationship they perceive between the category and the construct as well as provide an example or further input to support their thinking.

For instance, a researcher conducting a qualitative content analysis study of diaries written by women confined to prison concerning their activities and experiences during confinement, may have derived the category “educational opportunity” (EDUOPPTY) from the coded data defined in part (i.e., along with other relevant constructs) by the underlying construct “well-being.” Within the well-being construct, the researcher also identified three key subconstructs — physical well-being, mental well-being, and financial well-being — that play a central role in understanding the meaning of the well-being construct as well as deepening the definition of the EDUOPPTY category. In this example, the reflexive exercise (by way of the template, see below) has facilitated the researcher’s ability to record the connections between the category and key constructs — highlighting instances of the relationship between EDUOPPTY (e.g., how to use the exercise equipment and art classes) and physical well-being, mental well-being, as well as financial well-being — while aiding collaboration with the research team and adding transparency to the analysis process.

Reflexive template for category development

Analyzability & a Qualitative Content Analysis Case Study

The following is a modified excerpt from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 284-285).

Kuperberg and Stone (2008) present a case study where content analysis was used as the primary research method. Gender & SocietyIt is an example of how many of the Total Quality Framework (TQF) concepts can be applied — not only to the in-depth interview, focus group, observation, and case centered methods, discussed elsewhere in Research Design Review, but — to qualitative content analysis. The discussion below spotlights aspects of this study relevant to one of the four TQF components, Analyzability.

Purpose & Scope
The primary purpose of this content analysis study was to extend the existing literature on the portrayal of women’s roles in print media by examining the imagery and themes depicted of heterosexual college-educated women who leave the workforce to devote themselves to being stay-at-home mothers (a phenomenon referred to as “opting out”) across a wide, diverse range of print publications. More specifically, this research set out to investigate two areas of media coverage: the content (e.g., the women who are portrayed in the media and how they are described) and the context (e.g., the types of media and articles).

This study examined a 16-year period from 1988 to 2003. This 16-year period was chosen because 1988 was the earliest date on which the researchers had access to a searchable database for sampling, and 2003 was the year that the term “opting out” (referring to women leaving the workforce to become full-time mothers) became popular. The researchers identified 51 articles from 30 publications that represented a wide diversity of large-circulation print media. The researchers acknowledged that the sample “underrepresents articles appearing in small-town outlets” (p. 502).

Analyzability
There are two aspects of the TQF Analyzability component — processing and verification. In terms of processing, the content data obtained by Kuperberg and Stone from coding revealed three primary patterns or themes in the depiction of women who opt out: “family first, child-centric”; “the mommy elite”; and “making choices.” The researchers discuss these themes at some length and support their findings by way of research literature and other references. In some instances, they report that their findings were in contrast to the literature (which presented an opportunity for future research in this area). Their final interpretation of the data includes their overall assertion that print media depict “traditional images of heterosexual women” (p. 510).

Important to the integrity of the analysis process, the researchers absorbed themselves in the sampled articles and, in doing so, identified inconsistencies in the research outcomes. For example, a careful reading of the articles revealed that many of the women depicted as stay-at-home mothers were actually employed in some form of paid work from home. The researchers also enriched the discussion of their findings by giving the reader some context relevant to the publications and articles. For example, they revealed that 45 of the 51 articles were from general interest newspapers or magazines, a fact that supports their research objective of analyzing print media that reach large, diverse audiences.

In terms of verification, the researchers performed a version of deviant case analysis in which they investigated contrary evidence to the assertion made by many articles that there is a growing trend in the proportion of women opting out. Citing research studies from the literature as well as actual trend data, the researchers stated that the articles’ claim that women were increasingly opting out had weak support.

Kuperberg, A., & Stone, P. (2008). The media depiction of women who opt out. Gender & Society, 22(4), 497–517.