Data Gathering

Gathering Quality Ethnographic Data: 3 Key Considerations

The following is a modified excerpt from Applied Qualitative Research Design: A Total Quality Framework Approach (Roller & Lavrakas, 2015, pp. 204-206).

Data Gathering is one of two broad areas of the Total Quality Framework Credibility component that affects all qualitative research, incEthnography peacockluding ethnographic research. There are three primary aspects concerning the gathering of data in ethnography that require serious consideration by the researcher in the development of the study design. To optimize the measurement of ethnographic data, and hence the quality of the outcomes, researchers need to pay attention to:

  • How well the observers have identified and recorded all the information (e.g., verbal and nonverbal behavior, attitudes, context, sensory cues) pertinent to the research objectives and constructs of interest. A well-developed observation guide and observation grid can assist greatly in this effort. Not unlike the development of an in-depth interview or discussion guide, the ethnographer seeks to identify those observable events—including the specific individuals (or types of individuals), the verbal and nonverbal behaviors, attitudes, sensory and other environmental cues—that will further the researcher’s understanding of the issues. During the design development phase, the researcher might isolate the observations of interest by:
    • Looking at earlier ethnographic research on the subject matter and/or with similar study populations.
    • Interviewing the clients or those who have requested the research to learn everything they know about the topic and   their past work in the area.
    • Consulting the literature or other experts concerning the behaviors and other occurrences associated with particular constructs.
    • “Shagging around” (LeCompte & Goetz, 1982) the observation site(s) to casually assess the environment and begin to learn about the participants.

 

  • Observer effects, specifically—
    • Observer bias, that is, behavioral and other characteristics (e.g., personal attitudes, values, traits) of the observer that may alter the observed event or bias their observations. For example, an observer as a complete participant would bias the observational data if there was an attempt to “educate” participants on a subject matter for which the observer had personal expertise or knowledge.
    • Observer inconsistency, that is, an inconsistent manner in which the observer conducts the observations that creates unwarranted and unrepresentative variation in the data. For example, an on-site nonparticipant observer conducting in-home observations of the use of media and technology would be introducing inaccuracies in the data by observing and recording the use of television and gaming in some households but not in others where television and gaming activities took place.

 

  • Participant effects, specifically, the extent to which observed participants alter a naturally occurring event, leading to biased outcomes. This is often called the Hawthorne effect, whereby the people being observed, either consciously or unconsciously, change what is being measured in the observation because they are aware of the observer. For example, an ethnographer conducting an overt, on-site passive observation of teaching practices in a school district would come away with misleading data if one or more school teachers deviated from their usual teaching styles during the observations in order to more closely conform with district policies.

 

LeCompte, M. D., & Goetz, J. P. (1982). Ethnographic data collection in evaluation research. Educational Evaluation and Policy Analysis, 4(3), 387–400.

Roller, M. R., & Lavrakas, P. J. (2015). Applied qualitative research design: A total quality framework approach. New York: Guilford Press.

Qualitative Tech Solutions: Coverage & Validity Considerations

Back in 2018, Research Design Review posted an article titled “Five Tech Solutions to Qualitative Data Collection: What Strengthens or Weakens Data Quality?” The focus of this article is on a presentation given in May 2018 concerning technological alternatives TQF Credibilityto qualitative research data collection. Importantly, the aim of the presentation was, not to simply identify different approaches to data collection beyond the in-person and telephone modes but rather, to examine the strengths and limitations of these technological solutions from a data quality – specifically, Credibility – standpoint.

Broadly speaking, technological approaches to qualitative research data gathering offer clear advantages over in-person methods, particularly in the areas of:

  • Representation, e.g., geographic coverage, potential access to hard-to-reach population segments;
  • Cooperation, e.g., convenience and flexibility of time and place for participants, appropriateness for certain demographic segments (18-49 year olds*);
  • Validity associated with data accuracy, e.g., research capturing in-the-moment experiences do not rely on memory recall;
  • Validity associated with the depth of data, e.g., capturing multiple contextual dimensions through text, video, and images;
  • Validity associated with data accuracy and depth allowing for the triangulation of data;
  • Researcher effects, e.g., mitigated by the opportunity for greater reflection and consistency across research events;
  • Participant effects, e.g., mitigated by the multiple ways to express thoughts, willingness to discuss sensitive issues, and (possibly) a lower tendency for social desirability responding; and
  • Efficient use of resources (i.e., time, money, and staff).

There are also potential drawbacks to any technological solution, including those associated with:

  • Uneven Internet access and comfort with technology among certain demographic groups (e.g., sampling favors “tech savvy” individuals), hard-to-reach and marginalized segments of the population;
  • Difficulty in managing engagement, including the unique researcher skills and allocation of time required;
  • Potential participant burnout from researcher’s requests for multiple input activities and/or days of engagement. This is a type of participant effect that negatively impacts validity;
  • Nonresponse due to mode, e.g., unwillingness or inability to participate to a mostly text-based discussion;
  • Data accuracy, e.g., participant alters behavior in a study observing in-home meal preparation;
  • Missing important visual &/or verbal cues which may interfere with rapport building and an in-depth exploration of responses;
  • Difficulty managing analysis due to lots and lots of data (in volume & formats);
  • Fraud, misrepresentation – “Identity is fluid and potentially multiple on the Internet” (James and Bushner, 2009, p. 35) and people may not share certain images or video that reveal something “embarrassing” about themselves**; and
  • Security, confidentiality, anonymity (e.g., data storage, de-identification).

 

 

* https://www.pewresearch.org/internet/fact-sheet/internet-broadband/

** https://www.businesswire.com/news/home/20180409006050/en/Minute-Maid-Debuts-New-Campaign-Celebrates-Good

James, N., & Busher, H. (2009). Online interviewing. London: Sage Publications.

Five Tech Solutions to Qualitative Data Collection: What Strengthens or Weakens Data Quality?

Qualitative researchers have increasingly new ways to engage with their participants. Beyond the traditional and still most frequent approach of the in-person mode, qualitative researchers have a host of technological solutions at their disposal. Instead of in-person focus group discussions, for instance, the researcher might opt for asynchronous focus groups. Or rather than in-person multiple methods qualitative research, the researcher might design an all-tech solution that blends online observation with asynchronous groups or any one of several technological options for the in-depth interview method such as mobile video or the email IDI.

The following is a presentation given at the2018 annual conference of the American Association for Public Opinion Research. This presentation discusses five tech solutions to qualitative research data collection with particular consideration given to the aspects of these approaches that strengthen or weaken data quality. These quality considerations are discussed from the perspective of the Total Quality Framework and, specifically, the Credibility component which has to do with qualitative data collection.

An in-person explanation of slides is always preferred over simply combing through a slide deck. For commentary or answer to your questions, please feel free to use this form to contact me.