These 13 factors represented a first start towards thinking about qualitative research design within a quality framework.  This thinking will evolve over time, with additions and subtractions made as warranted.  One such needed adjustment is in the critical yet missing area of participant recruitment.  This omission was rightly pointed out in a recent comment to the blog by Andrea Lombardi, a qualitative research consultant in Milan.  As Andrea states, “…recruitment quality [is a] main factor for a good quality FG/IDI…”  In appreciation of the important role that the recruiting process plays in the ultimate utility and effectiveness of our qualitative research, here is an amended – what is now 18-factor – classification scheme incorporating five factors specific to the recruiting process:

18 Factors Impacting the Quality of Qualitative Research

The Recruiting Process

Potential variability associated with the:

• Group composition (e.g., level of demographic and product-use diversity)
• Sample frame & sampling
• Design of screener questions & interview questionnaire overall
• Recruitment interviewer (e.g., professionalism as well as gender, ethnicity, personality aspects)
• Recruiting-process standards (e.g., call-back and confirmation protocols)

The Environment

Potential variability associated with the:

• Particular venue/setting (incl., face-to-face and online)
• Presence of observers/interviewers as well as other participants (e.g., groups vs. IDIs)
• Audio & video recording

The Dynamics

Potential variability associated with:

• Professional participants (“cheaters”)
• Participants’ cultural/social/economic/gender/age diversity
• Cognitive processes/constructs
• Geographic/regional differences
• Dominators, group vs. individual think

The Interviewer/Moderator

Potential variability associated with the:

• Personal/personality aspects of the interviewer/moderator
• “Best” techniques utilized for specific topics, type of participants, venue
• Question formatting
• Question sequencing
• Use of projective techniques (e.g., what to use when, impact on the discussion overall, analytical schemes)

In February 2012, Gallup posted the results of a survey question that asked registered voters about the importance of various issues on their decision to vote for one presidential candidate over another.  The interviewer stated that he/she would be reading a list of issues and then instructed the respondent to

“…please tell me how important the candidates’ positions on that issue will be in influencing your vote for president – extremely important, very important, somewhat important, or not important.”

Similarly, the Kaiser Family Foundation’s February 2012 Kaiser Health Tracking Poll asked about the importance of health-related issues to the voting decision:

“…tell me how important [each of the following issues] will be to your vote. Would you say this issue will be extremely important to your vote for president, very important, somewhat important, or not too important to your vote?”

SurveyMonkey and others on the Web confuse things further by promoting a double-whammy scale effect with potential duplication at both the top and middle levels.  For instance, the SurveyMonkey “Market Research Template” recommends the following question:

5. If our new product were available today, how likely would you be to use it instead of competing products currently available from other companies?

○ Extremely likely

○ Very likely

○ Moderately likely

○ Slightly likely

○ Not at all likely

M/A/R/C Research in its “Best Practices for Constructing Quantitative Rating Scales that Minimize Scale Use Bias” states  that “anchors should be distinctly clear in meaning and create inter-scale-point ‘mentally interpreted distance’ that are as equal as possible.”  It uses the example of “somewhat” and “moderately,” asserting that these are not distinctly different scale points and their use will ultimately lead to “scale use bias.”

Much of this “bias” can be attributed to the burden that questions (as those above) pose to respondents as well as to researchers.  Asking respondents to ascertain the difference between “extremely” and “very” or “moderately” and “slightly” becomes a mental challenge as survey takers work their way through the four-step cognitive process:  1) interpreting the question to deduce the intended distinction between terms; 2) searching the mind for relevant information; 3) integrating that information into a judgment; and, 4) translating that judgment into a response.  This cognitive burden is exacerbated by the potentially damaging effect of adding to the perceived length of the survey which in turn increases the likelihood of breakoffs.

Likewise, the researcher is left with the burden of interpreting survey responses which, of course, are a product of respondents’ interpretations of confusing terminology.  Given the evidence that people generally don’t agree on what “very” anything – “very likely,” “very satisfied,” “very important” – means, it seems unreasonable to assume that the analyst can reach a realistic conclusion from extremely-very and moderately-slightly data.   Unless, of course, the analyst relies on his/her own interpretation of the terms which, of course, is meaningless since the responses belong to respondents not researchers.

It is easy to argue that interpretation (and potential for misinterpretation) is an issue in any question design, but the unnecessary use of duplicating terms presents an added layer of burden – to both respondents and researchers – that should be avoided in good research design.

What I find particularly interesting in the Grapentine-Teas plea for a theory-based approach to marketing research is the defense they wage in support of their argument – specifically, the idea that “theory expands knowledge sources.”  What I like about this is that, by embracing the research tools and methods from various related disciplines, such as sociology, marketing research design can bring an elevated, “holistic understanding” to our studies.  And this is good because a more-encompassing way of designing marketing research addresses the fundamental objective of our research, which is to understand how people think and what motivates behavior.

Grapentine and Teas also talk about the potential “barriers” to their redefinition proposition; highlighting the anticipated negative commentary that a theoretical approach in marketing research is too-academic and/or too-expensive for the speed-over-quality mentality among many marketing researchers.  This indeed may spell doom for their effort, but their cry for a more scientific basis to our marketing research designs, even with the acknowledgement that compromise – between true scientific rigor and the reality of research in the corporate world – is inevitable, is very welcomed.

In June 2011 I wrote a blog post where, not unlike Grapentine and Teas, I argue for “Taking Research Design to Higher Ground” and wonder “why researchers continue with the long-standing habit of avoiding honest experimentation and debates regarding their research methods.”  I conclude:

“The marketing research industry is jammed with talented researchers who understand great research.  Yet industry researchers have historically found themselves trapped on a never-ending wheel chasing the next research assignment, sometimes at the expense of good design.”

Like Grapentine and Teas, I encourage marketing researchers to step outside their “comfort zone” and think first and foremost on the strength of their designs.  Even if practical considerations impede a scientific path, marketing researchers owe it to themselves and the end-users they serve to question every design in terms of its ability to return reliable, valid results.

With survey length and grid questions contributing most to incompletes (see Lightspeed), it is reasonable to look for new ways to gain respondents’ attention and keep them attentively thoughtful throughout questionnaire completion.  Online platforms provide any number of optional features for incorporating “eye candy” into online surveys, including face scales and graphic images, as well as a slew of functions within a rich text editor.  All of which is great, as long as this visual stimulation doesn’t degrade the quality of the research; and, at the very least, researchers have some understanding how visual cues embedded in online survey designs impact response behavior.

A number of researchers have explored this issue.  Mick Couper, Frederick Conrad, and Roger Tourangeau wrote about “Visual Context Effects in Web Surveys” and concluded from their experimentation that visual images do in fact have an effect on survey response.  Achim Elfering and Simone Grebner found in their research that men and women interpret face scales differently – e.g., women were more likely to interpret a horizontal (“neutral”) line on a face as depicting “sad” while men were more likely to construe the neutral line as “happy.”  And, Vera Toepoel looked at visual context effects related to two key components – verbal language and images (pictures) – and concluded that ”there is a hierarchy of features that respondents attend to, with verbal language taking precedence over visual cues like pictures.”

This is an exciting area in research design and it is hoped that more researchers, esp., those who cry most loudly for dumbing down our designs to “research snacks,” will initiate their own experimentation with online research design to better their own studies as well as contribute to the industry.

Or do they?  Do all of our “why” questions and projective exercises actually elicit attitudes and opinions that are truly valuable in that they are reliable and honest?   This is an important question because, just as moderators search for the best approach in gaining new insights, they also want to feel confident in their findings.  This sense of reliability is a main ingredient to good research.

So, are our focus group designs – with all the built-in probes and tactics – producing good research?   The issue here is the trustworthiness of the results and whether what we learn from one focus group study is not too far afield from what we would learn if we were to rewind the calendar and conduct the study again.  Researchers are obligated to examine this issue and the certainty by which they can say that the attitudes expressed (or otherwise revealed) in their focus group research are dependable and the implications for future behavior are real.

Wouldn’t it be a shock if our “why” questions and projective techniques were in fact degrading the reliability of our focus group research?   Some experimentation has shown that asking people to explain or give reasons for their attitudes and behavior essentially alters their response.  Timothy Wilson and Sara Hodge, for example, in “Attitudes as Temporary Constructions” discuss various studies that all point to the same basic conclusion: introspection or asking research participants to analyze their reasons changes their attitudes, and can even lead to less-than-optimal decision-making behavior (i.e., people allow their reasoning to guide them to decisions they would not make otherwise and that ultimately turn out to be unsatisfactory choices).

Wilson and his colleagues, in their 1989 paper, isolated the effect of introspection and attitude change to people who were relatively unfamiliar with or less knowledgeable about the topic in question.  So, for instance, people who were not too familiar with a political candidate were more apt to change their attitudes toward the candidate compared to people with more knowledge of the individual.  It has been suggested that, in analyzing their reasons, less knowledgeable people are forced to consider any number of factors outside their original sphere of belief, making the newly-formed attitude fleeting and subject to further change.

These are just a couple of examples of the work that has been done exploring attitude strength and its association with “thinking too much.”  It is important to anyone who designs focus group research because it tells us that: 1) asking group participants to justify their attitudes and behavior (via the “why” question or projectives), in and of itself, can alter their thoughts; and, 2) the reasoning process – particularly among less knowledgeable participants (possibly non-customers, non-users) – invites a host of atypical considerations for any one individual that can fluctuate from moment to moment.  All of which speaks to the trustworthiness of our research findings.

If the purpose of research is to understand how people think then how do we do that without trespassing into the zone of “thinking too much” and affecting the very attitudes we are after?  Focus group research designs can address this in various ways.  For instance:  1) the moderator can build in more active listening skills that focus on picking up inter- and intra-participant attitudinal inconsistencies; 2) the moderator can carefully select projective techniques and shy from those that force participants to think deeply about something they know little about; and, 3) focus group discussions can be targeted towards people who have knowledge of the topic (e.g., customers, users) and therefore more likely to harbor a stable opinion.  These are just a few of the many design considerations that researchers can incorporate into their focus group studies to maximize honest reasoning from participants to produce reliable, insightful outcomes.

Socioeconomic status is an important factor which may carry a great deal of weight in health-related research but is largely ignored in other disciplines such as marketing research.  However the fact that one’s socioeconomic standing impacts their behavior and attitudes (e.g., toward greed) – and how they think – makes this a critical component in our research designs.

But what are the design elements that effectively measure socioeconomic status?  In 1958, Charles L. Vaughn published a paper in Public Opinion Quarterly titled “A scale for assessing socio-economic status in survey research.”  Back then, door-to-door interviews was the most likely mode of data collection and the “common method” for assessing socioeconomic status was (remarkably) by way of the interviewers’ subjective ratings of each respondent’s “dwelling unit” as well as “personal care and speech.”  Vaughn discusses several problems with this method, not the least of which is that the results were “not very reliable.”  To address these problems, Vaughn designed seven questions (with the seventh question split into two questions about the household) to measure socioeconomic status by which respondents’ answers were scaled against interviewers’ ratings.  While socioeconomic measures typically focus on income, occupation, and education, Vaughn omitted income in part because “the subject tends to irritate respondents and thus demoralize interviewers.”  Instead, Vaughn asked about occupation and education as well as status symbols of the time such as automobile ownership, telephone service, and housing (although he later dropped that last two questions because there was “so much respondent resistance”).

Vaughn’s Socioeconomic Status Questions

1.   Are you or is somebody else the chief wage earner in your home?

2.  What is (your, his, her) occupation?

3.  About how far did (you, he, she) go in school?

4.  Is there a car in your home?  Was it bought new or used?

5.  Is there a telephone in your home?  Is it a party or private line?

6.  Do you rent or own the place where you are living?

7a.How many bedrooms are there in your home?

7b.How many people live there?

Piff et al. utilized a modern-day scale to gauge social status, the MacArthur Scale of Subjective Social Status.  This scale was developed by Nancy Adler, PhD, professor of medical psychology at the University of California, San Francisco, in order to evaluate where people place themselves relative to others in terms of money, education, and job situation.  Respondents are presented with a “social ladder” and asked to mark the ladder depending on where they see themselves relative to the top and bottom rung.  Another version of this ladder is the “community ladder” which pegs an individual’s perceived status to their standing in the community.

Socioeconomic indicators are an important ingredient to our research outcomes.  How (or if) people respond to our research questions is greatly impacted by where they find themselves in a world made increasingly more complicated by social and economic disparities.  Research designs that better reflect the important role of socioeconomic data will come closer to understanding how people think which will ultimately lead to more targeted, effective decision making.

Social desirability bias is not a new concern in research design and its influence on the ultimate usefulness of our qualitative and quantitative research has been the focus of attention for quite some time.  Tourangeau, Rips, and Rasinski (2000) discuss social desirability in the context of sensitive questions:

“[The] notion of sensitive questions presupposes that respondents believe there are norms defining desirable attitudes and behaviors, and that they are concerned enough about these norms to distort their answers to avoid presenting themselves in an unfavorable light.”

Nancarrow and Brace – in their article “Saying the ‘right thing’: Coping with social desirability in marketing research” – address the under- and over-reporting associated with social desirability bias and outline numerous techniques that have been used to deal with the problem – e.g., emphasizing the need for honesty, promises of confidentiality, and question manipulation by softening the suggestion that the respondent should know the answer to a particular question or behave in certain way.

Online technology and the ever-growing online research designs that are emerging – within social media, mobile, bulletin boards, communities, and survey research –  have allayed social-desirability concerns.  The belief among some researchers is that one of the beauties of the virtual world is that inhabitants basically live in solitude, away from the influences of a social reality.  Paul Rubenstein of Accelerant Research has actually stated that a key advantage to online qualitative research is the obliteration of social desirability bias and hence the heightened validity of online vs. offline designs:

“[The] Internet affords a stronger sense of anonymity [compared to in-person interviewing] among study participants and typical response biases such as social desirability and other “faking strategies” are virtually eliminated online.  Qualitative data collected online tend to be “brutally honest” in nature as respondents feel wrapped in a cocoon of privacy and facelessness and have no apprehension about telling a moderator anything.”

The idea that researchers who design online studies can ignore potential bias due to social desirability seems misguided.  In fact, a good case can be made that the Internet and online technology have unleashed a dynamic capacity for posturing and the need for approval.  Popularity and even celebrity – so elusive to the everyday person in earlier times – have become preoccupations.  You only need to witness the apparent race for Facebook friends, LinkedIn connections, Twitter followers, and YouTube or blog views – as well as the “vanity” online self-publishing craze – to gain some insight into the potential competitiveness – i.e., pursuit of social stature – fueled by the realm of online.  In this way, the virtual social environment has encouraged a look-at-me way of thinking and behaving.

So, how real are those at-the-moment snippets transmitted by mobile research participants (which may be meant to impress the researcher more than inform)?  How honest are those product reviews or blog comments?   What is the extent of bravado being exhibited in our online communities, bulletin boards, and social network exchanges?  The answer is we don’t know, and yet it doesn’t take a great leap of faith to acknowledge that the individual attitudes and behavior we capture online are potentially distorted by an underlying need for social approval.

To paraphrase Mark Twain, the reports of the death of social desirability bias in online research are greatly exaggerated; and, to the contrary, social needs have blossomed in the online world.  More than ever, people are asking, “Do you like me?” and, in doing so, presenting the researcher with a critical design issue that impacts the quality of our outcomes.

Tourangeau, R., Rips, L., & Rasinski, K. 2000. The Psychology of Survey Response. Cambridge University Press.

“Questions & Answers: Selected Articles from Research Design Review” includes articles such as:

• The Questions of Race & Ethnicity  Every researcher on every study needs to decide whether the race-ethnicity questions provide meaningful, actionable information.  If these questions are deemed appropriate, careful design considerations are imperative to giving the respondent a clear path to response.

• The Vagueness of Our Terms: Are Positive Responses Really That Positive?  If the researcher is to have any hope of providing usable data to the client, attention has to be paid to clarifying survey responses.  After all, wouldn’t you want to know if the finding that 90% of your customers are “very likely” to buy from you again really means there is a 50-50 chance of a repeat purchase?

It is hoped that Questions & Answers, along with future posts in RDR, will bring greater awareness and understanding of the issues impacting qualitative research design and will ultimately lead to more useful, higher-quality outcomes.

Why is that?  In the February 2010 RDR post, “Qualitative Research & Thinking About How People Think,” I reference the four-step cognitive process (posited by others) considered necessary to respond “optimally” to research questions: interpretation, searching for relevant information, integration towards a judgment, and translation of a judgment into a response.   These four stages alone suggest that isolating the key reason for choosing store A over store B may be more complex, requiring more thoughtful contemplation than the quick response researchers typically encourage in order to keep the interview to a manageable length.  And that is why my judgment about how many miles I drive to work may change depending on whether I have included in my search for relevant information (the second stage of the cognitive process) the shortcut I take on certain days or whether I calculated the door-to-door distance versus some other parameter.  The judgments derived by research respondents can change from moment to moment for any number of reasons.

George Bishop and Stephen Mockabee in their December 2011 piece “Comparability of Measurement in Public Opinion Polls” discuss the “peril” of “ignoring the incomparable responses given to most survey questions,” and particularly “how the meaning-and-interpretation of survey questions can vary across respondents and over time even when the wording and context of the question itself remains identical.”  Meanings and interpretations (i.e., judgments) are continually changing within and across respondents depending on the social, economic, and political environment at any point in time.  As one solution, Bishop and Mockabee encourage the use of random probes to reveal (and track) the particular meanings respondents give to question wording.

In addition to social-economic-political contexts, the ever-changing landscape that gives birth to respondents’ judgments is equally impacted by the human condition.  Question interpretation, information retrieval, judgment formulation, and ultimate response may shift depending on a person’s general mood, physical condition, or surrounding stimulus and external cues.   All of these factors impinge on how we think and why identical questions at two different moments in time can elicit unequal responses.

All of this suggests that there is no truth to be gained from our respondents, just a judgment at one point in time.   Yet there is an important truth to be found in the meanings and interpretations that respondents give to our questions at the moment of asking.  Similar to Bishop and Mockabee, I believe that by building cognitive interviewing into our research designs – not just during the pre-field, testing phase but for the entirety of fielding – researchers will not only discover the basis from which responses are given but also gain the ability to segment respondents beyond the typical demographic and lifestyle characteristics to include how respondents group based on their interpretations of research questions.   And, as Bishop and Mockabee state, this would enable researchers to track fluctuations in meanings over time.

Back in 1996 I stated that online research “provides the researcher with a solution that is sensitive to both budget and time constraints,” a key justification for online research designs today.  Because our understanding of how people think and communicate in the online world was cloudy at best, I go on in this article to offer “fast, economical” alternatives to online designs -

• Developing an annual corporate research program (while minimizing costly ad-hoc research)
• Reducing sample size in survey as well as qualitative research (e.g., greater use of mini groups)
• Cutting out research services that are underutilized, e.g., written transcripts or full reports that are rarely read
• Asking for “volume-discount pricing” from research providers
• Moving the research function up the corporate ladder to create efficiencies and focus on less-costly design solutions

While these alternative approaches are as appropriate today as they were 15 years ago, the appreciable advancement of online technology has greatly increased the viability of online research designs.  And, although the near silence in the marketing research community concerning computer-mediated communication is a bit deafening, it is encouraging to see MarketTools TrueSample and other initiatives designed to address online respondent fraud.

But what about social media research where validation is difficult?  Moving forward, it would be useful for social media researchers (corporate side and consultants) to entertain the ideas espoused by those in communication studies, psychology, computer science, and other disciplines that examine online behavior and attitude formation. For example, Jenny Rosenberg and Nichole Egbert discuss in the Journal of Computer-Mediated Communication their study of the “self-presentation tactics” Facebook users employ to maintain a particular impression on their intended audience.  And Stephanie Rosenbloom in her New York Times article, “Putting Your Best Cyberface Forward,” references a variety of sources including Mark Leary, a psychologist at Duke, who studies impression management and explores the images people choose to create of themselves in the online sphere.

In the relatively controlled environment of online survey and community-style research designs, we may be learning to identify whether there is a dog at the other end of the computer or mobile phone screen; but social media researchers are strapped with the more daunting task of understanding how people think and who they choose to become in the virtually social context.  This – and its ramifications for research design – are worthy of more dialog.