Qualitative researchers are pretty good at distancing themselves from their quantitative colleagues, even to the point of bragging about their anything-goes right-brain sensibilities in contrast to the structured life of quantitative. So it boggles the mind when qual researchers so easily embrace certain quantitative concepts. One such concept is the randomization of stimuli in survey design in order to reduce primacy and recency effects.
I touched on this briefly in a February post when I discussed the relevance of cognitive-process theories in qualitative research (“Qualitative Research & Thinking About How People Think“). In that post, I argue that “the concept of primacy and recency effects are irrelevant in focus group research and, while randomizing the presentation order of stimuli is de rigueur in quantitative, not so in qualitative. To the contrary, I would suggest that not randomizing across group sessions adds a necessary component of control.”
So, why is a “component of control” important and how does that relate to whether the moderator randomizes or (more common in qualitative) rotates discussion stimuli such as concept boards? I have discussed the answer to this question in many private conversations with qualitative researchers who frankly have been quick to resist the idea of control in any form which of course stifles a true discussion. In one such conversation – after much talking around the issue – I ultimately resorted to a graphic, 4-slide depiction of my explanation. I present it here:
Research Design Review 
What if I have three ads to expose across three age bands? Suppose I have 2 interviews under each age band – so is it better to show same 2 ads out of three in one age band and then rotate it for other age band or change it among individuals of same age band.
For example : Showing Ad 1 and 2 to Respondent A and show Ad 2 and 3 to Respondent B.
LikeLike
Without knowing any other details, it sounds like you would show all 3 ads in each interview.
LikeLike
But why wouldn’t you rotate? You get massive order effects with certain concepts or indeed with designs. The only time I wouldn’t rotate or randomise is when there’s a natural progression and breaking it up would make no sense.
I take your point that there are lots of other things ging on; it just strikes me as difficult one, especially when some of the stimuli may provoke very different reactions.
LikeLike
Thanks, Alison, for the comment. The problem is that in qualitative you really don’t know if there are “massive order effects.” With all the uncontrolled variables, how could you possibly say with any confidence that the stimuli “provoke very different reactions.” You can’t. And as if that wasn’t enough….
Add to this the idea that I discuss in a February post, specifically that the quantitative concepts of primacy and recency are irrelevant in group discussions. If primacy and recency are irrelevant then rotation is irrelevant (and even detrimental from my point of view) because it is these concepts that would otherwise support qual researchers’ use of rotation.
LikeLike
Your last line on that slide sums it all – what effect did it have – we really don’t know because of all the other uncontrolled elements !
In quantitative research like any positivist approach – one can ascertain the influence of a variable provided you look at it in isolation therefore the assumption ‘Ceteris Paribus’. In a qual framework that assumption can never hold true therefore the futility of rotating stimulus.
It is interesting you pointing this out. Though logically it makes sense, I have also been guilty of not questioning this approach and following it mindlessly 🙂
LikeLike
Reshma, Thank you for your comment. The next time you are tempted to not question and “mindlessly” follow the standard rotation protocol, I suggest you stop and explain to the client why it is important not to rotate. You may find, as I have, that the client first gives you a quizzical look and then ‘gets it’ and says, “Okay, that makes perfect sense.”
LikeLike