Survey research is pretty good at allowing people to describe “things” in such a way that the researcher winds up with a fairly accurate idea of the thing being described. The most straight-forward example is a survey question that asks, “Which of the following features came with your new Toyota Corolla?” followed by a list of possible features. However, survey research can also get at descriptions of more experiential phenomena with questions such as, “On a scale from ‘1’ to ‘5’, how does each of the following statements describe your experience in buying a new home?” In these cases, the use of survey methods to research a great number of people, and compile and report the data as efficiently as possible, make good use of closed-ended questions to gain an understanding of respondents’ accounts of the “things” of interest. This can also be said of beliefs. Pew’s 2014 survey pertaining to the Christmas story that asked, “Do you believe that Jesus Christ was born to a virgin, or don’t you believe this?” is just one example of how a closed-ended survey question – coupled with similar questions related to different aspects of (for example) the Christmas story – can ultimately paint a descriptive portrait of someone’s beliefs, religious or otherwise.

All of these are, to some extent, concrete objects of description – a car, buying a home, a belief (you either believe or don’t believe) – that lend themselves to the discreteness associated with closed-ended survey question formats. But what about the nebulous world of feelings? Is it possible for the survey researcher to ascertain respondents’ feelings – that is, come to a description of what people are actually feeling about a thing, an experience, or belief – by way of these same closed-ended survey question techniques?

Some seem to think so. A major hotel brand has designed a feedback survey asking recent hotel guests to describe their “ideal” hotel by rating various amenities and features such as comfortable furniture and complimentary Wi-Fi. This gives the hotel a decent depiction of a person’s “ideal” hotel within the framework of what they can control, e.g., furniture décor and Internet services. The survey design, however, becomes seriously flawed when it goes on to ask, “How well do the following statements describe how your ‘ideal’ hotel would make you feel?”

Although an admirable research goal – that is, to learn how guests describe, not just the things that make a hotel “ideal” but also, the feelings and sensations these things arouse – the hotel has taken a wrong turn into the murky waters best traversed by qualitative methods. In this way, the hotel has misunderstood the design limitations of closed-ended survey questionnaire design.

A closer look at the question makes this apparent. The hotel’s “feeling” question asks the respondent to rate various statements, including:

• Allows me to live the good life.
• Helps to create good memories.
• Makes me feel calm and peaceful.
• Helps put a smile on my face and makes me feel happy.
• Broadens my horizons and helps me to discover new things.
• And the list goes on…

This question is a lose-lose for both the poor respondent and, more so, the poor researcher who has to deal with the resulting survey data. The respondent clearly has the difficult task of forming context and meaning around the researcher’s preconceived virtues of an ideal hotel. This requires lots of cognitive effort, involving multiple soul-searching questions: What is “the good life,” what significance does that have for me, and what relevance does that have for me in choosing a hotel? Or, I am not sure what is meant by “horizons” and how horizons are broadened, is that the same as discovering “new things,” and what are the new things that an ideal hotel could help me discover?

For the survey researcher, this question is even more complex. Assuming that the sole purpose of the question is not for marketing purposes, e.g., an advertising campaign to position the hotel as a sanctuary for those seeking “the good life,” the person having to analyze this survey data and operationalize it in order to reach useful conclusions is left powerless. While the researcher may have their own concept of what “the good life” or “good memories” mean, there is no way in a closed-ended survey question format that the researcher can begin to make meaning from this data.

Capturing feelings and sensations in order to capture “real,” personal experiences is a necessary and important goal of research with human beings. Yet, it is qualitative research methods – not closed-ended survey designs – that allow researchers to tap into those often elusive inner experiences.

Image captured from: http://izismile.com/2013/04/08/a_majestic_african_hotel_experience_in_kenya_21_pics.html