Analysis begins with the full question wording. There is much emphasis on sampling error, but many pollsters will tell you that the more difficult task is framing the questions adequately to measure what is intended to be measured. The questions must to be clear or the adage will apply: garbage in, garbage out.
There are different styles of questions and as with any research, it’s probably best to look at results from multiple angles to better understand an issue. This section addresses some of the different types of questions and a few tips on what to watch for.
A Good Team: Open-ended Questions and Pre-coded Response Categories
Those who work with data frequently value the ease with which questions having pre-coded response categories can be manipulated. The question, “Do you think things are going on the right track or wrong direction?” seems pretty straightforward, with simple responses that are easy to report. Consider an analysis of this question might be enhanced with a follow-up open-ended question, “Why do you say that?” The results to the first question provide the general mood; the second set of data substantiates that mood.
One frequently used open-ended question found in surveys is the “most important problem” question: “What is the most important problem facing the country today?” The results offer a plethora of responses. Most importantly, this item can provide a sense of how salient an issue is in the public mind. When conducting intense research on any topic, it’s easy to miscalculate its importance among the public. The most important problem question lets a researcher know which issues are foremost on the public’s mind without outside prompting.
Could the order of the questions that were asked have impacted the results? It is possible that the process of the interview changed the respondent in some way, perhaps sensitizing or desensitizing participants to respond a particular way. Consider the open-ended “most important problem” question. Let’s say you’re looking at a survey focusing on the condition of the US education system. The survey includes a variety of questions tapping public opinion on the quality of schools, state and federal funding, educational programming and testing, etc. In this case, the placement of the most important problem is key. Different results would be expected depending on whether the MIP question is asked before or after the barrage of specific questions on education.
Questions Tapping Respondent Intensity
Intensity measures can be very useful in determining how firmly held a position is. These questions seek to determine not what the respondent feels about an issue, but also how passionate or ambivalent they may feel. See examples of question intensity measures.
Questions that lead the respondent to an intended answer are called loaded questions. There are blatant examples that should be interpreted with caution. For the most part, professionally and scientifically constructed surveys will avoid using loaded questions. That’s not to say, however, that they don’t creep into even the best data collections on occasion. Questions that lead with long detailed descriptions of a position or policy should signal careful reading, making sure that a fair and balanced presentation of the positions has been made. See an example of a loaded question.
Ambiguous, Double-barreled, or Confusing Questions
Phrasing questions isn’t as easy as it appears. That’s one good reason most public opinion survey organizations access (and archive with) the Roper Center collections before designing new surveys, to review the work that has come before. Even the best survey organizations have had their share of confusing question wordings. Here are examples of double-barreled and confusing questions.
For further information please contact The Roper Center at 607.255.8129 or firstname.lastname@example.org.