Assessing the quality of a survey starts with reviewing fundamental details about the poll. Begin by investigating the answers to these questions:
Who conducted the survey?
Reputable organizations won’t risk conducting poor or unprofessional surveys . These organizations have experience in the process of conducting surveys—from sampling to designing questionnaires to training interviewers to reporting results. Reputable survey organizations deposit their data with an archive like the Roper Center that provides broad access for other researchers to learn from the data and scrutinize the results. Responsible polling groups will also adhere to professional standards and practices of disclosure, thereby making all of their data publicly available. However, different expectations apply to private polling organizations who are dealing with proprietary information.
Was there a sponsor? What was the purpose of the data collection?
If the sponsor of the poll has a vested interest in the outcome, this does not automatically translate to poor survey quality or reporting. After all, individuals and groups with special interests seek to know the truth on issues that are important to them. This is not to suggest that the motives of the sponsor be ignored, but rather to alert researchers to the possible biases that may occur during the survey process. They can therefore be particularly diligent in critiquing the methods used to gather and report results.
How was the survey done? What is the population being studied? What sampling techniques were used to assure a random sample of that population?
The description and size of the sample along with the exact interview dates should be reported in the methodology section of the report, press release, or other source of the data. The report should also include a description of the methods and modes of interviewing. Most US national public opinion polls today are conducted over the telephone, a method that has been very successful in securing a random sample or a probability sample.
Other common acceptable modes of interviewing are “in person” surveys, mail surveys, and self-administered surveys (such as exit polls). Sound research avoids sample surveys in which the respondents are self-selected. Call-in polls to vote for your favorite song on the local radio station are unscientific, not random and only represent those who chose to make the call, got through, and voted – at least once!
The latest and fastest-growing mode of interviewing is using the Internet. However, this technique is still being assessed because not everyone in the national adult population has online access and the validity of generalizing the results of a national sample has been questioned. Suffice to say, caution should be taken when analyzing any survey that requires respondents to have a particular mode of communication that is not generally available to everyone in the universe it purports to represent.
Principles of Disclosure
The policing agency for the industry, the National Council on Public Polls has issued a statement regarding standards of disclosure for organizations who release survey results to the public. The organization also provides information on how to assess the quality of a poll. According to the NCPP, a report or press release stating that a survey organization is in compliance with the Principles of Disclosure of the NCPP will have reported the following information:
- Sponsorship of the survey
- Dates of interviewing
- Size of the sample
- Size and description of the sub-sample, if reporting relies primarily on less than the total sample
- Complete wording of questions upon which the release is based
- The percentages upon which conclusions are based
For further information please contact The Roper Center at 607.255.8129 or firstname.lastname@example.org.