The Board of Directors of the Roper Center for Public Opinion Research voted June 22, 2018, to expand the Center’s collection to include surveys conducted by recently developed methods and to add new transparency protocols for data donations.
Expanding the collection is intended to encourage independent research into data collected by recently developed methods. Such research can be conducted only with an adequate understanding of the survey methodology. Therefore, the Center makes such survey data available to subscribers only if the data donor provides a) the survey dataset with sample weighting variables and b) adequate methodological disclosure to support meaningful investigation.
The Center makes no judgment as to the efficacy of new data collection methods. Given theoretical and empirical considerations, the Center maintains a clear distinction in the user interface to access these surveys and the probability-based surveys that form the basis of the Center’s existing collection. Priority is given to processing data collected via longstanding methods.
Conceptually there are two collections, one populated with materials that fit the Center’s longstanding collection protocols and a second for surveys conducted using recently developed methods. The default point of entry for users accessing the archive are surveys conducted using longstanding methods. A link is provided to allow users to search and download data collected using recently developed methods, with an explanatory message. Search results for the current collection and the new collection are clearly separated in order not to equate these methods. All pages on which new-collection data are available to include a prominent message noting that users are viewing data from the Center’s archive of data collected using recently developed methods. A link back to the primary collection of surveys conducted via longstanding methods is provided.
The new collection is not described as an archive of non-probability data. Some recently developed methods use probability sampling. Some use a hybrid approach, in which probability and non-probability data are combined. Other new approaches may not fit neatly into existing categories. As such we use the term “recently developed methods” to describe surveys collected using methods that depart from the Center’s existing standards. Those standards are maintained for the primary collection. Both collections are limited to quantitative data.
Disclosure is key to this approach. To support assessment by the research community, the following information is required for data collected using recently developed methods to be considered for the collection. The same disclosure is strongly encouraged for data collected using longstanding methods (The pre-existing disclosure requirements for surveys produced using longstanding methods are included in the “additional details” link below.)
- Survey sponsor, including all funding sources
- The population of which the results are said to be representative, and the justification for this research claim
- Full survey questionnaire with all instructions, prompts, visual aids
- Full survey dataset with sample weighting variable(s)
- Universe and sample frame (i.e., the population the research is seeking to represent and the frame from which the units within that population are selected)
- Coverage, meaning the proportion of the universe that had a nonzero probability of selection
- Respondent selection procedure
- Sampling method: Probability, non-probability or hybrid
- Mode: RDD telephone, IVR; listed-sample telephone with live interviewers; listed-sample telephone via IVR; other telephone (describe); opt-in online panel; other online (e.g., river samples, mobile apps; hybrid or other (describe))
- Unweighted sample size
- Interview dates
- Survey language(s)
- Response rate calculated to AAPOR standards, or sample disposition data adequate for the calculation of AAPOR-standard response rates. When AAPOR-standard response rates cannot be calculated, completion or participation rates shall be provided using another method that is fully disclosed
- Description of all sample weights and sources of weighting targets
- Field work provider, if outsourced
Disclosure of the following items also is encouraged:
- Use of incentives
- Breakoff rate (i.e., the percent of respondents who start the survey but do not finish it)
- Use of survey routers or chains
- Details of quality control checks (e.g., for logic, speeding, straightlining), including how they were performed and results of those checks, including percent of completed interviews excluded or dropped from the analysis
- Sample provider(s), and, if multiple, the share of sample from each provider
Center staff will review submissions of data collected through recently developed methods on three criteria:
- Whether the donor claims that the sample is representative (required for consideration)
- The face validity of this and other research claims
- Compliance with disclosure requirements
Submissions of data collected using recently developed methods that do not pass these disclosure criteria will not be made available to any users, but may be stored in the Roper Center system. When Center staff are unsure whether to accept a submission, they will refer it to a standing committee of the Board for disposition.
As a means of encouraging full disclosure by all donors, regardless of whether they use longstanding or recently developed methods, a Roper Center Transparency Score has been created to denote disclosure compliance for all holdings in the archive. Ratings are linked to all search results.
The scoring system is based on the disclosure lists above, aggregating points as more disclosure materials are submitted. We recognize that transparency scores for data produced via newly developed methods are elevated by the disclosure requirements for these data.
This policy is intended to respect the Center’s current, longstanding collection protocols and its longtime data donors while expanding the collection in recognition of the recently developed data collection methods now in use. This expansion of the archive, under these protocols, is intended to provide future historians the raw data they will need to look back on our era, provide the research community with an invaluable opportunity to assess new methods and extend the Center’s leadership in the responsible collection and dissemination of public opinion survey data.