Topics Topics

ERS Data Product Policy Recommendations and Standards

INTRODUCTION

The mission of ERS is to inform and enhance public and private decisionmaking on economic and policy issues related to agriculture, food, natural resources, and rural development. To accomplish this mission, ERS economists and social scientists develop and disseminate a broad range of science-based economic and statistical information to the public. This suite of data products encompasses estimates, forecasts, economic and statistical indicators, and data compiled from diverse sources where ERS adds value in the form of recompilation and/or subject-matter expertise. ERS disseminates its information to key stakeholders and the public through an array of outlets, including the ERS website ( www.ers.usda.gov).

As a Principal Federal Statistical Agency, ERS is committed to quality and professional standards of statistical practice. ERS uses modern statistical and economic theory and practice in all technical work; develops strong staff expertise in economics, statistics, and other disciplines relevant to its mission; implements ongoing quality-assurance programs to improve data validity and reliability and to improve the processes of collecting, editing, analyzing, and disseminating data; and develops strong and continuing relationships with appropriate professional organizations in relevant subject-matter areas. To carry out its mission, ERS assumes responsibility for determining sources of data, measurement methods, and methods of data collection and processing; employing appropriate methods of analysis; and ensuring the public availability of the data and documentation of the methods used. Within the constraints of resource availability, ERS continually works to improve the quality of its research and its data systems to provide theinformation necessary for the formulation of informed public policy.

Background

Statistical Policy Directives and other standards issued by the Office of Management and Budget (OMB) in its role as coordinator of the Federal statistical system [1] provide a foundation for individual agencies to achieve and safeguard scientific integrity. Specifically, OMB’s directives and standards are designed to preserve and enhance the objectivity, utility, and transparency of statistical products and their dissemination. Examples include:

  • Statistical Policy Directive Number 3 is intended to preserve the time value of principal economic indicators, strike a balance between timeliness and accuracy, prevent early access to information that may affect financial and commodity markets, and preserve the distinction between the policy-neutral release of data by statistical agencies and their interpretation by policy officials.
  • Statistical Policy Directive Number 4 enumerates procedures intended to ensure that statistical data releases adhere to data quality standards through equitable, policy-neutral, and timely release of information to the general public.
  • Standards and Guidelines for Statistical Surveys documents important technical and managerial practices that Federal agencies are required to adhere to and the level of quality and effort expected in all statistical activities to ensure consistency among and within statistical activities conducted across the Federal Government. [2]
  • Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies provides guidance to help agencies ensure and maximize the quality, utility, objectivity, and integrity of the information that they disseminate. [3]
  • OMB Circular A-130 prescribes procedural and analytic guidelines for implementing policies for the management of Federal information resources.
  • OMB Memo M-13-13 requires release of information in a way that supports downstream information processing and dissemination activities while ensuring stewardship around confidential information.

To ensure and maximize the quality of information disseminated by Federal agencies, including the objectivity, utility, and integrity, OMB issues guidelines that instruct agencies to treat information quality as integral to every stage of the information life cycle from creation or collection, processing, dissemination, use, and storage to disposition.

USDA has also published Information Quality Guidelines. Included in the guidelines are:

  • Statistical and Financial Guidelines that pertain to information disseminated by USDA agencies and offices that is obtained from original data collections, administrative records, or compilations of data from primary sources, as well as estimates and forecasts derived from statistical models, expert prediction, or a combination of the two.
  • A Definition of Influential Scientific, Financial, or Statistical Information that states, in non-rulemaking contexts, USDA agencies and offices will consider two factors—breadth and intensity—in determining whether scientific, financial, or statistical information is influential. “Information that has an intense impact on a broad range of parties should be regarded as influential.” [4]

Types of ERS Data Products

ERS continually assesses the needs of data users and provides a range of products that addresses those needs most effectively. ERS data products fall into four categories, defined by the type of data and value added:

  1. Primary Data that are made available to the public, such as ARMS and FoodAPS survey data.
  2. Model Results, such as Food Dollar, Food Price Forecasts, Food Availability, Commodity Supply and Use, and Trade Multipliers.
  3. Summary Statistics derived from single or multiple sources of primary data, such as FATUS and U.S. Bioenergy Statistics.
  4. Repackaged Data derived from existing ERS data products or published research results, such as Charts of Note and State Fact Sheets.

Several data products cut across more than one data type (primary data, model results, and summary statistics), such as Farm Income and Food Security reports. This would also include GIS-based models, such as the Food Access Research Atlas.

In addition to characterizing ERS data products by the type of data and value added, criteria have been developed that allow a distinctive ranking of data products in terms of adherence with the OMB and USDA definitions of Influential Scientific, Financial, or Statistical Information and in accordance with their importance to the agency’s mission:

  • Premier/marquee data products that are determined by senior ERS management to be influential and central to the agency’s mission, as well as adhering to all components of quality guidelines as applicable.
  • Core data products that are central to the agency’s mission but may not meet the definition of influential. Also in this category are foundational data (as mentioned in the ERS Strategic Plan), such as data that are inputs to premier data products.
  • Other data products that serve key agency stakeholders and the public.

RECOMMENDATIONS

OMB guidelines require agencies to develop a process for reviewing the quality of information before it is disseminated to the public to ensure that it meets OMB’s standards for objectivity, utility, and integrity. The recommendations below describe the principles that ERS will follow to ensure these quality standards are embedded in data products provided to key stakeholders and the public. The ERS Data Review Committee recommends the formulation of a Data Product Council (DPC) that will oversee and implement these policies and standards. The policy is applied broadly to all data products, and each data product must meet each standard of purpose, utility, objectivity, transparency, integrity, and accessibility.

These recommendations recognize there is a hierarchy among data products based on measurable attributes that distinguish various levels of product quality. All premier/marquee products must meet all levels defined in each standard, as applicable by data type. The extent to which each specific standard applies to core and other products may vary, depending on the type of data product. Initially, the DPC will review each product to determine the extent to which the standards apply and how the products currently conform. Data products other than premier or core should undergo a cost-benefits analysis to ascertain the value to the agency.

All policies and procedures described in this document are applicable to all ERS employees, contractors, visiting scholars, cooperators, or others to whom access to ERS data has been granted. [5]

OMB Quality Attributes and Proposed Standards

1. Purpose

There are institutional and organizational factors that make ERS uniquely qualified to: 1) develop and disseminate data products directly to key stakeholders and to public audiences through the ERS website, or 2) significantly influence the effectiveness and credibility of the Federal agency producing the statistics. As a principal Federal statistical agency, ERS produces and disseminates certain data required by pertinent legislative or regulatory mandates. The agency is a member of, and actively participates in, many interagency or non-Federal economic and statistical committees and working groups (e.g., USDA’s Interagency Commodity Estimates Committees) to achieve the mission of the Department. ERS also supports policy/program administration in other USDA agencies by providing economic/statistical analysis and through data-sharing.

Reasons and methods of data sharing should be documented whenever ERS provides data to, or receives data from, other Federal agencies.

1.1 ERS data products that are distributed to other Federal agencies on a regular basis and outside of the normal/interagency publication process should have documented agreements (e.g., MOU, MOA) that indicate the purpose of data sharing, length of agreement, etc. and adherence to data security standards.

Routine data-sharing arrangements between ERS and other Federal agencies should be documented. The preferred method is a Memorandum of Understanding (MOU) or Memorandum of Agreement (MOA). If such an agreement is not in place, division staff and data product authors should explore (in conjunction with the Associate Administrator) formalizing the data-sharing arrangement with an MOU/MOA. If an MOU/MOA is not appropriate, the justification and reasons should be documented.

1.2 Data Products that use restricted data from external Federal sources must have documented agreements (MOU, MOA) that indicate the purpose of data sharing and adherence to data security standards.

2. Utility

Utility refers to the usefulness of the information to intended primary users. [6] One aspect of utility is fitness for use (are data appropriate for intended audiences?). Another aspect could be timeliness, which can be measured by two characteristics: the length of the data collection’s production time (whether data are made available as quickly as possible to preserve the value of the data, e.g., the time from data collection until first release) and the frequency of the data collection or update. ERS achieves utility for its data products by staying informed of stakeholder needs and developing new data, models, and information products where appropriate, and to the extent practicable, making information widely available and easily accessible and helping users understand and use its products.

2.1 ERS is positioned as the preeminent or sole source provider of our premier data products.

As a Federal statistical agency, ERS must be knowledgeable about the issues and requirements of public policy and Federal programs pertinent to the USDA mission and be able to provide objective information that is relevant to policy and program needs. The unique alignment of resources and expertise create specific capabilities to produce important and influential data products that would otherwise not exist.

2.2 All data products must be branded (sourced) as coming from ERS.

Data products must be branded as coming from ERS (when they are released by ERS), as standard best practice for documentation, to more accurately measure impact and to improve ERS’s profile as a Principal Federal Statistical Agency. As applicable, data products should also cite the source of the data (such as from multiple Federal agencies).

2.3 Information provided by data products must be reported on an ongoing basis. As such, future releases of premier products must be reported on the ERS calendar. All data products should also provide the schedule for the next update on the ERS website.

To ensure equivalent and timely access to all users, a schedule and mode of release must be developed and publicly conveyed in the calendar year prior to the planned release of a data product. [7]

2.4 All data products should identify key internal and external stakeholders.

Persons or organizations that have a vested interest in the information that is being promoted in a data product are considered stakeholders in the process. Knowing the interests, positions, alliances, and importance to ERS of key stakeholders enables data product authors to interact more effectively with these individuals and adapt to their changing needs.

2.5 Data products must undergo outside review with key stakeholders for communications quality and content relevance every 5 years.

Measures of content relevance and quality of communication can include on-demand requests fulfilled, product downloads, number of formats in which data are available, degree of accessibility, [8] customer satisfaction with ease of use, results of usability testing, number of participants at user conferences, citations of agency data in the media, amount of technical support provided to data users, and exhibits and other promotional materials to inform the public about data products.

2.6 All data products must have a website feedback mechanism or employ alternative ways to communicate with users.

Recognizing the diversity of data users and their importance, all data products should employ a feedback/input mechanism—based on a strategy of engagement with users to help facilitate and prioritize data release. [9] Website Contact Forms, for example, allow for the elicitation of feedback from users in a secure and organized manner. Other forms of communication with data users include public meetings, such as those organized by NASS to alert data users on recent and pending changes in the various statistical and information programs important to agriculture and to seek comments and input on these programs. Practices to improve communication with users employed by various statistical agencies were highlighted in the 2009 CNSTAT review. [10]

2.7 Web usage statistics for data products should be regularly reported to appropriate ERS staff (quarterly, annually, or as appropriate to the release schedule) and evaluated.

Statistics about use may provide data on how many visits, page views, bounce rate, average time on site, location, traffic sources, content sources, and percentage of new visits. This type of information can assist with priority setting and product refinement.

3. Objectivity

Objectivity is a measure of whether disseminated information is accurate, reliable, and unbiased and whether that information is presented in an accurate, clear, complete, and unbiased manner. [11] Agencies should inform the public as to the strengths and limitations inherent in the information disseminated (e.g., possibility of errors, degree of reliability, and validity) so that users are fully aware of the quality of the information. [12]

3.1 All data products are reviewed for data quality prior to dissemination.

Data products produced by ERS are thoroughly reviewed by knowledgeable staff prior to dissemination to verify the validity of the data. The procedure used to conduct this review must be documented and available. Data are checked for internal consistency, consistency with other similar data sets or prior year versions of the same data set, and sources of error. Knowledgeable ERS subject-matter experts conduct “reasonableness” checks of the data. Where necessary, the data are edited and missing values are imputed using established statistical techniques to improve the utility of the data.

3.2 Subject to DPC recommendation, all data products must undergo an independent external review of methodology at least every 10 years.

The breadth and extent of review will be determined by the type of data product. For example, external peer review provides a robust means of evaluating data products that employ surveys or models. Reviewers from other institutions bring to the review process independent knowledge, experience, and perspectives different from those of the data producer. For compilations of data, for example, the review might focus on the appropriateness of the data used and the clarity and adequacy of the documentation.

3.3 Where statistically appropriate, all data products must report measures of accuracy that accompany data elements.

Different types of data products might use different accuracy measures. For example, forecast error would be reported for estimates or projections, and estimates of sampling error and nonsampling error components (coverage error, measurement error, nonresponse error, and processing error), to the extent practicable, should be reported for sample survey programs. [13] On the other hand, a data compilation can refer users to source agencies for information on data quality.

3.4 Data products should have an ongoing research program that examines methodology and operations.

For statistical agencies to be innovative and cost-efficient in methods or practices for data collection, analysis, and dissemination, research on methodology and operational procedures must be ongoing. Methodological research may be directed toward improving survey design and survey error rates, as well as developing innovative statistical methods for protecting data confidentiality. Research on operational procedures may be directed toward facilitating data collection in the field, improving the efficiency and reproducibility of data capture and processing, and enhancing the usability of Internet-based data dissemination systems.

3.5 The production process for premier data products should receive the highest priority for IT investment and must undergo an evaluation of IT approaches every 5 years.

Premier data products should reflect appropriate (e.g., modern, efficient) methods for data collection, processing, management, and dissemination commensurate with the level of importance to key stakeholders and the public.

4. Transparency

OMB requires that Federal agencies offer a high degree of transparency about data and methodologies used to derive statistics. These requirements enable the American public maximum access to government data and ensure reproducibility of government statistics, meaning “the capacity to use the documented methods on the same data set to achieve a consistent result.” [14]

4.1 Decisions to initiate, terminate, or substantially modify the content, form, frequency, or availability of premier data products should trigger appropriate advance public notice.

Stakeholders and the public should be made aware of upcoming changes to premier data products by a notice on the ERS website, and where appropriate, email or other types of communication. [15] Where appropriate, the Office of Communications should be notified directly.

4.2 All data products must be accompanied by accurate, transparent documentation that describes the source of the data, the methodology used to produce the data, definitions of the data items and variables contained in the data set, sources of error, and, if applicable, limitations of the data.

Many analytical problems and misinterpretation of data can be avoided by providing comprehensive documentation. OMB Statistical Policy Directive Number 4 states that “With the exception of compilations of statistical information collected and assembled from other statistical products, these [federal statistical] products shall contain or reference appropriate information on the strengths and limitations of the methodologies, data sources, and data used to produce them as well as other information such as explanations of other related measures to assist users in the appropriate treatment and interpretation of the data.”

OMB provides detailed guidelines for, and a comprehensive list of, necessary components to be included in survey documentation (and other types of government data to the extent they are applicable) in section 7.3 of the Standards and Guidelines for Statistical Surveys. Some sample documentation elements include a description of variables used to uniquely identify records in the data file; a description of the sample design, including strata and sampling unit identifiers to be used for analysis; and a description of sample weights, including adjustments for nonresponse and benchmarking and how to apply them.

4.3 Data products must be accompanied by a user’s guide that explains the best statistics for different purposes.

Data products on the ERS website should contain a user’s guide to explain how best to use and interpret the data. For data products that contain data that could be used for similar purposes as other data products, those products should contain—or reference another part of the ERS website that contains—a user’s guide to assist in distinguishing the best statistical series to use for the user’s intended purpose. As this represents coordination among several products, the DPC will work in consultation with ISD Web Services to formulate a strategy.

4.4 Premier data products must provide information on the update and revision history.

Data revisions can occur for a variety of reasons, including inclusion of new data or a change in the data source; seasonal adjustment and/or elimination of calendar effects; transition to a new base period; improvement of methodology due to a change in the statistical method or a change in classifications, concepts, and definitions; or elimination of errors. To ensure transparency of the revision procedure and where applicable, information should be provided that describes the revision procedure and contains information for assessment of the existing data sources and calculation methods, assessment of the quality of the new source, and assessment of the method to be applied in the revision.

4.5 All Premier data products must have an archival capability.

For purposes of reproducibility, ERS should be able to provide users with previous releases of the data product, as part of the ERS website or upon request.

5. Integrity

“Integrity” refers to the security of information—protection of the information from unauthorized access or revision, to prevent the information from being compromised through corruption or falsification. [16]

5.1 All data products must have a defined procedure for pre-dissemination review to ensure that privacy and confidentiality are fully protected and that data are properly secured.

Data products produced by ERS are thoroughly reviewed by knowledgeable staff prior to dissemination to ensure that information is protected commensurate with the risk and magnitude of harm that would result from the loss, misuse, or unauthorized access to or modification of such information.

5.2 Data storage and processing, prerelease security procedures, and release procedures will be reviewed every 3 years for all data products.

Procedures for data storage, security, and processing must comply with current OMB guidelines [17] and the ERS Data Security Policy, particularly for primary, proprietary, and sensitive data. Methods used for pre-release review must conform to applicable security requirements.

5.3 Staff assigned to production of premier data products will undergo training for all related policies and standards.

An effective Federal statistical agency has personnel policies that encourage the development and retention of a strong professional staff who are committed to the highest standards of quality work.

6. Accessibility

Data products have their most value when they are made available to the widest range of users for the widest range of purposes and impose no barriers to any person or group of persons. Therefore, accessibility refers to the ability of any user to obtain, manipulate, and save data. [18]

6.1 Data products must be released in common machine readable formats that facilitate ease of use by a range of audiences.

ERS data offerings will be augmented with open-data formats that are platform independent, machine readable, and available to the public without restrictions that would impede their re-use. [19] Such machine-readable formats minimize the obstacles to using information contained in data files. They ensure basic and replicable processes can be created to ingest the data, which can then be consumed by any software package; and meet the needs of the growing developer community. In addition, all data products should meet Section 508 Accessibility Standards to ensure full access by the visually or hearing impaired.

6.2 Premier data that are interactive/queriable products must undergo usability testing in the design/development to ensure they are intuitive, navigable, and produce expected results.

Usability testing can help ensure data products are designed to meet users’ needs.

6.3 Premier data products must take steps to conform to OMB Open Data Guidelines.

The above recommendations for data quality address many of the OMB principles for Open Data: Public, Accessible, Described, Reusable, Complete, Timely, and Managed Post-Release. [20] As a guide to implementing open data, ERS data products will be captured in agency and Federal Government metadata inventories. Data management procedures will be adopted going forward to support the quality and openness principles.


[1] 44 U.S.C. 3504(e).

[2] These standards apply to “Federal censuses and surveys” and, to the extent they are applicable, they “also cover the compilation of statistics based on information collected from individuals or firms…, applications/registrations, or other administrative records.”

[3] OMB defines “quality” to encompass “utility, objectivity, and integrity.”

[4] See http://www.ocio.usda.gov/policy-directives-records-forms/guidelines-quality-information/background for a complete explanation of the definition.

[7] Statistical Policy Directive Number 4 states that “Prior to the beginning of the calendar year, the releasing statistical agency shall annually provide the public with a schedule of when each regular or recurring statistical product is expected to be released during the upcoming calendar year by publishing it on its Web site.”

[8] Section 508 of the Rehabilitation Act requires Federal agencies to make their electronic and information technology accessible to people with disabilities.

[9] OMB Memo M-13-13 Open Data Policy, Managing Information as an Asset, Attachment, I. Definitions, Open Data, Managed Post-Release and III. Policy Requirements, 3c Strengthen Data Management and Release Practices create a process to engage with customers to help facilitate and prioritize data release.

[10] National Research Council. "Part II: Commentary," Principles and Practices for a Federal Statistical Agency: Fourth Edition. Washington, DC:The National Academies Press, 2009, pp. 14-54.

[11] Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies.

[12] OMB Statistical Policy Directive Number 4.

[13] For more detail, see Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies.

[14] “Federal Statistical Organizations’ Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Disseminated Information,” Federal Register, June 4, 2002, pp. 38467-70.

[15] OMB Circular A-130 states that agencies should “provide adequate notice when initiating, substantially modifying, or terminating significant information dissemination products.” OMB Statistical Policy Directive Number 4 states that “Statistical agencies shall announce, in an appropriate and accessible manner as far in advance of the change as possible, significant planned changes in data collection, analysis, or estimation methods that may affect the interpretation of their data series. In the first report affected by the change, the agency must include a complete description of the change and its effects and place the description on its Internet site, if the report is not otherwise available there.”

[18] OMB Memo M-13-13.

[19] White House, Executive Order – Making Open and Machine Readable the New Default for Government Information and OMB Memo M-13-13, Attachment, I. Definitions, Open Data, Accessible and III. Policy Requirements, 1a Use Machine-Readable and Open Formats.

[20] OMB Memo M-13-13, see especially Attachment, I. Definitions, Open Data.

Last updated: Monday, September 23, 2013

For more information contact: Mitch Morehart, Mark Denbaly, Lewrene Glaser, Pheny Weidman, and Mary Maher

Share or Save this Page

Text Only Options

Top of page


Text Only Options

Open the original version of this page.

Usablenet Assistive is a UsableNet product. Usablenet Assistive Main Page.