CICS Support for the management of Ocean and Climate Data originating from member Regional Associations of the US Integrated Ocean Observing System (IOOS)

2017 ANNUAL REPORT

CICS-UMD NOAA/NESDIS/NCEI Staff: Mathew Biddle (NOAA Collaborators: Dr. Krisa Arzayus, Dr. Rost Parsons, Dr. Huai-min Zhang);

Background

NOAA's National Centers for Environmental Information (NCEI) is an integration of NOAA/NESDIS National Data Centers in Asheville, NC; Boulder, CO; Stennis, MS; and Silver Spring, MD to provide access to the world's most comprehensive environmental data. The Silver Spring, MD location was formerly known as the National Oceanographic Data Center (NODC). NCEI is composed of two main centers: the Center for Weather and Climate; and the Center for Coasts, Oceans, and Geophysics (http://www.ncei.noaa.gov/). NCEI hosts and provides public access to over 20 petabytes of comprehensive atmospheric, coastal, oceanic, and geophysical data. NCEI provides scientific and public stewardship for the nation’s environmental data, maintains and updates archives of these data, and contributes scientific research and produce products from these data that help monitor global environmental changes.

NCEI has been funded by the Integrated Ocean Observing System (IOOS) Program Office which then funds CICS-MD to perform various data management tasks. IOOS is comprised of eleven Regional Associations (RAs), which serve the nation’s coastal communities, including the Great Lakes, the Caribbean and the Pacific Islands. IOOS is a national-regional partnership network working to provide new tools and forecasts to improve safety, enhance the economy, and protect our environment. CICS staff plays an important integral role in fostering the relationship between the various IOOS RAs and NCEI by acting as a liaison for IOOS RA archival at NCEI. CICS staff provides input on the certification of Regional Information Coordination Entities (RICEs). The certification formally establishes the role of the RICE in the U.S. IOOS and ensures that the data collected and distributed by the RICE are managed according to the best practices, as identified by NOAA.

Accomplishments

Archive and Scientific Stewardship of In Situ Data by Fostering relationships with IOOS Regional Associations. CICS staff archived numerous archival information packages at NCEI.  For example, CICS Research Assistant M. Biddle developed automations for three different data streams (1) IOOS RA SCCOOS, (2) IOOS RA NANOOS, and (3) IOOS RA PacIOOS over the FY15/16 time period.Together, these three automations consist of 31 Archival Information Packages and total to ~4+GB of data which get updated on a monthly basis. Acting as the IOOS-related data processing lead, M. Biddlecontinues to maintain the automated archival processes for the Southeast Coastal Ocean Observing System (SECOORA) and Great Lakes Observing System (GLOS) in-situ data. Both of these automated archival processes account for 98 archival packages, contributing 1+GB of in-situ observational data to the NCEI archive (Figure 1). Together, we now have 129 Archival Information Packages at NCEI that are being updated monthly with new data from five IOOS RAs, contributing to 5+GB of in-situ observational data (Figure 2). While the three new data streams were built upon the already established pathways for GLOS and SECOORA into the archive, NCEI fostered relationships with 6 other RAs to follow a similar paradigm in automated archiving.

Figure 1. Distribution of the Integrated Ocean Observing System (IOOS) Regional Association (RA) data currently archived at NCEI through the IOOS project effort.

Figure 2. Time-series plot of the currently archived Integrated Ocean Observing System (IOOS) Regional Association (RA) data at NCEI. This time-series only uses the start and end dates from the metadata extracted from each archival information package. Each bar on the time-series represents one archival information package.

Last year, CICS staff developed and delivered a cookbook on how to archive in-situ observational data at NCEI (Figure 3), “Archiving your data at NCEI: A Guide for IOOS Regional Association Data Managers”. Which webpage provides a step-by-step procedure on how to develop an automated archival processwith NCEI. The information provided therein has been propagated to other user communities, outside of the IOOS RAs, who are learning know how to better manage their data. This year, large portions of the oceanographic data management community, outside of IOOS, requested access and referenced to the NCEI-IOOS Goolge Site for archive discussions. The community response to this resource has indicated that this was a much-needed gap that has finally been filled.

CICS staff has developed a process to move information from an NCEI metadata management system to an NCEI standard operating procedure for automated archive, to facilitate timely archiving of consistent data sets coming from IOOS RAs. Previously, NCEI staff were required to manually transfer information from the metadata management system ATRAC into a standard operating procedure for NCEI IT staff to implement. Through the development of automated archival processes for IOOS RAs, CICS staff has developed an automated pathway for information to be transferred from the ATRAC system to an NCEI standard operating procedure. Thus significantly reducing the amount of effort required for NCEI personnel to establish a procedure for automatically archiving oceanographic data. This new paradigm is also a significant step forward in the recent merger and formation of NCEI. Since the ATRAC system was developed specifically at the National Climatic Data Center (NCDC) and will now be an integral part in the archival process at NCEI, and will be of significant importance in the new NCEI governance structure.

In addition, CICS staff consolidated NCEI wide feedback on the IOOS certification applications. The certification applications ensure that the data collected and distributed by the RAs are managed according to the best practices, as identified by NOAA. CICS staff have reviewed and consolidated responses from NCEI for two applications for certification.

Figure 3. A screen capture of the NCEI-IOOS Google Site cookbook page where Integrated Ocean Observing System Regional Association (IOOS RA) data managers can find information about archive activities at NCEI and how to format and archive their data through an automated process.

In an attempt to assist IOOS RA data providers in formatting their data following current best practices, CICS staff developed a set of “gold standard” example netCDF files. These “gold standard” netCDF examples are examples of how to format a netCDF file following the current discrete sampling geometries as identified by Unidata, Climate and Forecast (CF), and Attribute Conventions for Dataset Discovery (ACDD) netCDF metadata conventions. As data providers are generating new netCDF files for submission to the NCEI archive, they have been using these gold standard examples as a resource for compiling their data files. In addition to the example files, CICS staff have also compiled numerous documents which detail out using the netCDF gold standard example files on various compliance checkers, software tools, and web services. The results of this work can be viewed at the NCEI netCDF templates web page for template versions 1.1 (https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/) and 2 (https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/) in the Feature Type Templates and Examples table.

Planned work        

No future work is planned due to the lack of funding for the IOOS Project. However, below is a list of the future work if we were funded:

• Continue to act as the liaison for NCEI to work with the IOOS Program Office and interact with all eleven Regional Associations to enable smooth transfer of data into the archives and respond to any operational problems.
• Archive other IOOS related data sources, like ecosystem projects and animal telemetry data.
• Provide documentation for using NCEI services for IOOS RA archived data.

Products      

1. NCEI netCDF Gold Standard Example Files

(“Gold Standard Example” column at https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/#templatesexamples and https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/#templatesexamples)

2. NCEI netCDF Gold Standard Example File Documentation

(“Gold Standard File Testing Report” column at https://www.nodc.noaa.gov/data/formats/netcdf/v2.0/#templatesexamples and https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/#templatesexamples)

Presentations       

Biddle, M.,NCEI-IOOS Regional Archiving. 2017 IOOS Data Management and Communication Meeting. Silver Spring, MD. March 2017.

Biddle, M.,NetCDF Gold Standard Examples: A Walkthrough of the NCEI Templates. 2017 NOAA Environmental Data Management Workshop. Bethesda, MD. Jan 2017.

Biddle, M., NetCDF Overview to the National Institute of Fisheries Science (NIFS). NCEI NIFS face-to-face workshop. Silver Spring, MD. Oct 2016

Other              

Nomination for NCEI Innovative Product Award for developing documentation for and data examples promoting the use of NCEI netCDF templates. As quoted from a colleague: “His goal is our goal: to make data more usable, more interoperable, and more relevant.”