Last updated: 17 Dec, 2003

This project was formulated as a result of a DFO Science study that looked at how different kinds of data were managed in the sector. One of the obvious points was that though many regions handle the same data, they do so in different ways. These differences can be complementary or redundant. It was suggested that a study that would look in more detail at data management and exchange from the point of view of documenting current practices and attempting to converge to standard practices as appropriate. Such a task can be a very large job considering the diverse types of data handled by Science. In an effort to focus the problem, we chose to limit the details to dealing with data collected as ocean profiles.

The proposal for the work is provided in annex 1. It describes five areas to treat and these form the five major sections of this report.

Participants are mindful that Canada is not alone in the management of ocean data and we wanted to be sure that this project took appropriate steps to ensure that international standards, where they existed, were considered. This was done only in the section on QC standards. It was deemed not useful to examine the variety of formats nor variables nor exchange practices of international bodies. We chose to keep our work focused on matters internal to Canada with due regard to how data exchange with international partners would be impacted.

• Review and document currently existing archival formats in use at MEDS and regions with a view to rationalization (particularly in Newfoundland and Gulf regions)
• Examine the xml study carried out under SSF funding and determine its relevance to data exchange for profile data and possibly other data types.
• Assuming the relevance of the xml work, document how each partner would accomplish reading and writing xml and decide on exchange procedures.
• We will work as appropriate to extend the present xml structure to handle more than ocean profile data and document this work.
• We will test the use of OODB technology working in combination with the current xml profile structure as a future strategy for archiving profile data.

This was the first area tackled and was done by surveying the formats of profile data that are routinely sent to MEDS. Initially we present a brief summary of the structure of each format, then we perform an intercomparison of these formats. A sample of each format is provided in annexes. Following this, we compare each format against the content that was built for the xml bricks study. A brief introduction to xml bricks is given, followed by tables which compare the content of profile formats and bricks.

Review of profile formats

Comparing contents of profiles and xml bricks

• Evaluation of QC standards currently in use and relevant standards proposed by BioChem.
• Evaluation and documentation of the differences between MEDS QC and regional centre QC and ensure there is agreement on common issues.
• Agreement on a common baseline set of QC procedures to be implemented by MEDS and regional centers.
• Documentation of the procedures so that clients using the data will know what has been done.
• We will agree on the use of climatologies including those that already exist and those that may need to be created.

This was the next major area in the standards project, and was done by examining QC documentation from different countries and organizations. A web page was created that organized all the various links to documentation into a table. This page was then used to compile documentation explaining what each place does for quality control. This documentation includes tables which compare the QC procedures of various countries to those for GTSPP.

Documentation Webpage

Evaluation and comparison of QC procedures

• Agreement on the types of duplications that could and can be treated.
• Specification and implementation of a means of version identification and control.
• Agreement on a common strategy for referencing cruises.

The issue of duplicates handling was approached by building on the preliminary analysis conducted by Sencial, Kennedy, and Benjamin. The issue of how to identify duplications was the first area covered, and it was followed by a description of how MEDS deals with duplicates. A section describing an alternate way of handling duplicates was also written.

Data versions and duplicates

• Compare the use of codes for referencing variables used by all partners seeking to identify commonalities
• Agree either to use the same coding (first choice), or to some other strategy (second choice) that allows unambiguous matching of variables between partners..
• Implement a common repository, a mode of access by all and where and how the master copy will be maintained.
• Implementation of the agreed coding standard by all parties.

Data dictionary (xls file) (if you are asked for a password click "cancel")

• Review and document the present exchange practices to identify weaknesses in maintaining timely and complete archives at MEDS.
• We will set out solutions to solve these weaknesses taking advantage of the solutions proposed and implemented in other sub-projects in this proposal.
• We will set targets for timeliness and completeness. These targets will document the roles and responsibilities of all partners.
• We will implement the solutions.

Current exchange practices between MEDS and other organizations were documented to allow for comparison.  For each organization, a list of questions concerning exchange practices were answered.  The answers to these questions allow for the identification of weaknesses in maintaining timely and complete archives at MEDS.

Documentation of exchange practices

Project Proposal

1. Project Title: Standards for Ocean Profile Data Management and Exchange
2. Name, Title/Position and E-mail of the Project Manager

Bob Keeley - ISDM

Robert.Keeley@dfo-mpo.gc.ca

3. Name of the Project Investigators:

Joe Linguanti - IOS

Bernard Pelchat – IML

Robert Nowlen - Gulf

John O'Neill – BIO

Dave Senciall - NWAFC

4. Objective(s) of the Work

Electronic data exchange is an every day procedure. At present, all regions of DFO have their own techniques for processing, storing and exchanging data with each other. This diversity creates problems in differing data and information content being collected, stored and exchanged. The objective of this project is to standardize all aspects of data and information management and exchange procedures starting with ocean profile data. With experience, we intend to broaden the coverage of these standards to other data held by DFO Science.

5. Aspect(s) of the DFO Science priorities to be addressed

MEDS acts as the designated archive for a number of data types including ocean profile data. A significant portion of the workload associated with this is consumed by dealing with the different formats of data received from contributors. Similar problems are faced by all Institutes receiving data. In developing standards for ocean profile data we will be reducing the format handling overheads but more importantly standardizing the data and information content. All this will improve the effectiveness of managing data within DFO Science.

6. Summary of Proposed Work

• Review existing practices (formats, contents, QC procedures, duplicates management, version control, exchange procedures)
• Review of standards emerging from SSF XML project.
• Agreement on standards
• Develop implementation plans for mapping or conversion of existing archives and procedures to the agreed standards
• Carry out implementation

7. Major Deliverables

• National standards for managing ocean profile data
• Improved data exchange efficiency

1. Background

Developing standards for all aspects of handing data within DFO Science is a large undertaking. We will approach this problem by starting in an area that is familiar and already has good cooperation between regions. Dealing with ocean profile data is a suitable choice because it meets these criteria. Many of the regions and MEDS have a long history of managing and exchanging these types of data and the problems associated with standardization are fairly well understood. There is also some complimentary work associated with setting standards for biological and chemical discrete samples that is very relevant to this project. We intend to build on and supplement the activities of that project.

The BioChem project is a multi-region project to handle plankton and water sample data (at present) within a single relational data base structure. To do this, it is recognized that all regions must reach agreement on data base design and content. The design part is mostly set now and participants are coming to address questions of content. At a recent BioChem meeting, three sub projects were initiated to start the process. Each of these is relevant to the goal of developing standards.

The first sub project was to poll participants and develop a draft on common quality control procedures for water sample data. That draft is complete and will form the basis of the standard. The present draft does not discuss details of how climatologies should be produced or used as well as other unresolved issues. This draft is a good start. There remains to resolve the outstanding questions.

The second sub project was to address management of duplicates. There is a draft document that describes various facets of this issue and has been circulated for discussion. Resolution of the questions raised is needed.

A third sub project has been to poll participants to develop a common data dictionary. A first form of this has been circulated. Some Institutes use codes based on GF-3 while others do not. We must come to agreement on whether a single data dictionary is possible, or whether multiple dictionaries with one to one translators are the appropriate strategy.

This third sub project is closely related to another project that was carried out last fiscal year concerning the use of xml for exchanging ocean profile data. BIO, MEDS and IOS were involved in this project, funded by the SSF. A report is in final preparation and will be available soon. A strategy was developed for dealing with the varied ways the three partners referred to both data and metadata in their archive and processing systems. Developing a data dictionary was one aspect to the project.

A larger part of the xml project was to develop an xml file structure that reflects the inherent structures of ocean profile data and metadata. The project was able to develop the xml so that working software was written to both write a common xml structure from native archive formats of partners, and read an xml file and translate to the native formats. We intend to build on this xml project in the work of developing standards by including the exchange of data in xml as a fourth sub project to the Standards Proposal.

The above elements combine to impact how effectively a designated data centre can ensure their data holdings are up-to-date compared to data providers. The solutions to each of these sub projects must have practical application to ensuring timely and effective data exchange. We propose a fifth sub project that will look at the issue of improving timeliness and completeness of designated data centre holdings. It will contribute ideas to the other four sub projects and may develop standards beyond those considered in the other sub projects.

2. Methods

1. QC Standards

• Evaluation of QC standards currently in use and relevant standards proposed by BioChem.
• Evaluation and documentation of the differences between MEDS QC and regional centre QC and ensure there is agreement on common issues.
• Agreement on a common baseline set of QC procedures to be implemented by MEDS and regional centers.
• Documentation of the procedures so that clients using the data will know what has been done.
• We will agree on the use of climatologies including those that already exist and those that may need to be created.

2. Duplicates Handling

• Agreement on the types of duplications that could and can be treated.
• Specification and implementation of a means of version identification and control.
• Agreement on a common strategy for referencing cruises.

3. Common Data Dictionary

• Compare the use of codes for referencing variables used by all partners seeking to identify commonalities.
• Agree either to use the same coding (first choice), or to some other strategy (second choice) that allows unambiguous matching of variables between partners..
• Implement a common repository, a mode of access by all and where and how the master copy will be maintained.
• Implementation of the agreed coding standard by all parties.

4. Formats – Internal and Exchange

• Review and document currently existing archival formats in use at MEDS and regions with a view to rationalization (particularly in Newfoundland and Gulf regions)
• Examine the xml study carried out under SSF funding and determine its relevance to data exchange for profile data and possibly other data types.
• Assuming the relevance of the xml work, document how each partner would accomplish reading and writing xml and decide on exchange procedures.
• We will work as appropriate to extend the present xml structure to handle more than ocean profile data and document this work.
• We will test the use of OODB technology working in combination with the current xml profile structure as a future strategy for archiving profile data.

5. Ensuring timely and complete archives.

• Review and document the present exchange practices to identify weaknesses in maintaining timely and complete archives at MEDS.
• We will set out solutions to solve these weaknesses taking advantage of the solutions proposed and implemented in other sub-projects in this proposal.
• We will set targets for timeliness and completeness. These targets will document the roles and responsibilities of all partners.
• We will implement the solutions.

3. Relationship to existing hardware and software systems

9. Communication of Results

The results of these discussions will be documents that specify the standards to which each data provider will adhere in their own processing systems and when exchanging data.

Each participant will develop an annex to the standards document described above. In the annex they will provide details of the steps to be taken to implement the standards. There will be resource implications to this work and so each annex will need to estimate what these will be.

 

10. Potential Opportunities for Co-ordination/Partnership/Leveraging

 

11. Funding :

Item	2003/04
	Contract	In Kind (days)
MEDS: student	10K
MEDS: host meeting of participants	10K
MEDS: contributions to design		20
BIO: contributions to design		20
BIO: build standard QC tests	5K
IML: implementation of some agreed standards	5K
IML: contributions to design		20
NWAFC: student	10K
NWAFC: contributions to design and implementation		25
Gulf:: contributions to design		10
Pacific: Documentation by contractor	5K
Pacific: contributions to design		20
Pacific: Implementation of agreed standards	5K
Total

 

12. References