Data Ingestion Portal

Wake up your data - set them free for Blue Society

How can you join the European operational oceanography data exchange

Organisations in Europe that are operating coastal and marine monitoring stations and that are not yet connected are invited to make their data streams part of the European operational oceanography data exchange and ultimately discoverable and accessible at the EMODnet Physics portal.

The following image illustrates how operators can include the data streams of their platforms. It starts with contacting the central helpdesk at: cmems-service@ifremer.fr

Operational Oceanography Workflow // oo-workflow-v3.jpg (74 K)

The CMEMS-INSTAC helpdesk will perform a short interview and then bring the platforms operator in contact with the right contact at one of the EuroGOOS RDACs, who will give further instructions and help with arranging the data exchange. 

The following cases can exist:

  • If the data provider can set up the data flow according the defined standards, the regional coordinator only has to link and include the new catalogue and data stream
  • If the data provider cannot setup the data flow (because of lack of experience, technical capacity etc), the regional coordinator has to work on harvesting the data from the provider, harmonize and format these data and make them available from the regional catalogue.

At regional level according to the platform type and parameters the following principles will apply:

Data acquisition:
Data are collected through direct links with the institutions

  • Direct connection is established (usually through ftp protocol) between the RDAC and the data provider
  • Information is provided about the required metadata that should be supplied together with data (ex. station position, date, frequency of measurement, platform name, depth of each sensor, contact person, PI, etc.)
  • Guidance is also provided on how the required daily and monthly files should be created.
  • Information exchanged about the QC procedures
  • Data are provided in the originator’s native format, no need for conversion to NetCDF.
  • The conversion to a unique NetCDF format is performed by the RDAC staff.

Quality Control:

  • Apply automatic quality control procedures on each parameter, elaborated in coherence with international agreement (in particular SeaDataNet).
  • Procedures applied after agreement with the data originators in order to avoid conflicts and effort duplications.

Validation/Assessment:

  • Assess the consistency of the data over a period of time in an area. The aim is to detect possible incoherencies with nearby data  that could not be detected by automatic QC.

In common practice connecting a new data provider will involve the following steps:

  1. Contacting the CMEMS-INSTAC helpdesk (cmems-service@ifremer.fr) for a short interview and directing the data producer (PU) to the right contact at one of the RDACs
  2. Meeting by RDAC contact with data producer to present data flow, infrastructures, common standards, vocabularies (the meeting might be joined by a EuroGOOS and EMODnet Physics representative)
  3. Identification of PU technical capabilities and needs
  4. Identification and collection of the required metadata that should be supplied together with data (ex. station position, date, frequency of measurement, platform name, depth of each sensor, contact person, PI, etc.) according harmonized vocabularies (SeaDataNet, EDMO for institution, unique platform name (e.g. WMO number, ICES platform Id, etc)
  5. Setting up of a permanent data collection channel (the most used is ftp protocol for fixed stations and drifting buoys or ARGO, THREDDS for HFRadar) between the PU and RDAC
  6. If needed, guidance on how the required daily and monthly files should be created; data are to be provided in the originator’s native format, no need for conversion to NetCDF which will be performed by the RDACs staff.
  7. Information exchange about the QC procedures.
  8. Quality control. QC procedures are applied after agreement with the data originators in order to avoid conflicts and effort duplications. Quality control procedures are automatic and are applied on each parameter. These procedures are documented in a EuroGOOS DATAMEQ report.
  9. RDAC procedure for data indexing are updated and data are stored in a ftp repository (folders to separate latest data from older data and to split operational data from research opportunity data are used)
  10. Routinely (three times a day), the EMODnet Physics collects new data files from all RDACs and make them available for discovery, pre-viewing, download (NetCDF and ASCII csv), and machine-to-machine interoperability (WMS, WFS and web services).
  11. Data flow monitoring. EMODnet Physics applies both automatic data flow monitoring (controls if data is available, if any connection/data flow failure occurred, etc.) and periodic manual controls.
  12. Periodically RDAC, CMEMS-INSTAC, and EMODnet Physics assess the consistency of the data in order to identify possible incoherencies in both data and metadata and dataflow. In case any actor of the pipeline identifies an error, this notification goes both downstream and upstream in order to track it and let the right actor to correct it.
  13. There can be cases whereby data from a new PU at first is exchanged and displayed at the EMODnet Physics portal. Such a pilot after a while should be followed by a structural data exchange through the RDACs and CMEMS-INSTAC.
  14. In addition arrangements can be made with a SeaDataNet data centre for further validation of the collected datasets and inclusion in the SeaDataNet data management infrastructure for long term stewardship. Alternatively the data provider can decide to ingest the datasets by means of the Data Submission service at this portal whereby it will be received by a SeaDataNet data centre for further processing.