From heterogeneous marine sensors to sensor web: (near) real-time open data access adopting OGC sensor web enablement standards

Abstract

Sensor engineering is continuously evolving as devices become cheaper, smaller, more intelligent, and more efficient. Today, oceanographic sensors aim at monitoring marine processes by means of physical, chemical, and biological variables, and use different data formats, units, parameters, resolutions, data quality standards, and protocols. Therefore, integration and interoperability at European level represent a challenge.

To cope with this challenge, the Sensor Web Enablement (SWE) standards, developed by the Open Geospatial Consortium (OGC), are a good solution, as they ensure interoperability and long-term archiving of data series with complete information for all devices.

The interoperability allows integrating information from different sources or pre-existing architectures, such as those developed in the SANY project (http://www.opengeospatial.org/ogc/regions/SANY) or by the US Integrated Ocean Observing System (https://github.com/ioos).

In this paper, we illustrate the real-time data management system, developed by the Italian National Oceanographic Data Centre (NODC) at the Istituto Nazionale di Oceanografia e di Geofisica Sperimentale, OGS in Trieste – Italy, designed to share data acquired by a large number of heterogeneous observing platforms. This system adopts Observations and Measurements (O&M) and Sensor Model Language (SensorML) as data and metadata formats, as well as the Sensor Observation Service (SOS) released by the 52°North as server and Web client for open data access.

The work done shows that the choice to manage real-time data using OGC Sensor Web Enablement (SWE) standards can be considered a valid solution with pros and cons. Nevertheless, in the future, the SWE community will grow, and with it, the number of applications to manage SWE standards to simplify their adoption.

Introduction

Interoperability is, by definition, ‘the ability of computer systems to exchange and make use of information’ [1].

A great challenge within the European oceanographic community is to identify a standard system for sharing data. In this context, the Open Geospatial Consortium (OGC) [2] has established standards focused on sensors and networks of sensors, called Sensor Web Enablement (SWE). The standards proposed by the OGC Consortium are a good solution to meet the need for interoperable data available in near-real time. These standards make sensor metadata and sensor observations easier to collect and share.

The main objective of this work is to describe our experience and approach with the use of SWE standards. The decision to adopt these standards was the result of the need to share real-time data at European level. The observing platforms, managed by OGS, were recently included in European research infrastructures, greatly boosting the interoperability of the real-time operations. At a national level, OGS data management system was initially developed to deal with real-time stations for local civil protection. The developed structure was composed of some simple and rudimentary elements and applications without any standardization.

The involvement in some European projects, aimed at sharing real-time data, gave us the opportunity to develop a standardized system for interoperable data.

This entails an evolution in our data management that has forced us to heavily modify applications involving several complex changes to the structure of the real-time database used to store data in order to include the elements used for standardization. Such blending is visible in the actual data flow, where old elements are mixed with the new ones. Using SWE standards seemed the best solution to share data at European level.

This led us to the development of new applications to manage the data flow using SWE standards, such as Observations and Measurements (O&M) and Sensor Model Language (SensorML).

Finally, the applications developed by 52°North allow us to adopt and adapt the real-time data management system to our needs. The first version of the 52°North software Sensor Observation Service (SOS) used was 3.2 together with O&M and SensorML version 1.0 and over time these have been updated to the latest version of SOS (4.3) and O&M and SensorML (2.0).

Sensor web enablement

The OGC is an international organization involved in standardization to improve the sharing of the world’s geospatial data (Fig. 1), its members develop and maintain SWE standards. Sensor Web is an innovative concept, it is considered a new type of Web-based monitoring of different kinds of real-time phenomena. In the near future, each type of sensor will be accessible at global level. From this point of view, millions of sensors will be connected in a large network, and geo-located observations will be shared. Each sensor will contribute to a global vision of physical phenomena [3].

Fig. 1

Open Geospatial Consortium Sensor Web Enablement schema, modified [17]

These standards enable users to discover, access, and use all types of sensors, transducers, and sensor data repositories via the Web, by means of standard protocols and interfaces.

Among the OGC’s SWE standards, we have adopted and applied the following:

• O&M: models and schema for encoding sensor observations;

• SensorML: models and schema for describing sensor characteristics;

• SOS: web service to access SensorML and O&M.

Observations and measurements

O&M is the OGC standard [4] that defines a schema for measurements and observations. These models can be used for the exchange of information between different scientific and technical communities.

Communications take place by Hypertext Transfer Protocol (HTTP) POSTand GET standard request methods, using operations such as InsertObservation and GetObservation (utilized to store and extract information in a standard way).

Sensor model language

As O&M standard, SensorML is used to provide a standard description of the technical details of sensors, such as system capabilities, physical properties, electrical requirements, operational capabilities, and system input and output. It is defined by the OGC as a model for standardizing information about sensors [5]. As with O&M, the connection with SOS occurs using standard operations: InsertSensor, DescribeSensor, UpdateSensor, and DeleteSensor.

Sensor observation service

The integration of a sensor’s data in an application using a common approach is complicated. A good solution is the use of the SOS that allows the management of information and measurements collected from different sensors. It uses an interoperable approach, adopting OGC specifications [6]. This solution is based on different elements: Observations (measurements acquired by each sensor), Procedure (describing the sensor), ObservedProperty (parameter acquired), FeatureofInterest (indicating sensor positions, amongst other things), and Offering (representing a group of sensors). These different elements are accessible using standard HTTP methods.

Methods

This section describes the experience in the use of SWE standards gained by the Italian National Oceanographic Data Centre (NODC) at the Istituto Nazionale di Oceanografia e di Geofisica Sperimentale, OGS in Trieste – Italy, to share real-time data acquired by two observatories: MAMBO1 and E2M3A. We describe the data flow for managing real-time data, illustrating in detail the elements composing it.

Lastly, a short semantic description of O&M and SensorML profiles developed is illustrated.

The workflow (Fig. 2) developed for this purpose is based on six different elements, which are [7]:

• Observatories (MAMBO1 and E2M3A);

• Real-Time Loader (RTLoader);

• Real-Time Database (RTDB);

• Real-Time Web Service (RTWS);

• Real-Time Sensor Observation Service (RTSOS), and

• Real-Time SOS Web Client (RTWebSOS) using version 4.3 of the 52°North implementation (http://52north.org/).

Fig. 2

Overview of data flow describing the system components developed for data management and their interactions

The data acquired by the two observatories are sent in (near) real time to the OGS land server. Then, the RTLoader converts data coming from different kinds of instruments and with different source formats into a homogeneous format and then stores them in an RTDB. Subsequently, the stored data are evaluated by the application of different standard validation protocols and later read by an application loader that periodically queries new measurements. These data are added to the RTSOS server and visualized using RTWebSOS (REST interface). The stored data are also read by a web service (RTWS) whose goal is to produce NetCDF-CF OceanSites files for data distribution.

Observatories

Two marine observatories acquiring meteo-oceanographic data in (near) real time are currently maintained by OGS: the meteo-marine buoy MAMBO1 (Monitoraggio AMBientale Operativo1), located in the Gulf of Trieste, North Adriatic Sea, and the deep observatory named E2M3A (Eastern Mediterranean Multidisciplinary Moored Array), located in the South Adriatic Sea (Fig. 3). The coastal observatory MAMBO1, located at the outer limit of the Miramare Marine Protected Area at a depth of approximately 18 m, is equipped with a meteorological station and oceanographic sensors. Physical and biogeochemical parameters are continuously monitored at one and/or two levels (1 m and 10 m); additional information can be found at the Web page http://nettuno.ogs.trieste.it/ilter/GoTTs/. Data are acquired twice every hour and transferred via GSM modem to the shore-based receiving station. They are archived at the National Oceanographic Data Centre (NODC-OGS).

Fig. 3

OGS device positions: in red, the location of MAMBO1 in the Gulf of Trieste and E2M3A in the South Adriatic Sea

The deep-sea observatory system E2M3A (http://nettuno.ogs.trieste.it/e2-m3a), is a two-component array, composed of a surface buoy (principal array) and a subsurface mooring (secondary array). The former allows real-time transmission of meteorological parameters as well as surface marine data. The subsurface mooring is equipped with physical sensors at different nominal depths (2, 15, 120, 350, 550, 750, 900, 1000, and 1200 m) and acoustic current profilers located at 320 m and 1200 m. The observatory is deployed to monitor air–sea interactions, and physical and biochemical properties of the water mass, as well as to investigate convective events and the carbon cycle in the open sea [8]. The data are acquired and transmitted through a satellite system allowing real data transfer from the platform to the land station.

For both stations, the original data are acquired in different formats (binary, ASCII), which are set by the manufacturers of the sensors (SeaBird, Sunburst, Young, Pro-Oceanus Systems, etc.), and then stored locally on a dedicated server in OGS. Every 30 min a procedure checks if new data have been acquired and eventually, if in binary format, they are converted by using specific procedures driven by RTLoader.

Real-time loader

The process of converting, processing, and loading data from the observatories is typically asynchronous and prone to disruption due to the long transmission and processing chain. To simplify the development of this application, Real-Time Loader (RTLoader), the Java framework ‘Apache Camel’, was chosen. It offers all the components needed to carry out this kind of task with sufficient resilience, such as transactional routes and persistent queues.

The queues ensure the asynchronicity of the process, thus implying more resilience. The performance of this component is tuned to handle a large amount of small input files. In case of large files, memory consumption grows. For this reason, the configuration of the hardware was designed with particular attention.

The main steps of the process are:

• Continuous check for updates of input data files, sent by observatories, which can also reside on remote systems;

• Parsing of input files in several formats, defined by different sensor manufacturers (SeaBird, Sunburst, Young, Pro-Oceanus Systems, etc.), and their conversion into lists of Java objects, one object for every input row, containing measurements of several parameters (Text2Java);

• Conversion of all these kinds of Java objects into a unique O&M format, regardless of the input formats, containing data and metadata (Java2OM). Metadata of instruments reside in the relational database and are obtained from it. The result of this phase is a list of O&M items;

• Conversion of O&M items into JPA entities (Java Persistence API, used to map java objects to database tables) and their entry into the relational database using JPA Camel components (OM2Db);

• Use of Camel to orchestrate all these components using several ‘routes’ (the main concept in a routing and mediation engine) and decoupling the different phases with queues. This provides the process asynchronicity needed to obtain a good level of resilience with limited resources (e.g., relational database) and a reduction of the memory footprint, useful for achieving good scalability.

Real-time database

The Real-Time Database (RTDB) is a PostgreSQL relational database used to store observations [9]. To manage information from the stations, more than 30 tables are used, distributed on three distinct schemata: a ‘public’ schema, used to store real-time data and metadata, an ‘oceansites’ schema, which includes information needed to produce NetCDF OceanSites files, and an ‘sos’ schema developed specifically to integrate sensor information needed to load measurements to the SOS server.

The structure of the database has three branches:

• section dedicated to describing the instruments, their characteristics, and their position (deployment);

• section used to store data and related metadata (measurements);

• section reserved to store the vocabularies used to standardize data and metadata and the data used by quality control algorithms (common vocabularies and quality control).

Then, data are stored in the database, and a sequence of validation procedures (data quality control procedures) is applied to the information to qualify the data values [10]. The procedure has been developed following the European protocols [11, 12] and is gradually being tuned to the regional statistics [13].

The quality control procedures include the following series of automatic checks [14]: checks for missing data and data format completeness, check of the date/time and of the measuring position, check for duplicate vertical profiles or measures, check for spikes by testing the data for large differences between adjacent values, and check for invalid values by comparison with minimum and maximum values set for each parameter archived.

Real-time web service

The Real-Time Web Service (RTWS) is a RESTful Web Service [9] that accepts simple HTTP requests to extract data from the database. These requests are parameterized for each device (site), featuretype (timeseries- TS or profile - PR), datatype (mooring – MO or CTD profile - CT) and period (DAY, MONTH or YEAR) e.g. {ws application}/search/site/E2M3A/featuretype/TS/ datatype/MO/period/DAY}.

Without any further temporal parameters (start date and end date), the last day (month or year) before the request is downloaded. It is written using Java and open source libraries, such as Spring and Jersey.

Real-time sensor observation service

This standardized Real-Time Sensor Observation Service (RTSOS) is a Web service realized using version 4.3 of the 52°North (http://52north.org) Sensor Observation Service. Real-time data are loaded by an application through the RESTful standard request InsertObservation, using O&M version 2.0. Currently, the information relating to sensors is loaded manually via the SOS Client (http://nodc.ogs.trieste.it/sos/client) with a standard request (InsertSensor) by SensorML version 2.0.

The description of each different sensor can be accessed via OGC SensorML standard request (DescribeSensor). The related data are stored in a dedicated PostgreSQL/PostGIS database, which can be extracted using GetObservation requests.

Real-time SOS web client

In this work, we use the Real Time Sensor Observation Service Web client (RTWebSOS - JavaScript Sensor Web Client, version 1.0.0, https://github.com/52North/js-sensorweb-client) developed by 52°North [15], which is an application with a user-friendly interface (Fig. 4) that allows plotting and downloading data. It hides SWE protocols and gives the opportunity for anyone to interact with SWE technology. Specifically, this Web client is usable with common browsers, and provides a direct link to data, using functions such as searching, plotting, and downloading.

Fig. 4

Screenshot of OGS Sensor Observation Service Web Client (http://nodc.ogs.trieste.it/sosWeb/) showing a temperature time series recorded at the E2M3A

From the main page (http://nodc.ogs.trieste.it/sosWeb/), it is possible to search sensors, identify phenomena, and define time intervals. The user can overlay multiple time series for visual comparison and can download measurements.

Semantic interoperability

Interoperability of sensors is guaranteed by OGC Sensor Web Enablement standards, such as Observations and Measurements (O&M) and Sensor Model Language (SensorML). They provide a standard way to exchange information, whereas semantic interoperability is assured by the adoption of SeaDataNet Common Vocabularies (http://www.seadatanet.org/Standards-Software/Common-Vocabularies), defined as follows: “Common vocabularies consist of lists of standardized terms that cover a broad spectrum of disciplines of relevance to the oceanographic and wider community. Using standardized sets of terms solves the problem of ambiguities associated with data markup and also enables records to be interpreted by computers. This opens up data sets to a whole world of possibilities for computer aided manipulation, distribution and long-term reuse” [16]. These ontologies are used in several European and Italian projects (e.g. SeaDataNet, EmodNet, ODIP and Ritmare).

To adopt these vocabularies, new O&M and SensorML profiles are being developed for describing and encoding sensor observations and characteristics.

Standard definitions are included in the XML files to define sensor categories (L05), sensor devices (L22), observable properties (P01, P02 and P03), and their storage units (P06):

<!-- System Identifiers -->

<sml:IdentifierList>

<sml:identifier>

<sml:Term definition="http://vocab.nerc.ac.uk/collection/L22/current/TOOL0018">

<sml:label>Long_Name</sml:label>

<sml:value>Sea-Bird SBE 37-SMP MicroCAT C-T Sensor</sml:value>

</sml:Term>

</sml:identifier>

</sml:IdentifierList>

<!-- System Classifiers -->

<sml:classifier>

<sml:Term definition="http://vocab.nerc.ac.uk/collection/P02/current/DOXY">

<sml:label>Intended Application4</sml:label>

<sml:value>Dissolved oxygen parameters in the water column</sml:value>

</sml:Term>

</sml:classifier>

<sml:classifier>

<sml:Term definition="http://vocab.nerc.ac.uk/collection/L05/current/130">

<sml:label>Sensor Type</sml:label>

<sml:value>CTD</sml:value>

</sml:Term>

</sml:classifier>

Specifically, the following vocabularies are adopted: Parameter Usage Vocabulary (P01), SeaDataNet Parameter Discovery Vocabulary (P02), SeaDataNet Agreed Parameter Groups (P03), British Oceanographic Data Centre (BODC) data storage units (P06), SeaDataNet device categories (L05), SeaDataNet keyword types (L19), SeaVoX Device Catalogue (L22), and SeaDataNet metadata entities (L23).

To precisely identify the information on observations and characteristics of sensors, the URI of the terms is used (e.g. http://vocab.nerc.ac.uk/collection/P01/current/PHXXZZXX):

<!-- System Output -->

<sml:OutputList>

<sml:output name="pH">

<swe:Quantity definition="http://vocab.nerc.ac.uk/collection/P01/current/PHXXZZXX">

<swe:description>pH per unit volume of the water body</swe:description>

<swe:uom xlink:href="http://vocab.nerc.ac.uk/collection/P06/current/UUPH"code="pH_units"/>

</swe:Quantity>

</sml:output>

</sml:OutputList>.

Results and discussion

This paper has detailed the system developed at NODC/OGS to share real-time oceanographic data acquired by fixed observatories. The need to manage a high number of different acquisition platforms and to make them interoperable at European level has increased the necessity to update the real-time management system. As a first step, we identified a system based on standard elements, later adopted for the real-time management system. The best solution was the OGC SWE standards to manage several sensor buoys.

Consequently, we updated the system following these standards: O&M to upload data, SensorML to describe sensors, SOS to integrate observations and sensor descriptions and SOS Web Client to visualize and download information. These elements were integrated in a pre-existing structure composed of: RTLoader used to homogenize the data format and upload data, RT Database utilized to store data and RT Web Service developed to extract real-time data.

The inclusion of SWE standards in the old system was really complex, visible in the complicated structure of the data-flow. However, this choice was the right one in the long term, because SWE standards are nowadays increasingly used in the European oceanographic community, and the sharing of real-time data applying these standards is one of the main goals for several European infrastructures.

The Fixed-Point Open Ocean Observatory network (FixO3) project recently proposed a good example of interoperability, adopting SWE standards as a solution for sharing data acquired by different observatories.

Through a demo version of a SOS Web client, it is possible to visualize and to compare data acquired by OGS platforms with other different platforms (Fig. 5). An advantage of this application is the opportunity to match the same parameters acquired using different sensors located in different parts of Europe, thus evidencing the interoperability between platforms.

Fig. 5

Screenshot of an application demo of SWE standards developed in the framework of the FixO3 EU project (in blue, the position of the sensors). A comparison between temperature time series recorded at different platforms is shown

Other communities are working in this direction. For instance, within the Joint European Research Infrastructure Network for Coastal Observatories (JERICO-NEXT), OGS, together with other partners, is discussing the opportunity to choose SWE standards as a method to share real-time data. In SeaDataNet II, we tested the connection between historical and real-time data adopting the inclusion of SWE services in the metadata index of individual data sets (Common Data Index). This approach will be extensively developed during the next phase of this project (SeaDataCloud). These aims are ambitious and not without challenges because consolidated methodologies would have to be modified.

Conclusions

The results of the development of this system showed that the choice to manage real-time data using OGC SWE standards represents a good solution with some pros and cons. Positive elements are the interoperability and the opportunity to pass from local to global sharing of data. Thanks to the work done by 52°North, a further pro is the possibility to use applications that easily embed SWE standards.

On the other hand, a negative element faced in this work was the need to update a pre-existing system. In particular, this required the modification of the applications used to manage the archiving process and of the structure of the real-time database.

A complex aspect in the adoption of OGC standards was the development of a common metadata profile for sensors (SensorML) and measurements (O&M). The European oceanographic community is still working on these definitions, adopting common SeaDataNet vocabularies.

In the future, for the benefit of users, we encourage the oceanographic SWE community to increase the number of real-time sensors adopting this system in order to have a global vision of physical phenomena. Millions of sensors will be connected in a large network, observations will be shared, and each sensor will contribute to an integrated vision.

In this way, the SWE community will grow, and with it, the number of applications managing SWE standards, thus simplifying their adoption.