Driver and Quality Control
Modules for data extraction and quality control facilitate harmonization and provides a near-real-time assessment of data. In Earth system applications, scientists usually face the burden of collecting and transforming data of different sources. That task is considerable complex and time-consuming, especially if metadata and technical specifications need to be properly aggregated. In that context, modules for data transformation and quality control have been developed to provide harmonization and quality control of data recorded by automatic systems (e.g. buoys or meteorological stations) and field-campaigns. The data and system operations are openly available by the O2A, the observation to archive and analysis framework. The quality control module relies on web-based applications, like the sensor REST-API, to request measurement properties and subsequently build a table of devices, and search - either by absolute or derivative values - for correctness and validity of observations. Thus, a product-oriented quality control is performed and delivers quality-flagged data to the data service system. The algorithm includes an extensible list of tests, originally based on the ARGO system and made operational after review and assessment of different approaches at the end application. The quality flagging scheme in use is ordinal and compliant with a commonly used UNESCO standard. Further review of the literature and of actual observation networks supports the development of an improved flagging scheme. The new scheme should include first and secondary quality flagging. The latter is a cumulative representation of test results, i.e. the processing history, and can be used together with a scoring system. The long-term goal is a generic approach that support cross-cutting activities in Earth system research through standardized harmonization and quality control of data. Particularly, the modularization will allow data services and scientists to build a seamless way to collect data, as well as attributions to data, with considerable optimization for cross-compartmental data driven approaches.
Figure 1 - The modular system for data flow uses the “driver” and “quality control” in a generic or specific “ingest” that connects to other components (“sensor” and “data service”) to provide interaction on both ends for operator and scientist.
For a continuously improved documentation, please refer to http://data.awi.de/o2a-doc and https://spaces.awi.de/x/22WjEw
For the operational data web system with quality-flagged data, please refer to https://dashboard.awi.de/data-xxl/