ProcessLevelsAndQC

Process Levels

The IMOS NetCDF File Naming Convention defines five data process levels, as described in the table below. One NetCDF file will only contain one process level. This means that the output from processing a data set could consist of up to five NetCDF files.

The IMOS Toolbox produces data to fit only within the first two levels - raw and QC'd data. Raw (level 0) data is that which has been retrieved as is by the toolbox. QC-d (level 1) data is that which has been passed through pre-processing routines, automatic and manual quality control procedures. The pre-processing routines will potentially transform data values and/or add new derived parameters. The manual and automatic quality control procedures will flag data in the dataset (for example to flag outliers and spikes) but will not remove any data from the original dataset.

IMOS data process level definitions

Level	File Version	Brief Description	Detailed Description
0	FV00	Raw Data	Raw data is defined as unprocessed data and data products that have not undergone quality control. The data may be in engineering units or physical units, time and locations details can be in relative units and values can be pre-calibration measurements. Level 0 data is not suitable for public access within IMOS.
1	FV01	Quality Controlled Data	Quality controlled data have passed quality assurance procedures such as automated routines and sensor calibration or visual inspection and removal of obvious errors. The data are in physical units using standard SI metric units with calibration and other pre-processing routines applied, all time and location values are in absolute coordinates to comply with standards and datum. Data includes flags for each measurement to indicate the estimated quality of the measurement. Metadata exists for the data or for the higher level dataset that the data belongs to. This is the standard IMOS data level and is what should be made available to AODN and to the IMOS community.
2	FV02	Derived Products	Derived products require scientific and technical interpretation. Normally these will be defined by the community that collects or utilises the data.
3	FV03	Interpreted Products	These products require researcher driven analysis and interpretation, model based interpretation using other data and / or strong prior assumptions.
4	FV04	Knowledge Products	These products require researcher driven scientific interpretation and multidisciplinary data integration and include model-based interpretation using other data and / or strong prior assumptions.

Quality Control

Every data point in an IMOS NetCDF file has an associated quality control flag value which indicates the trustworthiness of the data. There are a number of different sets of QC flags which are available for use; the standard IMOS set is shown below.

A file can either contain one 'blanket' QC value which is attributed to every data point, or an individual QC value for every data point. For example, in a raw data file, every point will have the same QC flag, ('0' if using the standard IMOS QC set).

IMOS standard QC set

Flag Value	Meaning	Description
0	No QC performed	The level at which all data enter the working archive. They have not yet been quality controlled.
1	Good data	Top quality data in which no malfunctions have been identified and all real features have been verified during the quality control process.
2	Probably good data	Good data in which some features (probably real) are present but these are unconfirmed. Code 2 data are also data in which minor malfunctions may be present but these errors are small and / or can be successfully corrected without seriously affecting the overall quality of the data.
3	Bad data that are potentially correctable	Suspect data in which unusual, and probably erroneous features are observed.
4	Bad data	Obviously erroneous values are observed.
5	Value Changed	Altered by a QC Centre, with original values (before the change) preserved in the history record of the profile.
6	Not used	Flag 6 is reserved for future use.
7	Not used	Flag 7 is reserved for future use.
8	Not used	Flag 8 is reserved for future use.
9	Missing Value	Indicates that the element is missing.