diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 00000000..73c1734b --- /dev/null +++ b/.travis.yml @@ -0,0 +1,10 @@ +language: java +#notifications: +# email: false +jdk: + - openjdk6 + - openjdk7 + - oraclejdk7 +# - oraclejdk8 +install: true +script: mvn clean install \ No newline at end of file diff --git a/52n-sos-importer-bindings/pom.xml b/52n-sos-importer-bindings/pom.xml index b6a50826..b98e3410 100644 --- a/52n-sos-importer-bindings/pom.xml +++ b/52n-sos-importer-bindings/pom.xml @@ -33,7 +33,7 @@ src/main/resources - datafile_configuration.xsd + import-configuration.xsd target/generated/xmlbeans diff --git a/52n-sos-importer-bindings/src/main/resources/datafile_configuration.xsd b/52n-sos-importer-bindings/src/main/resources/datafile_configuration.xsd deleted file mode 100644 index 115adfa6..00000000 --- a/52n-sos-importer-bindings/src/main/resources/datafile_configuration.xsd +++ /dev/null @@ -1,790 +0,0 @@ - - - - - - TODO enter docuementation here. - - - - Character for marking non content lines in CSV files. Length restriction set to 1 character. - - - - - - - - - - The UCUM code used to observed property. - - - - - Character for separating element in content lines in CSV files. Length restriction set to 1 character. - - - - - Encoding of the datafile, e.g. "UTF-8" - - - - - Integer identifying the used EPSG (between 0... - - - - - - - - - - - Defining the first line with data. Line counting starts with 0. - - - - - - - - - - A ID for linking resources to column and vice versa. - - - - - - - - A link to another resource using an ID. - - - - - - - - The key of a key-value-pair. - - - - - - Indicates the membership of this column in a POSITION or DATE_TIME group. - - - - - - - - - - - - - - - - - - - - - - - - - Human readable name of a resource like FOI, Sensor, Observed Property,... - - - - - Reference the columns in the data file. Column counting starts with 0. - - - - - - - - - - Password for the remote data file. Length restriction of 256 characters. - - - - - - - - - - Operating System compliant definition of the path to the local data file. - - - - - - - - Optional regular expression the data files have to match if Path is pointing to an directory. Specify the expression in conformance to http://docs.oracle.com/javase/6/docs/api/index.html?java/util/regex/Pattern.html - - - - - - - - Character for indicating text in elements in CSV files. Length restriction set to 1 character. - - - - - - - - - - Type of the column. It defines the class of content. The application handles the optional metadata to parse the content depending on this type definition, for example. - - - - - - Do not export this column. It should be ignored by the application. - - - - - The result of the performed observation, in most cases some value. - - - - - The date or time or date and time of the performed observation. - - - - - The position of the performed observation. - - - - - The feature of interest. - - - - - The observed property. - - - - - The Unit of measure using UCUM codes. - - - - - The sensor id. - - - - - - - - - The value of a key-value-pair. Potential values: - TYPE->MeasuredValue: NUMERIC | COUNT | BOOLEAN | TEXT; TYPE->DATE_TIME: (COMBINATION | UNIX_TIME). - - - - - - - - Any URL compliant with RFC 3986 "URI" (https://tools.ietf.org/html/rfc3986) - - - - - - - Parameter used for generated resources - - - - - - - - - Any URL compliant with RFC 3986 "URI" (https://tools.ietf.org/html/rfc3986) or better RFC 1738 "URL" (https://tools.ietf.org/html/rfc1738) - - - - - - - - Define to use (case "true") or not to use (case "false") the potential header information to search for metadata in the data file. - - - - - Username for the remote data file. Length restriction of 256 characters. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - The unit of measurement related to THIS entity. The content of this element MUST match a ID element of a UnitOfMeasurement from AdditionalMetadata OR a column Number. - - - - - - - - - - - The observed property related to THIS entity. The content of this element MUST match a ID element of a ObservedProperty from AdditionalMetadata OR a column Number. - - - - - - - - - - - The sensor related to THIS entity. The content of this element MUST match a ID element of a Sensor from AdditionalMetadata OR a column Number. - - - - - - - - - - - The feature of interest related to THIS entity. The content of this element MUST match a ID element of a FeatureOfInterest from AdditionalMetadata OR a column Number. - - - - - - - - - - - A feature of interest (linked to a column by URI element and relatedFOI element in column metadata) - - - - - - - - - - A unit of measure related to a measured value column (linked by URI element and relatedUOM in column metadata in measured value column) - - - - - - - - - - A observed property (linked to a column by URI element and relatedObservedProperty element in column metadata) - - - - - - - - - - A sensor (linked to a column by URI element and relatedFOI element in column metadata) - - - - - - - - - - Altitude element of a position - - - - - - - - - - - - Additional metadata not contained in any column. - - - - - - - A sensor (linked to a column by ID element and relatedSensor element in column metadata) - - - - - An observed property (linked to a column by URI element and relatedObservedProperty element in column metadata) - - - - - - A unit of measurement related to a measured value column (linked by URI element and relatedUOM in column metadata in measured value column) - - - - - In the case of having a column of FOIs, the related position information is stored here. - - - - - - - - - - - - - - A column in the data file. This elements holds the configuration how to parse the data elements in this column. In the case of a MeasuredValue column, the elements Related* are only present, if multiple resources of type * are present in the current set-up/file. - - - - - - - - - A position using x,y,z (unit for each), epsg-code or the group this position is build from - - - - - This optional element links THIS measured value column to a DateTimke group in the model by its group ID. It is defined in a Metadata element with the Key.GROUP. When this element is not defined, all measured value columns share the same date time group. - - - - - The feature of interest related to THIS measured value column. The content of this element MUST match a ID element of a FeatureOfInterest from AdditionalMetadata OR a column Number. - - - - - Column number of the measured value column this column is related to. - - - - - - - - - - The observed property related to THIS measured value column. The content of this element MUST match a ID element of a ObservedProperty from AdditionalMetadata OR a column Number. - - - - - WHY NOT LIMITED TO 1? The sensor related to THIS measured value column. The content of this element MUST match a ID element of a Sensor from AdditionalMetadata OR a column Number. - - - - - The unit of measurement related to THIS measured value column. The content of this element MUST match a ID element of a UnitOfMeasurement from AdditionalMetadata OR a column Number. - - - - - - - - Information about each column in the data file. Holding instructions how to parse the elements of each column. - - - - - - - - - - Metada regarding the CSV format and each column in the data file. - - - - - - - The global decimal mark. It delimits the decimal places of a numeric value. For example, '.' would be the decimal separator for the numeric value "1234.56". - - - - - - - - - - - Optional credentials for the remote data file. - - - - - - - - - - - The file holding the observations in CSV format. - - - - - - - - - - If set to TRUE the LocaleFile.Path or RemoteFile.URL - contains a regular expression needing special handling - before retrieving the data file. - - - - - - - If set to TRUE the last modified date of the datafile - will be used as date value for the observation of this - data file. - - - - - - - If available and "useDateFromLastModifiedDate" is set - to TRUE, the date value will be set to n days before - last modified date. - - - - - - - - - - - - If present, the contained regular expression will be - applied to the file name of the datafile to extract - date information in combination with the - "dateInfoPattern". Hence, this pattern is used to - extract the date string and the dateInfoPattern is used - to convert this date string into valid date information. - The pattern MUST contain one group that holds all date - information! - - - - - - - MUST be set, if regExDateInfoInFileName is set, for - converting the date string into valid date information. - Supported pattern elements: y, M, d, H, m, s, z, Z - - - - - - - Identifies the header line. MUST be set in the case of - having the header line repeatedly in the data file. - - - - - - - - The decimal mark delimits the decimal places of a numeric value. For example, '.' would be the decimal separator for the numeric value "1234.56". - - - - - - - - - - Reference to a position group defined in the CsvMetadata.ColumnAssignments - - - - - - - - - - Latitude element of a position - - - - - - - - - - - - A data file located on the same machine. - - - - - - - - - - - - Longitude element of a position - - - - - - - - - - - - A key-value-pair holding some additional metadata. - - - - - - - - The value of a key-value-pair. Potential values: - TYPE->MeasuredValue: NUMERIC | COUNT | BOOLEAN | TEXT - TYPE->DATE_TIME: COMBINATION | UNIX_TIME - IMPORT_STRATEGY->SingleObservation | SweArrayObservationWithSplitExtension - HUNK_SIZE->Integer value defining the number of rows that should be combined - in one SWEArrayObservation. - - - - - - - - - Offering in the SOS which is used for the import. Might be generated from sensor id. - - - - - - - If set to "true", the offering should be generated from the sensor id. If set to "false", the xs:string content of element "Offering" should be used as offering for inserting the data. - - - - - - - - - Basic CSV file parameters. - - - - - - - - - - - - A position using x,y,z, unit for each, epsg-code - - - - - - - Altitude element of a position. Presence depends on CRS. - - - - - - - - - Links to a position group defined in the CsvMetadata.ColumnAssignments.Column.Metadata.Key::Group - - - - - - - - A data file located on a remote server which might required credentials to be accessed. - - - - - - - - - - - Metadata regarding the SOS limited to URL and offering. - - - - - - - Offering in the SOS which is used for the import. Might be generated from sensor id. - - - - - Indicates the specification version implemented by the SOS instance. E.g. 1.0.0, 2.0.0. - - - - - If version 2.0.0 is used, the binding has to be defined, e.g. SOAP, POX binding. - - - - - - - - - Configuration for a data file containing observation. These observation should be imported to a running SOS instance using the transactional interface. It is used by the 52n-sensorweb-importer to store the metadata and re-use it. - - - - - - - - - - - diff --git a/52n-sos-importer-bindings/src/main/resources/import-configuration.xsd b/52n-sos-importer-bindings/src/main/resources/import-configuration.xsd new file mode 100644 index 00000000..59f43783 --- /dev/null +++ b/52n-sos-importer-bindings/src/main/resources/import-configuration.xsd @@ -0,0 +1,932 @@ + + + + + + TODO enter documentation here. + + + + Character for marking non content lines in CSV files. Length restriction set to 1 character. + + + + + + + + + + The UCUM code used to observed property. + + + + + Character for separating element in content lines in CSV files. Length restriction set to 1 character. + + + + + Encoding of the datafile, e.g. "UTF-8" + + + + + Integer identifying the used EPSG (between 0... + + + + + + + + + + + Defining the first line with data. Line counting starts with 0. + + + + + + + + + + A ID for linking resources to column and vice versa. + + + + + + + + A link to another resource using an ID. + + + + + + + + The key of a key-value-pair. + + + + + + Indicates the membership of this column in a POSITION or DATE_TIME group. + + + + + + + + + + + + + + + + + + + + + + + + + Human readable name of a resource like FOI, Sensor, Observed Property,... + + + + + Reference the columns in the data file. Column counting starts with 0. + + + + + + + + + + Password for the remote data file. Length restriction of 256 characters. + + + + + + + + + + Operating System compliant definition of the path to the local data file. + + + + + + + + Optional regular expression the data files have to match if Path is pointing to an directory. Specify the expression in conformance to http://docs.oracle.com/javase/6/docs/api/index.html?java/util/regex/Pattern.html + + + + + + + + Character for indicating text in elements in CSV files. Length restriction set to 1 character. + + + + + + + + + + Type of the column. It defines the class of content. The application handles the optional metadata to parse the content depending on this type definition, for example. + + + + + + Do not export this column. It should be ignored by the application. + + + + + The result of the performed observation, in most cases some value. + + + + + The date or time or date and time of the performed observation. + + + + + The position of the performed observation. + + + + + The feature of interest. + + + + + The observed property. + + + + + The Unit of measure using UCUM codes. + + + + + The sensor id. + + + + + + + + + The value of a key-value-pair. Potential values: + TYPE->MeasuredValue: NUMERIC | COUNT | BOOLEAN | TEXT; TYPE->DATE_TIME: (COMBINATION | UNIX_TIME). + + + + + + + + Any URL compliant with RFC 3986 "URI" (https://tools.ietf.org/html/rfc3986) + + + + + + + Parameter used for generated resources + + + + + + + + + Any URL compliant with RFC 3986 "URI" (https://tools.ietf.org/html/rfc3986) or better RFC 1738 "URL" (https://tools.ietf.org/html/rfc1738) + + + + + + + + Define to use (case "true") or not to use (case "false") the potential header information to search for metadata in the data file. + + + + + Username for the remote data file. Length restriction of 256 characters. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + The unit of measurement related to THIS entity. The content of this element MUST match a ID element of a UnitOfMeasurement from AdditionalMetadata OR a column Number. + + + + + + + + + + + The observed property related to THIS entity. The content of this element MUST match a ID element of a ObservedProperty from AdditionalMetadata OR a column Number. + + + + + + + + + + + The sensor related to THIS entity. The content of this element MUST match a ID element of a Sensor from AdditionalMetadata OR a column Number. + + + + + + + + + + + The feature of interest related to THIS entity. The content of this element MUST match a ID element of a FeatureOfInterest from AdditionalMetadata OR a column Number. + + + + + + + + + + + A feature of interest (linked to a column by URI element and relatedFOI element in column metadata) + + + + + + + + + + A unit of measure related to a measured value column (linked by URI element and relatedUOM in column metadata in measured value column) + + + + + + + + + + A observed property (linked to a column by URI element and relatedObservedProperty element in column metadata) + + + + + + + + + + A sensor (linked to a column by URI element and relatedFOI element in column metadata) + + + + + + + + + + Altitude element of a position + + + + + + + + + + + + Additional metadata not contained in any column. + + + + + + + A sensor (linked to a column by ID element and relatedSensor element in column metadata) + + + + + An observed property (linked to a column by URI element and relatedObservedProperty element in column metadata) + + + + + + A unit of measurement related to a measured value column (linked by URI element and relatedUOM in column metadata in measured value column) + + + + + In the case of having a column of FOIs, the related position information is stored here. + + + + + + + + + + + + + + A column in the data file. This elements holds the configuration how to parse the data elements in this column. In the case of a MeasuredValue column, the elements Related* are only present, if multiple resources of type * are present in the current set-up/file. + + + + + + + + + A position using x,y,z (unit for each), epsg-code or the group this position is build from + + + + + This optional element links THIS measured value column to a DateTimke group in the model by its group ID. It is defined in a Metadata element with the Key.GROUP. When this element is not defined, all measured value columns share the same date time group. + + + + + The feature of interest related to THIS measured value column. The content of this element MUST match a ID element of a FeatureOfInterest from AdditionalMetadata OR a column Number. + + + + + Column number of the measured value column this column is related to. + + + + + + + + + + The observed property related to THIS measured value column. The content of this element MUST match a ID element of a ObservedProperty from AdditionalMetadata OR a column Number. + + + + + WHY NOT LIMITED TO 1? The sensor related to THIS measured value column. The content of this element MUST match a ID element of a Sensor from AdditionalMetadata OR a column Number. + + + + + The unit of measurement related to THIS measured value column. The content of this element MUST match a ID element of a UnitOfMeasurement from AdditionalMetadata OR a column Number. + + + + + + + + Information about each column in the data file. Holding instructions how to parse the elements of each column. + + + + + + + + + + Metada regarding the CSV format and each column in the data file. + + + + + + + The global decimal mark. It delimits the decimal places of a numeric value. For example, '.' would be the decimal separator for the numeric value "1234.56". + + + + + + + + + + + Optional credentials for the remote data file. + + + + + + + + + + + The file holding the observations in CSV format. + + + + + + + + + + + + + + + + If set to TRUE the LocaleFile.Path or + RemoteFile.URL contains a regular expression + needing special handling before retrieving the + data file. + + + + + + + If set to TRUE the last modified date of the + datafile will be used as date value for the + observation of this data file. + + + + + + + If available and "useDateFromLastModifiedDate" + is set to TRUE, the date value will be set to n + days before last modified date. + + + + + + + + + + + + If present, the contained regular expression + will be applied to the file name of the datafile + to extract date information in combination with + the "dateInfoPattern". Hence, this pattern is + used to extract the date string and the + dateInfoPattern is used to convert this date + string into valid date information. The pattern + MUST contain one group that holds all date + information! + + + + + + + MUST be set, if regExDateInfoInFileName is set, + for converting the date string into valid date + information. Supported pattern elements: y, M, + d, H, m, s, z, Z + + + + + + + Identifies the header line. MUST be set in the + case of having the header line repeatedly in the + data file. + + + + + + + Identifies the beginning of an new sample in the data + file. Requires the presence of the following + attributes: + * sampleDateOffset + * sampleDateExtractionRegEx + * sampleDatePattern + * sampleDataOffset + A "sample" is a single sampling run having additional + metadata like date information which is not contained + in the lines. + + + + + + + Defines the offset in lines between the first line of + a sample and the line containing the date information. + + + + + + + + + + + + Regular expression to extract the date information from + the line containing the date information of the current + sample. The expression MUST result in ONE group. This + group will be parsed to a java.util.Date using + "sampleDatePattern" attribute. + + + + + + + Defines the pattern used to parse the date information + of the current pattern. + + + + + + + Defines the offset in lines from sample beginning till + the first lines with data. + + + + + + + + + + + + Defines the offset in lines from sample beginning till + the line containing the sample size in lines with data. + + + + + + + + + + + + Defines a regular expression to extract the sample size. + The regular expression MUST result in ONE group which + contains an integer value. + + + + + + + + + The decimal mark delimits the decimal places of a numeric value. + For example, '.' would be the decimal separator for the numeric + value "1234.56". + + + + + + + + + + + Reference to a position group defined in the CsvMetadata.ColumnAssignments + + + + + + + + + + + A regular expression identifying lines that can be ignored during the + import process. + + + + + + + + + + + Latitude element of a position + + + + + + + + + + + + A data file located on the same machine. + + + + + + + + + + + + Longitude element of a position + + + + + + + + + + + + A key-value-pair holding some additional metadata. + + + + + + + + The value of a key-value-pair. Potential values: + TYPE->MeasuredValue: NUMERIC | COUNT | BOOLEAN | TEXT + TYPE->DATE_TIME: COMBINATION | UNIX_TIME + IMPORT_STRATEGY->SingleObservation | SweArrayObservationWithSplitExtension + HUNK_SIZE->Integer value defining the number of rows that should be combined + in one SWEArrayObservation. + + + + + + + + + Offering in the SOS which is used for the import. Might be generated from sensor id. + + + + + + + If set to "true", the offering should be generated from the sensor id. If set to "false", the xs:string content of element "Offering" should be used as offering for inserting the data. + + + + + + + + + Basic CSV file parameters. + + + + + + + + + + + + A position using x,y,z, unit for each, epsg-code + + + + + + + Altitude element of a position. Presence depends on CRS. + + + + + + + + + Links to a position group defined in the CsvMetadata.ColumnAssignments.Column.Metadata.Key::Group + + + + + + + + A data file located on a remote server which might required credentials to be accessed. + + + + + + + + + + + Metadata regarding the SOS limited to URL and offering. + + + + + + + + Offering in the SOS which is used for the + import. Might be generated from sensor id. + + + + + + + Indicates the specification version + implemented by the SOS instance. E.g. 1.0.0, + 2.0.0. + + + + + + + If version 2.0.0 is used, the binding has to + be defined, e.g. SOAP, POX binding. + + + + + + + + An additional timeout buffer for connect and socket + timeout when using import strategy + "SweArrayObservationWithSplitExtension". Scale is in + milliseconds, e.g. 1000 => 1s more connect and socket + timeout. + The size of this value is related to the set-up of the + SOS server, importer, and the HUNK_SIZE value. + The current OX-F SimpleHttpClient implementation uses + a default value of 5s, hence setting this to 25,000 + results in 30s connection and socket timeout. + + + + + + + + + + + + + + Configuration for a data file containing observation. These observation should be imported to a running SOS instance using the transactional interface. It is used by the 52n-sensorweb-importer to store the metadata and re-use it. + + + + + + + + + + + diff --git a/52n-sos-importer-core/src/main/java/org/n52/sos/importer/Constants.java b/52n-sos-importer-core/src/main/java/org/n52/sos/importer/Constants.java index fd7bc7b7..3d4d0ce4 100644 --- a/52n-sos-importer-core/src/main/java/org/n52/sos/importer/Constants.java +++ b/52n-sos-importer-core/src/main/java/org/n52/sos/importer/Constants.java @@ -53,9 +53,9 @@ * TODO move to shared module all constants that need to be shared! */ public class Constants { - + private static final Logger logger = LoggerFactory.getLogger(Constants.class); - + public static final String BOOLEAN = "BOOLEAN"; public static final String COMBINATION = "COMBINATION"; public static final String COUNT = "COUNT"; @@ -65,14 +65,14 @@ public class Constants { private static final int DEFAULT__COLOR_BACKGROUND_COMPONENT_GREEN = 238; private static final int DEFAULT__COLOR_BACKGROUND_COMPONENT_RED = 238; public static final Color DEFAULT_COLOR_BACKGROUND = Color.getHSBColor( - Color.RGBtoHSB(DEFAULT__COLOR_BACKGROUND_COMPONENT_RED, - DEFAULT__COLOR_BACKGROUND_COMPONENT_GREEN, + Color.RGBtoHSB(DEFAULT__COLOR_BACKGROUND_COMPONENT_RED, + DEFAULT__COLOR_BACKGROUND_COMPONENT_GREEN, DEFAULT__COLOR_BACKGROUND_COMPONENT_BLUE, null)[0], - Color.RGBtoHSB(DEFAULT__COLOR_BACKGROUND_COMPONENT_RED, - DEFAULT__COLOR_BACKGROUND_COMPONENT_GREEN, + Color.RGBtoHSB(DEFAULT__COLOR_BACKGROUND_COMPONENT_RED, + DEFAULT__COLOR_BACKGROUND_COMPONENT_GREEN, DEFAULT__COLOR_BACKGROUND_COMPONENT_BLUE, null)[1], - Color.RGBtoHSB(DEFAULT__COLOR_BACKGROUND_COMPONENT_RED, - DEFAULT__COLOR_BACKGROUND_COMPONENT_GREEN, + Color.RGBtoHSB(DEFAULT__COLOR_BACKGROUND_COMPONENT_RED, + DEFAULT__COLOR_BACKGROUND_COMPONENT_GREEN, DEFAULT__COLOR_BACKGROUND_COMPONENT_BLUE, null)[2]); public static final Font DEFAULT_LABEL_FONT = new Font("SansSerif", Font.PLAIN, 12); public static final Font DEFAULT_LABEL_FONT_BOLD = new Font("SansSerif", Font.BOLD, 12); @@ -89,7 +89,7 @@ public class Constants { public static final String XML_SCHEMA_PREFIX = "xsi"; public static final String XML_SCHEMA_NAMESPACE = "http://www.w3.org/2001/XMLSchema-instance"; public static final QName XML_SCHEMALOCATION_QNAME = new QName(XML_SCHEMA_NAMESPACE,"schemaLocation",XML_SCHEMA_PREFIX); - public static final String XML_SOS_IMPORTER_SCHEMA_LOCATION = "https://raw.github.com/52North/sos-importer/master/52n-sos-importer-bindings/src/main/xsd/datafile_configuration.xsd"; + public static final String XML_SOS_IMPORTER_SCHEMA_LOCATION = "https://raw.github.com/52North/sos-importer/master/52n-sos-importer-bindings/src/main/xsd/import-configuration.xsd"; public static final String UNICODE_OFFERING_PREFIX = "_offering-"; public final static String SEPARATOR_STRING = "SEP"; public static final String SPACE_STRING = Lang.l().spaceString(); @@ -124,20 +124,20 @@ public class Constants { public static final String WMS_VIEW_SELECT_TOOL_ICON_PNG_PATH = "/org/n52/sos/importer/view/position/noxin_crosshairs.png"; public static final String WMS_DEFAULT_URL = "http://osmtud.dyndns.org/wms/wms/"; public static final String WMS_GET_CAPABILITIES_REQUEST = "?VERSION=1.1.0&REQUEST=GetCapabilities"; - + private static final String WMS_EXTERNAL_FILE_PATH = System.getProperty("user.home") + File.separator + ".SOSImporter" + File.separator; private static final String WMS_INTERNAL_FILE_PATH = "/org/n52/sos/importer/view/position/"; private static final String WMS_FILE_NAME = "wms.properties"; public static final String DEFAULT_FEEDER_JAR_NAME_START = "52n-sos-importer-feeder-"; - + public final static int DIALOG_WIDTH = 800; - + public final static int DIALOG_HEIGHT = 600; public static final String WMS_DEFAULT_BACKGROUND_LAYER_NAME = "OSMBackground"; - + /* * CHANGEABLE VALUES */ @@ -147,7 +147,7 @@ public class Constants { private static String wms_url = "wms_url"; private static String wms_layer = "wms_layer"; - + /** * TODO implement loading of language parameter from configuration file * @return {@link org.n52.sos.importer.view.i18n.Lang.l().getClass().getSimpleName()} @@ -159,28 +159,28 @@ public static String language() { public static String WMS_URL() { final Properties props = load(); String wmsUrl = WMS_DEFAULT_URL; - if (props != null && + if (props != null && props.getProperty(wms_url) != null && !props.getProperty(wms_url).equals("")) { wmsUrl = props.getProperty(wms_url); } - + logger.debug("WMS url: '{}'", wmsUrl); - + return wmsUrl; } - + public static String WMS_BACKGROUND_LAYER_NAME() { final Properties props = load(); String wmsLayer = WMS_DEFAULT_BACKGROUND_LAYER_NAME; - if (props != null && + if (props != null && props.getProperty(wms_layer) != null && !props.getProperty(wms_layer).equals("")) { wmsLayer = props.getProperty(wms_layer); } - + logger.debug("WMS layer: '{}'", wmsLayer); - + return wmsLayer; } @@ -200,7 +200,7 @@ private static Properties load() { logger.info("Load default settings from jar file"); filePath = WMS_INTERNAL_FILE_PATH + WMS_FILE_NAME; is = Constants.class.getClass().getResourceAsStream(filePath); - } else { + } else { logger.info("Load settings from " + file); is = new FileInputStream(file); } @@ -228,21 +228,21 @@ public static void save() { final File folder = new File(WMS_EXTERNAL_FILE_PATH); final Properties props = load(); if (!folder.exists()) { - - final boolean successful = folder.mkdir(); + + final boolean successful = folder.mkdir(); if (!successful) { logger.warn("WMS properties could not be saved."); logger.warn("No writing permissions at " + folder); return; - } + } } - + final File file = new File(WMS_EXTERNAL_FILE_PATH + WMS_FILE_NAME); - + try { //save properties final OutputStream os = new FileOutputStream(file); - props.store(os, null); - logger.info("Save settings at " + file.getAbsolutePath()); + props.store(os, null); + logger.info("Save settings at " + file.getAbsolutePath()); } catch (final IOException e) { logger.error("WMS properties could not be saved.", e); } diff --git a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/Configuration.java b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/Configuration.java index 26f9f88c..a479a056 100644 --- a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/Configuration.java +++ b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/Configuration.java @@ -34,6 +34,7 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; +import java.util.LinkedList; import java.util.regex.Pattern; import javax.xml.namespace.QName; @@ -71,1062 +72,1196 @@ */ public final class Configuration { - private static final Logger LOG = LoggerFactory.getLogger(Configuration.class); - - private static final String POSITION_PARSEPATTERN_LATITUDE = "LAT"; - private static final String POSITION_PARSEPATTERN_LONGITUDE = "LON"; - private static final String POSITION_PARSEPATTERN_ALTITUDE = "ALT"; - private static final String POSITION_PARSEPATTERN_EPSG = "EPSG"; - // TODO read from configuration file - public static final String SOS_200_EPSG_CODE_PREFIX = "http://www.opengis.net/def/crs/EPSG/0/"; - public static final String SOS_100_EPSG_CODE_PREFIX = "urn:ogc:def:crs:EPSG::"; - public static final String REGISTER_SENSOR_SML_SYSTEM_TEMPLATE = "./SML_1.0.1_System_template.xml"; - private static final String NS_SWE_1_0_1 = "http://www.opengis.net/swe/1.0.1"; - private static final String NS_SOS_1_0_0 = "http://www.opengis.net/sos/1.0"; - public static final QName QN_SOS_1_0_OFFERING = new QName(NS_SOS_1_0_0, "offering"); - public static final QName QN_SOS_1_0_ID = new QName(NS_SOS_1_0_0, "id"); - public static final QName QN_SOS_1_0_NAME = new QName(NS_SOS_1_0_0, "name"); - public static final QName QN_SWE_1_0_1_SIMPLE_DATA_RECORD = new QName(NS_SWE_1_0_1,"SimpleDataRecord"); - public static final QName QN_SWE_1_0_1_DATA_RECORD = new QName (NS_SWE_1_0_1,"DataRecord"); - public static final QName QN_SWE_1_0_1_ENVELOPE = new QName (NS_SWE_1_0_1,"Envelope"); - public static final String SML_ATTRIBUTE_VERSION = "version"; - public static final String SML_VERSION = "1.0.1"; - public static final char UNICODE_REPLACER = '_'; - public static final Pattern UNICODE_ONLY_REPLACER_LEFT_PATTERN = Pattern.compile(UNICODE_REPLACER + "+"); - private static final String COLUMN_SEPARATOR_SPACE = "Space"; - private static final String COLUMN_SEPARATOR_TAB = "Tab"; - public static final String SOS_SENSOR_ALREADY_REGISTERED_MESSAGE_START = "Sensor with ID"; - public static final String SOS_SENSOR_ALREADY_REGISTERED_MESSAGE_END = "is already registered at this SOS"; - public static final String SOS_EXCEPTION_CODE_NO_APPLICABLE_CODE = "NoApplicableCode"; - public static final String SOS_EXCEPTION_OBSERVATION_DUPLICATE_CONSTRAINT = "observation_time_stamp_key"; - public static final String SOS_OBSERVATION_ALREADY_CONTAINED = "observation already contained in sos"; - public static final String SOS_OBSERVATION_TYPE_TEXT = "TEXT"; - public static final String SOS_OBSERVATION_TYPE_COUNT = "COUNT"; - public static final String SOS_OBSERVATION_TYPE_BOOLEAN = "BOOLEAN"; - public static final String OGC_DISCOVERY_ID_TERM_DEFINITION = "urn:ogc:def:identifier:OGC:1.0:uniqueID"; - public static final String OGC_DISCOVERY_LONG_NAME_DEFINITION = "urn:ogc:def:identifier:OGC:1.0:longName"; - public static final String OGC_DISCOVERY_SHORT_NAME_DEFINITION = "urn:ogc:def:identifier:OGC:1.0:shortName"; - public static final String OGC_DISCOVERY_INTENDED_APPLICATION_DEFINITION = "urn:ogc:def:classifier:OGC:1.0:application"; - public static final String OGC_DISCOVERY_OBSERVED_BBOX_DEFINITION = "urn:ogc:def:property:OGC:1.0:observedBBOX"; - - public static final String SOS_EXCEPTION_OBSERVATION_ALREADY_CONTAINED = "This observation is already contained in SOS database!"; - - public static final String SOS_200_OFFERING_ALREADY_REGISTERED_MESSAGE_START = "The offering with the identifier"; - public static final String SOS_200_OFFERING_ALREADY_REGISTERED_MESSAGE_END = "still exists in this service and it is not allowed to insert more than one procedure to an offering!"; - - public static final String SOS_200_DUPLICATE_OBSERVATION_CONSTRAINT = "observation_featureofinterestid_observablepropertyid_proced_key"; - - public static HashMap EPSG_EASTING_FIRST_MAP = null; - static { - EPSG_EASTING_FIRST_MAP = new HashMap(); - EPSG_EASTING_FIRST_MAP.put("default", false); - EPSG_EASTING_FIRST_MAP.put("4326",false); - EPSG_EASTING_FIRST_MAP.put("4979",false); - EPSG_EASTING_FIRST_MAP.put("21037",true); - - } - - public enum ImportStrategy { - /** - * Each value will be inserted as single observation into the SOS. - */ - SingleObservation, - /** - * The file will be read at once. For each identified timeseries - * an OM_SWEArrayObservation will be created and inserted into the - * SOS using the "SplitDataArrayIntoObservations" of the 52North - * implementation. - */ - SweArrayObservationWithSplitExtension; - } - - private SosImportConfiguration importConf; - private final File configFile; - - private Pattern localeFilePattern = null; - - public Configuration(final String pathToFile) throws XmlException, IOException { - LOG.trace("Configuration({})",pathToFile); - configFile = new File(pathToFile); - final SosImportConfigurationDocument sosImportDoc = - SosImportConfigurationDocument.Factory.parse(configFile); - // Create an XmlOptions instance and set the error listener. - final XmlOptions validateOptions = new XmlOptions(); - final ArrayList errorList = new ArrayList(); - validateOptions.setErrorListener(errorList); - - // Validate the XML. - final boolean isValid = sosImportDoc.validate(validateOptions); - - // If the XML isn't valid, loop through the listener's contents, - // printing contained messages. - if (!isValid) { - for (int i = 0; i < errorList.size(); i++) { - final XmlError error = errorList.get(i); - - LOG.error("Message: {}; Location: {}", - error.getMessage(), - error.getCursorLocation().xmlText()); - } - final String msg = "Configuration is not valid and could not be parsed."; - throw new XmlException(msg, null, errorList); - } else { - importConf = sosImportDoc.getSosImportConfiguration(); - setLocaleFilePattern(); - } - } - - private void setLocaleFilePattern() { - if (isRegularExpressionForLocalFileAvailable()) { - localeFilePattern = Pattern.compile(importConf.getDataFile().getLocalFile().getRegularExpresssionForAllowedFileNames()); - } - } - - /** - * For testing only! - */ - protected Configuration(final SosImportConfiguration importConf) { - if (importConf == null) { - throw new IllegalArgumentException("SosImportConfiguration MUST NOT be null or INVALID"); - } - this.importConf = importConf; - configFile = null; - setLocaleFilePattern(); - } - - private boolean isRegularExpressionForLocalFileAvailable() - { - return importConf.getDataFile().isSetLocalFile() && - importConf.getDataFile().getLocalFile().isSetRegularExpresssionForAllowedFileNames() && - importConf.getDataFile().getLocalFile().getRegularExpresssionForAllowedFileNames() != null && - !importConf.getDataFile().getLocalFile().getRegularExpresssionForAllowedFileNames().isEmpty(); - } - - /** - * Returns a File instance pointing to the data file defined in XML import - * configuration. - * @return a new File instance pointing to - * DataFile.LocalFile.Path or
- * null, if element is not defined in config - */ - public File getDataFile() { - LOG.trace("getDataFile()"); - if (importConf.getDataFile() != null && - importConf.getDataFile().isSetLocalFile() && - !importConf.getDataFile().getLocalFile().getPath().equalsIgnoreCase("") ) { - // Path for LocalFile set to something, so return a new File using is - return new File(importConf.getDataFile().getLocalFile().getPath()); - } - LOG.error("DataFile.LocalFile.Path not set!"); - return null; - } - - public File getConfigFile() { - return configFile; - } - - /** - * Returns a truth value according to the presence of the remote file - * element in the xml document. - * - * @return true if it is a remote file - */ - public boolean isRemoteFile() { - return importConf.getDataFile().getRemoteFile() != null; - } - - /** - * Returns the host name of the ftp server. - * - * @return ftp host - */ - public String getFtpHost() { - final String[] splitString = importConf.getDataFile().getRemoteFile().getURL().split("/"); - return splitString[0]; - } - - /** - * Returns a string, that indicates the path of subdirectories, where the - * ftp file is located. - * - * @return subdirectory structure - */ - public String getFtpSubdirectory() { - final String[] splitString = importConf.getDataFile().getRemoteFile().getURL().split("/"); - String result = ""; - // certain file - if (!isFtpUrlRegex()) { - for (int i = 1; i < splitString.length-1; i++) { - result += splitString[i]; - } - } else - // regular expression - { - // TODO - } - - return result; - } - - /** - * Returns the name of the ftp file. - * - * @return ftp file name - */ - public String getFtpFile() { - final String[] splitString = importConf.getDataFile().getRemoteFile().getURL().split("/"); - String result; - // certain file - if (!isFtpUrlRegex()) { - result = splitString[splitString.length-1]; - } else - // regular expression - { - // TODO - result = null; - } - return result; - } - - public boolean isFtpUrlRegex() { - LOG.trace("isSosUrlRegex()"); - return importConf.getDataFile().getReferenceIsARegularExpression(); - } - - /** - * - * @return - * @throws MalformedURLException - */ - public URL getSosUrl() throws MalformedURLException { - LOG.trace("getSosUrl()"); - if (!importConf.getSosMetadata().getURL().equalsIgnoreCase("") ){ - return new URL(importConf.getSosMetadata().getURL()); - } - LOG.error("SosMetadata.URL not set!"); - return null; - } - - public String getUser() { - return importConf.getDataFile().getRemoteFile().getCredentials().getUserName(); - } - - public String getPassword() { - return importConf.getDataFile().getRemoteFile().getCredentials().getPassword(); - } - - /** - * The number of the first line with data. Line counting starts at 0. - * @return - */ - public int getFirstLineWithData() { - return importConf.getCsvMetadata().getFirstLineWithData(); - } - - public char getCsvSeparator() { - final String sep = importConf.getCsvMetadata().getParameter().getColumnSeparator(); - if (sep.equals(Configuration.COLUMN_SEPARATOR_SPACE)) { - return ' '; - } else if (sep.equals(Configuration.COLUMN_SEPARATOR_TAB)) { - return '\t'; - } else { - return sep.charAt(0); - } - } - - public char getCsvQuoteChar() { - return importConf.getCsvMetadata().getParameter().getTextIndicator().charAt(0); - } - - public char getCsvEscape() { - return importConf.getCsvMetadata().getParameter().getCommentIndicator().charAt(0); - } - - /** - * Returns the ids of measured value columns. - * @return An int[] if any measured value column is found. null - * if no column is found. - */ - public int[] getMeasureValueColumnIds() { - LOG.trace("getMeasureValueColumnIds()"); - final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); - final ArrayList ids = new ArrayList(); - for (final Column column : cols) { - if (column.getType().equals(Type.MEASURED_VALUE)){ - if (LOG.isDebugEnabled()) { - LOG.debug(String.format("Found measured value column: %d", column.getNumber())); - } - ids.add(column.getNumber()); - } - } - ids.trimToSize(); - if (ids.size() > 0) { - final int[] result = new int[ids.size()]; - for (int i = 0; i < result.length; i++) { - result[i] = ids.get(i); - } - return result; - } - return null; - } - - /** - * Returns the column id for the given measured value column if available. - * If not -1. - * @param mvColumnId - * @return The column id of the sensor related to this measure value column - * or -1 if no sensor column is available for this column - */ - public int getColumnIdForSensor(final int mvColumnId) { - LOG.trace(String.format("getColumnIdForSensor(%d)", - mvColumnId)); - // check for RelatedSensor element and if its a number -> return number - final Column c = getColumnById(mvColumnId); - if (c.getRelatedSensorArray() != null && c.getRelatedSensorArray().length > 0) { - final RelatedSensor rS = c.getRelatedSensorArray(0); - if (rS.isSetNumber()) { - if (LOG.isDebugEnabled()) { - LOG.debug(String.format("Found RelatedSensor column for measured value column %d: %d", - mvColumnId, - rS.getNumber())); - } - return rS.getNumber(); - } else if (rS.isSetIdRef()) { - if (LOG.isDebugEnabled()) { - LOG.debug(String.format("Found RelatedSensor %s is not a column but a Resource.", rS.getIdRef() )); - } - } else { - LOG.error(String.format("RelatedSensor element not set properly: %s", rS.xmlText())); - } - } - // if element is not set - // get column id from ColumnAssignments - final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); - for (final Column column : cols) { - if (column.getType().equals(Type.SENSOR)) { - if (LOG.isDebugEnabled()) { - LOG.debug(String.format("Found related sensor column for measured value column %d: %d", - mvColumnId, - column.getNumber())); - } - return column.getNumber(); - } - } - return -1; - } - - /** - * - * @param mvColumnId - * @return - */ - public Column getColumnById(final int mvColumnId) { - LOG.trace(String.format("getColumnById(%d)",mvColumnId)); - final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); - for (final Column column : cols) { - if (column.getNumber() == mvColumnId) { - if (LOG.isDebugEnabled()) { - LOG.debug(String.format("Column found for id %d", - mvColumnId)); - } - return column; - } - } - LOG.error(String.format("CsvMetadat.ColumnAssignments not set properly. Could not find Column for id %d.", - mvColumnId)); - return null; - } - - /** - * Returns the SensorType linked to the column, identified by the given id, - * by its RelatedSensor.IdRef element. If no sensor could be found - * null is returned. - * @param mvColumnId - * @return - */ - public SensorType getRelatedSensor(final int mvColumnId) { - LOG.trace(String.format("getRelatedSensor(%d)", - mvColumnId)); - final Column c = getColumnById(mvColumnId); - if (c.getRelatedSensorArray() != null && - c.getRelatedSensorArray().length > 0 && - c.getRelatedSensorArray(0) != null && - c.getRelatedSensorArray(0).isSetIdRef()) { - final String sensorXmlId = c.getRelatedSensorArray(0).getIdRef(); - if (importConf.getAdditionalMetadata() != null && - importConf.getAdditionalMetadata().getSensorArray() != null && - importConf.getAdditionalMetadata().getSensorArray().length > 0) { - for (final SensorType s : importConf.getAdditionalMetadata().getSensorArray()) { - if (s.getResource() != null && s.getResource().getID() != null && s.getResource().getID().equals(sensorXmlId)) { - if (LOG.isDebugEnabled()) { - LOG.debug(String.format("Sensor found for id '%s': %s", - sensorXmlId, - s.xmlText())); - } - return s; - } - } - LOG.debug(String.format("No Sensor found for column '%s'.", - sensorXmlId)); - return null; - } else { - LOG.error("Element AdditionalMetadata.Sensor not found."); - } - } - LOG.debug(String.format("RelatedSensor element not found for given measured value column id %s", - mvColumnId)); - return null; - } - - /** - * - * @param mvColumnId - * @return - */ - public int getColumnIdForFoi(final int mvColumnId) { - LOG.trace(String.format("getColumnIdForFoi(%d)", - mvColumnId)); - // check for RelatedFOI element and if its a number -> return number - final Column c = getColumnById(mvColumnId); - if (c.getRelatedFOIArray() != null && c.getRelatedFOIArray().length > 0) { - final RelatedFOI rF = c.getRelatedFOIArray(0); - if (rF.isSetNumber()) { - LOG.debug(String.format("Found RelatedFOI column for measured value column %d: %d", - mvColumnId, - rF.getNumber())); - return rF.getNumber(); - } else if (rF.isSetIdRef()) { - LOG.debug(String.format("Found RelatedFOI %s is not a column but a Resource.", rF.getIdRef() )); - } else { - LOG.error(String.format("RelatedFOI element not set properly: %s", rF.xmlText())); - } - } - // if element is not set - // get column id from ColumnAssignments - final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); - for (final Column column : cols) { - if (column.getType().equals(Type.FOI)) { - LOG.debug(String.format("Found related feature of interest column for measured value column %d: %d", - mvColumnId, - column.getNumber())); - return column.getNumber(); - } - } - return -1; - } - - /** - * - * @param foiUri - * @return - */ - public Position getFoiPosition(final String foiUri) { - LOG.trace(String.format("getFoiPosition(%s)", - foiUri)); - // get all elements from foi positions and compare the uri - if (importConf.getAdditionalMetadata() != null && - importConf.getAdditionalMetadata().getFOIPositionArray() != null && - importConf.getAdditionalMetadata().getFOIPositionArray().length > 0) { - final FOIPosition[] foiPos = importConf.getAdditionalMetadata().getFOIPositionArray(); - for (final FOIPosition pos : foiPos) { - if (pos.getURI() != null && - pos.getURI().getStringValue() != null && - pos.getURI().getStringValue().equals(foiUri)){ - // if element is found -> fill position - final org.x52North.sensorweb.sos.importer.x02.PositionDocument.Position p = pos.getPosition(); - if (p.isSetAlt() && - p.isSetEPSGCode() && - p.isSetLat() && - p.isSetLong()) { - return getModelPositionXBPosition(p); - } - } - } - } - return null; - } - - /** - * @param p {@link org.x52North.sensorweb.sos.importer.x02.PositionDocument.Position} - * @return {@link org.n52.sos.importer.feeder.model.Position} - */ - public Position getModelPositionXBPosition( - final org.x52North.sensorweb.sos.importer.x02.PositionDocument.Position p) { - LOG.trace("getPosition()"); - final String[] units = new String[3]; - final double[] values = new double[3]; - - if (p.isSetAlt()) { - units[Position.ALT] = p.getAlt().getUnit(); - values[Position.ALT] = p.getAlt().getFloatValue(); - } else { - units[Position.ALT] = Position.UNIT_NOT_SET; - values[Position.ALT] = Position.VALUE_NOT_SET; - } - units[Position.LAT] = p.getLat().getUnit(); - values[Position.LAT] = p.getLat().getFloatValue(); - - units[Position.LONG] = p.getLong().getUnit(); - values[Position.LONG] = p.getLong().getFloatValue(); - - final int epsgCode = p.getEPSGCode(); - return new Position(values, units, epsgCode); - } - - /** - * - * @param mvColumnId - * @return - */ - public FeatureOfInterestType getRelatedFoi(final int mvColumnId) { - LOG.trace(String.format("getRelatedFoi(%d)", - mvColumnId)); - final Column c = getColumnById(mvColumnId); - if (c.getRelatedFOIArray() != null && - c.getRelatedFOIArray(0) != null && - c.getRelatedFOIArray(0).isSetIdRef()) { - final String foiXmlId = c.getRelatedFOIArray(0).getIdRef(); - if (importConf.getAdditionalMetadata() != null && - importConf.getAdditionalMetadata().getFeatureOfInterestArray() != null && - importConf.getAdditionalMetadata().getFeatureOfInterestArray().length > 0) { - for (final FeatureOfInterestType foi : importConf.getAdditionalMetadata().getFeatureOfInterestArray()) { - if (foi.getResource() != null && foi.getResource().getID() != null && foi.getResource().getID().equals(foiXmlId)) { - LOG.debug(String.format("Feature of Interest found for id '%s': %s", - foiXmlId, - foi.xmlText())); - return foi; - } - } - LOG.debug(String.format("No Feature of Interest found for column '%s'.", - foiXmlId)); - return null; - } else { - LOG.error("Element AdditionalMetadata.FeatureOfInterest not found."); - } - } - LOG.debug(String.format("RelatedFOI element not found for given measured value column id %s", - mvColumnId)); - return null; + private static final Logger LOG = LoggerFactory.getLogger(Configuration.class); + + private static final String POSITION_PARSEPATTERN_LATITUDE = "LAT"; + private static final String POSITION_PARSEPATTERN_LONGITUDE = "LON"; + private static final String POSITION_PARSEPATTERN_ALTITUDE = "ALT"; + private static final String POSITION_PARSEPATTERN_EPSG = "EPSG"; + // TODO read from configuration file + public static final String SOS_200_EPSG_CODE_PREFIX = "http://www.opengis.net/def/crs/EPSG/0/"; + public static final String SOS_100_EPSG_CODE_PREFIX = "urn:ogc:def:crs:EPSG::"; + public static final String REGISTER_SENSOR_SML_SYSTEM_TEMPLATE = "./SML_1.0.1_System_template.xml"; + private static final String NS_SWE_1_0_1 = "http://www.opengis.net/swe/1.0.1"; + private static final String NS_SOS_1_0_0 = "http://www.opengis.net/sos/1.0"; + public static final QName QN_SOS_1_0_OFFERING = new QName(NS_SOS_1_0_0, "offering"); + public static final QName QN_SOS_1_0_ID = new QName(NS_SOS_1_0_0, "id"); + public static final QName QN_SOS_1_0_NAME = new QName(NS_SOS_1_0_0, "name"); + public static final QName QN_SWE_1_0_1_SIMPLE_DATA_RECORD = new QName(NS_SWE_1_0_1,"SimpleDataRecord"); + public static final QName QN_SWE_1_0_1_DATA_RECORD = new QName (NS_SWE_1_0_1,"DataRecord"); + public static final QName QN_SWE_1_0_1_ENVELOPE = new QName (NS_SWE_1_0_1,"Envelope"); + public static final String SML_ATTRIBUTE_VERSION = "version"; + public static final String SML_VERSION = "1.0.1"; + public static final char UNICODE_REPLACER = '_'; + public static final Pattern UNICODE_ONLY_REPLACER_LEFT_PATTERN = Pattern.compile(UNICODE_REPLACER + "+"); + private static final String COLUMN_SEPARATOR_SPACE = "Space"; + private static final String COLUMN_SEPARATOR_TAB = "Tab"; + public static final String SOS_SENSOR_ALREADY_REGISTERED_MESSAGE_START = "Sensor with ID"; + public static final String SOS_SENSOR_ALREADY_REGISTERED_MESSAGE_END = "is already registered at this SOS"; + public static final String SOS_EXCEPTION_CODE_NO_APPLICABLE_CODE = "NoApplicableCode"; + public static final String SOS_EXCEPTION_OBSERVATION_DUPLICATE_CONSTRAINT = "observation_time_stamp_key"; + public static final String SOS_OBSERVATION_ALREADY_CONTAINED = "observation already contained in sos"; + public static final String SOS_OBSERVATION_TYPE_TEXT = "TEXT"; + public static final String SOS_OBSERVATION_TYPE_COUNT = "COUNT"; + public static final String SOS_OBSERVATION_TYPE_BOOLEAN = "BOOLEAN"; + public static final String OGC_DISCOVERY_ID_TERM_DEFINITION = "urn:ogc:def:identifier:OGC:1.0:uniqueID"; + public static final String OGC_DISCOVERY_LONG_NAME_DEFINITION = "urn:ogc:def:identifier:OGC:1.0:longName"; + public static final String OGC_DISCOVERY_SHORT_NAME_DEFINITION = "urn:ogc:def:identifier:OGC:1.0:shortName"; + public static final String OGC_DISCOVERY_INTENDED_APPLICATION_DEFINITION = "urn:ogc:def:classifier:OGC:1.0:application"; + public static final String OGC_DISCOVERY_OBSERVED_BBOX_DEFINITION = "urn:ogc:def:property:OGC:1.0:observedBBOX"; + + public static final String SOS_EXCEPTION_OBSERVATION_ALREADY_CONTAINED = "This observation is already contained in SOS database!"; + + public static final String SOS_200_OFFERING_ALREADY_REGISTERED_MESSAGE_START = "The offering with the identifier"; + public static final String SOS_200_OFFERING_ALREADY_REGISTERED_MESSAGE_END = "still exists in this service and it is not allowed to insert more than one procedure to an offering!"; + + public static final String SOS_200_DUPLICATE_OBSERVATION_CONSTRAINT = "observation_featureofinterestid_observablepropertyid_proced_key"; + + public static HashMap EPSG_EASTING_FIRST_MAP = null; + static { + EPSG_EASTING_FIRST_MAP = new HashMap(); + EPSG_EASTING_FIRST_MAP.put("default", false); + EPSG_EASTING_FIRST_MAP.put("4326",false); + EPSG_EASTING_FIRST_MAP.put("4979",false); + EPSG_EASTING_FIRST_MAP.put("21037",true); + + } + + public enum ImportStrategy { + /** + * Each value will be inserted as single observation into the SOS. + */ + SingleObservation, + /** + * The file will be read at once. For each identified timeseries + * an OM_SWEArrayObservation will be created and inserted into the + * SOS using the "SplitDataArrayIntoObservations" of the 52North + * implementation. + */ + SweArrayObservationWithSplitExtension; + } + + private SosImportConfiguration importConf; + private final File configFile; + + private Pattern localeFilePattern = null; + + public Configuration(final String pathToFile) throws XmlException, IOException { + LOG.trace("Configuration({})",pathToFile); + configFile = new File(pathToFile); + final SosImportConfigurationDocument sosImportDoc = + SosImportConfigurationDocument.Factory.parse(configFile); + // Create an XmlOptions instance and set the error listener. + final XmlOptions validateOptions = new XmlOptions(); + final ArrayList errorList = new ArrayList(); + validateOptions.setErrorListener(errorList); + + // Validate the XML. + final boolean isValid = sosImportDoc.validate(validateOptions); + + // If the XML isn't valid, loop through the listener's contents, + // printing contained messages. + if (!isValid) { + for (int i = 0; i < errorList.size(); i++) { + final XmlError error = errorList.get(i); + + LOG.error("Message: {}; Location: {}", + error.getMessage(), + error.getCursorLocation().xmlText()); + } + final String msg = "Configuration is not valid and could not be parsed."; + throw new XmlException(msg, null, errorList); + } else { + importConf = sosImportDoc.getSosImportConfiguration(); + setLocaleFilePattern(); + } + } + + private void setLocaleFilePattern() { + if (isRegularExpressionForLocalFileAvailable()) { + localeFilePattern = Pattern.compile(importConf.getDataFile().getLocalFile().getRegularExpresssionForAllowedFileNames()); + } + } + + /** + * For testing only! + */ + protected Configuration(final SosImportConfiguration importConf) { + if (importConf == null) { + throw new IllegalArgumentException("SosImportConfiguration MUST NOT be null or INVALID"); + } + this.importConf = importConf; + configFile = null; + setLocaleFilePattern(); + } + + private boolean isRegularExpressionForLocalFileAvailable() + { + return importConf.getDataFile().isSetLocalFile() && + importConf.getDataFile().getLocalFile().isSetRegularExpresssionForAllowedFileNames() && + importConf.getDataFile().getLocalFile().getRegularExpresssionForAllowedFileNames() != null && + !importConf.getDataFile().getLocalFile().getRegularExpresssionForAllowedFileNames().isEmpty(); + } + + /** + * Returns a File instance pointing to the data file defined in XML import + * configuration. + * @return a new File instance pointing to + * DataFile.LocalFile.Path or
+ * null, if element is not defined in config + */ + public File getDataFile() { + LOG.trace("getDataFile()"); + if (importConf.getDataFile() != null && + importConf.getDataFile().isSetLocalFile() && + !importConf.getDataFile().getLocalFile().getPath().equalsIgnoreCase("") ) { + // Path for LocalFile set to something, so return a new File using is + return new File(importConf.getDataFile().getLocalFile().getPath()); + } + LOG.error("DataFile.LocalFile.Path not set!"); + return null; + } + + public File getConfigFile() { + return configFile; + } + + /** + * Returns a truth value according to the presence of the remote file + * element in the xml document. + * + * @return true if it is a remote file + */ + public boolean isRemoteFile() { + return importConf.getDataFile().getRemoteFile() != null; + } + + /** + * Returns the host name of the ftp server. + * + * @return ftp host + */ + public String getFtpHost() { + final String[] splitString = importConf.getDataFile().getRemoteFile().getURL().split("/"); + return splitString[0]; + } + + /** + * Returns a string, that indicates the path of subdirectories, where the + * ftp file is located. + * + * @return subdirectory structure + */ + public String getFtpSubdirectory() { + final String[] splitString = importConf.getDataFile().getRemoteFile().getURL().split("/"); + String result = ""; + // certain file + if (!isFtpUrlRegex()) { + for (int i = 1; i < splitString.length-1; i++) { + result += splitString[i]; + } + } else + // regular expression + { + // TODO + } + + return result; + } + + /** + * Returns the name of the ftp file. + * + * @return ftp file name + */ + public String getFtpFile() { + final String[] splitString = importConf.getDataFile().getRemoteFile().getURL().split("/"); + String result; + // certain file + if (!isFtpUrlRegex()) { + result = splitString[splitString.length-1]; + } else + // regular expression + { + // TODO + result = null; + } + return result; + } + + public boolean isFtpUrlRegex() { + LOG.trace("isSosUrlRegex()"); + return importConf.getDataFile().getReferenceIsARegularExpression(); + } + + /** + * + * @return + * @throws MalformedURLException + */ + public URL getSosUrl() throws MalformedURLException { + LOG.trace("getSosUrl()"); + if (!importConf.getSosMetadata().getURL().equalsIgnoreCase("") ){ + return new URL(importConf.getSosMetadata().getURL()); + } + LOG.error("SosMetadata.URL not set!"); + return null; + } + + public String getUser() { + return importConf.getDataFile().getRemoteFile().getCredentials().getUserName(); + } + + public String getPassword() { + return importConf.getDataFile().getRemoteFile().getCredentials().getPassword(); + } + + /** + * The number of the first line with data. Line counting starts at 0. + * @return + */ + public int getFirstLineWithData() { + return importConf.getCsvMetadata().getFirstLineWithData(); + } + + public char getCsvSeparator() { + final String sep = importConf.getCsvMetadata().getParameter().getColumnSeparator(); + if (sep.equals(Configuration.COLUMN_SEPARATOR_SPACE)) { + return ' '; + } else if (sep.equals(Configuration.COLUMN_SEPARATOR_TAB)) { + return '\t'; + } else { + return sep.charAt(0); + } + } + + public char getCsvQuoteChar() { + return importConf.getCsvMetadata().getParameter().getTextIndicator().charAt(0); + } + + public char getCsvEscape() { + return importConf.getCsvMetadata().getParameter().getCommentIndicator().charAt(0); + } + + /** + * Returns the ids of measured value columns. + * @return An int[] if any measured value column is found. null + * if no column is found. + */ + public int[] getMeasureValueColumnIds() { + LOG.trace("getMeasureValueColumnIds()"); + final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); + final ArrayList ids = new ArrayList(); + for (final Column column : cols) { + if (column.getType().equals(Type.MEASURED_VALUE)){ + LOG.debug("Found measured value column: {}", column.getNumber()); + ids.add(column.getNumber()); + } + } + ids.trimToSize(); + if (ids.size() > 0) { + final int[] result = new int[ids.size()]; + for (int i = 0; i < result.length; i++) { + result[i] = ids.get(i); + } + return result; + } + return null; + } + + public int[] getIgnoredColumnIds() { + LOG.trace("getIgnoredColumnIds()"); + final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); + final ArrayList ids = new ArrayList(); + for (final Column column : cols) { + if (column.getType().equals(Type.DO_NOT_EXPORT)){ + LOG.debug("Found ignored column: {}", column.getNumber()); + ids.add(column.getNumber()); + } + } + ids.trimToSize(); + if (ids.size() > 0) { + final int[] result = new int[ids.size()]; + for (int i = 0; i < result.length; i++) { + result[i] = ids.get(i); + } + return result; + } + return null; + } + + /** + * Returns the column id for the given measured value column if available. + * If not -1. + * @param mvColumnId + * @return The column id of the sensor related to this measure value column + * or -1 if no sensor column is available for this column + */ + public int getColumnIdForSensor(final int mvColumnId) { + LOG.trace(String.format("getColumnIdForSensor(%d)", + mvColumnId)); + // check for RelatedSensor element and if its a number -> return number + final Column c = getColumnById(mvColumnId); + if (c.getRelatedSensorArray() != null && c.getRelatedSensorArray().length > 0) { + final RelatedSensor rS = c.getRelatedSensorArray(0); + if (rS.isSetNumber()) { + if (LOG.isDebugEnabled()) { + LOG.debug(String.format("Found RelatedSensor column for measured value column %d: %d", + mvColumnId, + rS.getNumber())); + } + return rS.getNumber(); + } else if (rS.isSetIdRef()) { + if (LOG.isDebugEnabled()) { + LOG.debug(String.format("Found RelatedSensor %s is not a column but a Resource.", rS.getIdRef() )); + } + } else { + LOG.error(String.format("RelatedSensor element not set properly: %s", rS.xmlText())); + } + } + // if element is not set + // get column id from ColumnAssignments + final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); + for (final Column column : cols) { + if (column.getType().equals(Type.SENSOR)) { + if (LOG.isDebugEnabled()) { + LOG.debug(String.format("Found related sensor column for measured value column %d: %d", + mvColumnId, + column.getNumber())); + } + return column.getNumber(); + } + } + return -1; + } + + /** + * + * @param mvColumnId + * @return + */ + public Column getColumnById(final int mvColumnId) { + LOG.trace(String.format("getColumnById(%d)",mvColumnId)); + final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); + for (final Column column : cols) { + if (column.getNumber() == mvColumnId) { + if (LOG.isDebugEnabled()) { + LOG.debug(String.format("Column found for id %d", + mvColumnId)); + } + return column; + } + } + LOG.error(String.format("CsvMetadat.ColumnAssignments not set properly. Could not find Column for id %d.", + mvColumnId)); + return null; + } + + /** + * Returns the SensorType linked to the column, identified by the given id, + * by its RelatedSensor.IdRef element. If no sensor could be found + * null is returned. + * @param mvColumnId + * @return + */ + public SensorType getRelatedSensor(final int mvColumnId) { + LOG.trace(String.format("getRelatedSensor(%d)", + mvColumnId)); + final Column c = getColumnById(mvColumnId); + if (c.getRelatedSensorArray() != null && + c.getRelatedSensorArray().length > 0 && + c.getRelatedSensorArray(0) != null && + c.getRelatedSensorArray(0).isSetIdRef()) { + final String sensorXmlId = c.getRelatedSensorArray(0).getIdRef(); + if (importConf.getAdditionalMetadata() != null && + importConf.getAdditionalMetadata().getSensorArray() != null && + importConf.getAdditionalMetadata().getSensorArray().length > 0) { + for (final SensorType s : importConf.getAdditionalMetadata().getSensorArray()) { + if (s.getResource() != null && s.getResource().getID() != null && s.getResource().getID().equals(sensorXmlId)) { + if (LOG.isDebugEnabled()) { + LOG.debug(String.format("Sensor found for id '%s': %s", + sensorXmlId, + s.xmlText())); + } + return s; + } + } + LOG.debug(String.format("No Sensor found for column '%s'.", + sensorXmlId)); + return null; + } else { + LOG.error("Element AdditionalMetadata.Sensor not found."); + } + } + LOG.debug(String.format("RelatedSensor element not found for given measured value column id %s", + mvColumnId)); + return null; + } + + /** + * + * @param mvColumnId + * @return + */ + public int getColumnIdForFoi(final int mvColumnId) { + LOG.trace(String.format("getColumnIdForFoi(%d)", + mvColumnId)); + // check for RelatedFOI element and if its a number -> return number + final Column c = getColumnById(mvColumnId); + if (c.getRelatedFOIArray() != null && c.getRelatedFOIArray().length > 0) { + final RelatedFOI rF = c.getRelatedFOIArray(0); + if (rF.isSetNumber()) { + LOG.debug(String.format("Found RelatedFOI column for measured value column %d: %d", + mvColumnId, + rF.getNumber())); + return rF.getNumber(); + } else if (rF.isSetIdRef()) { + LOG.debug(String.format("Found RelatedFOI %s is not a column but a Resource.", rF.getIdRef() )); + } else { + LOG.error(String.format("RelatedFOI element not set properly: %s", rF.xmlText())); + } + } + // if element is not set + // get column id from ColumnAssignments + final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); + for (final Column column : cols) { + if (column.getType().equals(Type.FOI)) { + LOG.debug(String.format("Found related feature of interest column for measured value column %d: %d", + mvColumnId, + column.getNumber())); + return column.getNumber(); + } + } + return -1; + } + + /** + * + * @param foiUri + * @return + */ + public Position getFoiPosition(final String foiUri) { + LOG.trace(String.format("getFoiPosition(%s)", + foiUri)); + // get all elements from foi positions and compare the uri + if (importConf.getAdditionalMetadata() != null && + importConf.getAdditionalMetadata().getFOIPositionArray() != null && + importConf.getAdditionalMetadata().getFOIPositionArray().length > 0) { + final FOIPosition[] foiPos = importConf.getAdditionalMetadata().getFOIPositionArray(); + for (final FOIPosition pos : foiPos) { + if (pos.getURI() != null && + pos.getURI().getStringValue() != null && + pos.getURI().getStringValue().equals(foiUri)){ + // if element is found -> fill position + final org.x52North.sensorweb.sos.importer.x02.PositionDocument.Position p = pos.getPosition(); + if (p.isSetAlt() && + p.isSetEPSGCode() && + p.isSetLat() && + p.isSetLong()) { + return getModelPositionXBPosition(p); + } + } + } + } + return null; + } + + /** + * @param p {@link org.x52North.sensorweb.sos.importer.x02.PositionDocument.Position} + * @return {@link org.n52.sos.importer.feeder.model.Position} + */ + public Position getModelPositionXBPosition( + final org.x52North.sensorweb.sos.importer.x02.PositionDocument.Position p) { + LOG.trace("getPosition()"); + final String[] units = new String[3]; + final double[] values = new double[3]; + + if (p.isSetAlt()) { + units[Position.ALT] = p.getAlt().getUnit(); + values[Position.ALT] = p.getAlt().getFloatValue(); + } else { + units[Position.ALT] = Position.UNIT_NOT_SET; + values[Position.ALT] = Position.VALUE_NOT_SET; + } + units[Position.LAT] = p.getLat().getUnit(); + values[Position.LAT] = p.getLat().getFloatValue(); + + units[Position.LONG] = p.getLong().getUnit(); + values[Position.LONG] = p.getLong().getFloatValue(); + + final int epsgCode = p.getEPSGCode(); + return new Position(values, units, epsgCode); + } + + /** + * + * @param mvColumnId + * @return + */ + public FeatureOfInterestType getRelatedFoi(final int mvColumnId) { + LOG.trace(String.format("getRelatedFoi(%d)", + mvColumnId)); + final Column c = getColumnById(mvColumnId); + if (c.getRelatedFOIArray() != null && + c.getRelatedFOIArray(0) != null && + c.getRelatedFOIArray(0).isSetIdRef()) { + final String foiXmlId = c.getRelatedFOIArray(0).getIdRef(); + if (importConf.getAdditionalMetadata() != null && + importConf.getAdditionalMetadata().getFeatureOfInterestArray() != null && + importConf.getAdditionalMetadata().getFeatureOfInterestArray().length > 0) { + for (final FeatureOfInterestType foi : importConf.getAdditionalMetadata().getFeatureOfInterestArray()) { + if (foi.getResource() != null && foi.getResource().getID() != null && foi.getResource().getID().equals(foiXmlId)) { + LOG.debug(String.format("Feature of Interest found for id '%s': %s", + foiXmlId, + foi.xmlText())); + return foi; + } + } + LOG.debug(String.format("No Feature of Interest found for column '%s'.", + foiXmlId)); + return null; + } else { + LOG.error("Element AdditionalMetadata.FeatureOfInterest not found."); + } + } + LOG.debug(String.format("RelatedFOI element not found for given measured value column id %s", + mvColumnId)); + return null; + } + + public Position getPosition(final String group, final String[] values) throws ParseException { + LOG.trace(String.format("getPosition(group:%s,%s)", + group, + Arrays.toString(values))); + final Column[] cols = getAllColumnsForGroup(group, Type.POSITION); + // combine the values from the different columns + final String[] units = new String[3]; + final double[] posValues = new double[3]; + int epsgCode = -1; + for (final Column c : cols) { + // boolean isCombination = false; // now every position is of type combination + for (final Metadata m : c.getMetadataArray()) { + // check for type combination + // if (m.getKey().equals(Key.TYPE) && m.getValue().equals(Configuration.POSITION_TYPE_COMBINATION)) { + // isCombination = true; + // } + // get parse pattern and parse available values + /*else*/ if (m.getKey().equals(Key.PARSE_PATTERN)) { + String pattern = m.getValue(); + pattern = pattern.replaceAll(Configuration.POSITION_PARSEPATTERN_LATITUDE, "{0}"); + pattern = pattern.replaceAll(Configuration.POSITION_PARSEPATTERN_LONGITUDE, "{1}"); + pattern = pattern.replaceAll(Configuration.POSITION_PARSEPATTERN_ALTITUDE, "{2}"); + pattern = pattern.replaceAll(Configuration.POSITION_PARSEPATTERN_EPSG, "{3}"); + + final MessageFormat mf = new MessageFormat(pattern); + Object[] tokens = null; + try { + tokens = mf.parse(values[c.getNumber()]); + } catch (final ParseException e) { + throw new NumberFormatException(); + } + + if (tokens == null) { + throw new NumberFormatException(); + } + + Object[] latitude, longitude, height; + + if (tokens.length > 0 && tokens[0] != null) { + latitude = parseLat((String) tokens[0]); + posValues[Position.LAT] = (Double) latitude[0]; + units[Position.LAT] = (String) latitude[1]; + } + if (tokens.length > 1 && tokens[1] != null) { + longitude = parseLon((String)tokens[1]); + posValues[Position.LONG] = (Double) longitude[0]; + units[Position.LONG] = (String) longitude[1]; + } + if (tokens.length > 2 && tokens[2] != null) { + height = parseAlt((String)tokens[2]); + posValues[Position.ALT] = (Double) height[0]; + units[Position.ALT] = (String) height[1]; + } + if (tokens.length > 3 && tokens[3] != null) { + epsgCode = Integer.valueOf((String)tokens[3]); + } + } + // get additional information + // LATITUDE + else if (m.getKey().equals(Key.POSITION_LATITUDE)) { + final Object[] latitude = parseLat(m.getValue()); + posValues[Position.LAT] = (Double) latitude[0]; + units[Position.LAT] = (String) latitude[1]; + } + // LONGITUDE + else if (m.getKey().equals(Key.POSITION_LONGITUDE)) { + final Object[] longitude = parseLon(m.getValue()); + posValues[Position.LONG] = (Double) longitude[0]; + units[Position.LONG] = (String) longitude[1]; + } + // ALTITUDE + else if (m.getKey().equals(Key.POSITION_ALTITUDE)) { + final Object[] altitude = parseAlt(m.getValue()); + posValues[Position.ALT] = (Double) altitude[0]; + units[Position.ALT] = (String) altitude[1]; + } + // EPSG + else if (m.getKey().equals(Key.POSITION_EPSG_CODE)) { + epsgCode = Integer.valueOf(m.getValue()); + } + } + + } + return new Position(posValues, units, epsgCode); + } + + private Object[] parseAlt(final String alt) throws ParseException { + LOG.trace(String.format("parseAlt(%s)", + alt)); + double value = Position.VALUE_NOT_SET; + String unit = Position.UNIT_NOT_SET; + + String number; + if (alt.contains("km")) { + unit = "km"; + number = alt.replace("km", ""); + } + else if (alt.contains("mi")) { + unit = "mi"; + number = alt.replace("mi", ""); + } + else if (alt.contains("m")) { + unit = "m"; + number = alt.replace("m", ""); + } + else if (alt.contains("ft")) { + unit = "ft"; + number = alt.replace("ft", ""); + } + else { + number = alt; + } + value = parseToDouble(number); + + return new Object[] {value, unit}; + } + + private Object[] parseLon(final String lon) throws ParseException { + LOG.trace(String.format("parseLon(%s)", + lon)); + double value; + String unit = ""; + + String number; + //TODO handle inputs like degrees/minutes/seconds, n.Br. + if (lon.contains("°")) { + unit = "°"; + final String[] part = lon.split("°"); + number = part[0]; + } else if (lon.contains("m")) { + unit = "m"; + number = lon.replace("m", ""); + } else { + number = lon; + } + value = parseToDouble(number); + + if (unit.equals("")) { + if (value <= 180.0 && value >= -180.0) { + unit = "deg"; + } + else { + unit = "m"; + } + } + + final Object[] result = {value, unit}; + return result; + } + + private Object[] parseLat(final String lat) throws ParseException { + LOG.trace(String.format("parseLat(%s)", + lat)); + double value; + String unit = ""; + + String number; + //TODO handle inputs like degrees/minutes/seconds, n.Br. + if (lat.contains("°")) { + unit = "°"; + final String[] part = lat.split("°"); + number = part[0]; + } else if (lat.contains("m")) { + unit = "m"; + number = lat.replace("m", ""); + } else { + number = lat; + } + value = parseToDouble(number); + + if (unit.equals("")) { + if (value <= 90.0 && value >= -90.0) { + unit = "deg"; + } + else { + unit = "m"; + } + } + + final Object[] result = {value, unit}; + return result; + } + + public double parseToDouble(final String number) throws ParseException{ + LOG.trace(String.format("parseToDouble(%s)", + number)); + final DecimalFormatSymbols symbols = new DecimalFormatSymbols(); + final char dSep = getDecimalSeparator(); + symbols.setDecimalSeparator(dSep); + symbols.setGroupingSeparator(getThousandsSeparator(dSep)); + + Number n; + final DecimalFormat formatter = new DecimalFormat(); + formatter.setDecimalFormatSymbols(symbols); + n = formatter.parse(number); + + return n.doubleValue(); + } + + private char getThousandsSeparator(final char dSep) { + if (dSep == '.') { + return ','; + } else if (dSep == ',') { + return '.'; + } else { + return 0; + } + } + + private char getDecimalSeparator() { + return importConf.getCsvMetadata().getDecimalSeparator().charAt(0); + } + + /** + * Returns all columns of the corresponding group + * @param group a {@link java.lang.String String} as group identifier + * @return a Column[] having all the group id + * group or
+ */ + public Column[] getAllColumnsForGroup(final String group, final Enum t) { + LOG.trace("getAllColumnsForGroup()"); + if (group == null) { return null; } + final Column[] allCols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); + final ArrayList tmpResultSet = new ArrayList(allCols.length); + for (final Column col : allCols) { + if (col.getType() != null && + col.getType().equals(t) ) { + // we have a position or dateTime + // check the Metadata kvps + if (col.getMetadataArray() != null && col.getMetadataArray().length > 0) { + findGroup: + for (final Metadata meta : col.getMetadataArray()) { + if (meta.getKey().equals(Key.GROUP) && + meta.getValue().equals(group)) { + tmpResultSet.add(col); + break findGroup; + } + } + } + } + } + tmpResultSet.trimToSize(); + Column[] result = new Column[tmpResultSet.size()]; + result = tmpResultSet.toArray(result); + return result; + } + + /** + * Returns the group id of the first date time group found in + * CsvMetadata.ColumnAssignments.Column[] + * @return a {@link java.lang.String String} + */ + public String getFirstDateTimeGroup() { + LOG.trace("getFirstDateTimeGroup()"); + final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); + for (final Column col : cols) { + if (col.getType().equals(Type.DATE_TIME)){ + // it's DATE_TIME -> get group id from metadata[] + if (col.getMetadataArray() != null && col.getMetadataArray().length > 0) { + for (final Metadata m : col.getMetadataArray()) { + if (m.getKey().equals(Key.GROUP)) { + if (LOG.isDebugEnabled()) { + LOG.debug(String.format("First date time group found: %s", + m.getValue())); + } + return m.getValue(); + } + } + } + } + } + LOG.error("No date time group found in configuration."); + return null; + } + + /** + * Returns the uom with the given id or null + * @param idRef + * @return UnitOfMeasurementType instance with + * id == idRef,
or null + */ + public UnitOfMeasurementType getUomById(final String idRef) { + LOG.trace(String.format("getUomById('%s')", + idRef)); + final UnitOfMeasurementType[] uoms = importConf.getAdditionalMetadata().getUnitOfMeasurementArray(); + for (final UnitOfMeasurementType uom : uoms) { + if (uom.getResource().getID().equals(idRef)) { + return uom; + } + } + return null; + } + + /** + * Checks all columns in CsvMetadata.ColumnAssignments.Column[] and returns + * the id of the first column with Type "UOM" + * @param mVColumnId + * @return the id of the first uom column or -1 if not found + */ + public int getColumnIdForUom(final int mVColumnId) { + LOG.trace(String.format("getColumnIdForUom(%s)", + mVColumnId)); + final Column[] cols = importConf.getCsvMetadata(). + getColumnAssignments().getColumnArray(); + for (final Column col : cols) { + if (col.getType().equals(Type.UOM)) { + return col.getNumber(); + } + } + return -1; + } + + /** + * Returns the op with the given id or null + * @param idRef + * @return + */ + public ObservedPropertyType getObsPropById(final String idRef) { + LOG.trace(String.format("getObsPropById('%s')", + idRef)); + final ObservedPropertyType[] ops = + importConf.getAdditionalMetadata().getObservedPropertyArray(); + for (final ObservedPropertyType op : ops) { + if (op.getResource().getID().equals(idRef)) { + return op; + } + } + return null; + } + + /** + * Checks all columns in CsvMetadata.ColumnAssignments.Column[] and returns + * the id of the first column with Type "OBSERVED_PROPERTY" + * @param mVColumnId + * @return the id of the first op column or -1 if not found + */ + public int getColumnIdForOpsProp(final int mVColumnId) { + LOG.trace(String.format("getColumnIdForOpsProp(%s)", + mVColumnId)); + final Column[] cols = importConf.getCsvMetadata(). + getColumnAssignments().getColumnArray(); + for (final Column col : cols) { + if (col.getType().equals(Type.OBSERVED_PROPERTY)) { + return col.getNumber(); + } + } + return -1; + } + + public Offering getOffering(final Sensor s) { + LOG.trace("getOffering()"); + if( importConf.getSosMetadata().getOffering().isSetGenerate() && + importConf.getSosMetadata().getOffering().getGenerate()) { + return new Offering(s.getName(), s.getUri()); + } else { + final String o = importConf.getSosMetadata().getOffering().getStringValue(); + return new Offering(o,o); + } + } + + public String getFileName() { + return configFile.getName(); + } + + @Override + public String toString() { + return String.format("Configuration [file=%s]", configFile); + } + + public String getType(final int mVColumnId) { + for (final Column col : importConf.getCsvMetadata().getColumnAssignments().getColumnArray()) { + if (col.getNumber() == mVColumnId) { + for (final Metadata m : col.getMetadataArray()) { + if (m.getKey().equals(Key.TYPE)) { + return m.getValue(); + } + } + } + } + return null; + } + + public SensorType getSensorFromAdditionalMetadata() { + LOG.trace("getSensorFromAdditionalMetadata()"); + if (importConf.getAdditionalMetadata() != null && + importConf.getAdditionalMetadata().getSensorArray() != null && + importConf.getAdditionalMetadata().getSensorArray().length == 1) { + return importConf.getAdditionalMetadata().getSensorArray(0); + } + return null; + } + + public boolean isOneMvColumn() { + return (getMeasureValueColumnIds().length == 1); + } + + public String getSosVersion() { + LOG.trace("getSosVersion()"); + return importConf.getSosMetadata().getVersion(); + } + + public String getSosBinding() { + LOG.trace("getSosBinding()"); + if (importConf.getSosMetadata().isSetBinding()) + { + return importConf.getSosMetadata().getBinding(); + } + LOG.info("Optional SosMetadata.Binding not set!"); + return null; + } + + public int getExpectedColumnCount() { + return importConf.getCsvMetadata().getColumnAssignments().sizeOfColumnArray(); + } + + public Pattern getLocaleFilePattern() { + return localeFilePattern; + } + + public String getRegExDateInfoInFileName() { + return importConf.getDataFile().getRegExDateInfoInFileName(); + } + + public String getDateInfoPattern() { + return importConf.getDataFile().getDateInfoPattern(); + } + + public boolean isDateInfoExtractionFromFileNameSetupValid() { + return importConf.getDataFile().isSetRegExDateInfoInFileName() && + importConf.getDataFile().isSetRegExDateInfoInFileName() && + !getRegExDateInfoInFileName().isEmpty() && + importConf.getDataFile().isSetDateInfoPattern() && + getRegExDateInfoInFileName().indexOf("(") >= 0 && + getRegExDateInfoInFileName().indexOf(")") > 1 && + !getDateInfoPattern().isEmpty(); + } + + public boolean isUseDateInfoFromFileModificationSet() { + return importConf.getDataFile().isSetUseDateFromLastModifiedDate() && + importConf.getDataFile().getUseDateFromLastModifiedDate(); + } + + /** + * @return true, if all required attributes are available for + * importing sample based files,
+ * else false. + */ + public boolean isSamplingFile() { + return importConf.getDataFile().isSetSampleStartRegEx() && + !importConf.getDataFile().getSampleStartRegEx().isEmpty() && + importConf.getDataFile().isSetSampleDateOffset() && + importConf.getDataFile().isSetSampleDateExtractionRegEx() && + !importConf.getDataFile().getSampleDateExtractionRegEx().isEmpty() && + importConf.getDataFile().getSampleDateExtractionRegEx().indexOf("(") >= 0 && + importConf.getDataFile().getSampleDateExtractionRegEx().indexOf(")") > 1 && + importConf.getDataFile().isSetSampleDatePattern() && + !importConf.getDataFile().getSampleDatePattern().isEmpty() && + importConf.getDataFile().isSetSampleDataOffset() && + importConf.getDataFile().isSetSampleSizeOffset() && + importConf.getDataFile().isSetSampleSizeRegEx() && + !importConf.getDataFile().getSampleSizeRegEx().isEmpty() && + importConf.getDataFile().getSampleSizeRegEx().indexOf("(") >= 0 && + importConf.getDataFile().getSampleSizeRegEx().indexOf(")") > 1; + } + + public String getSampleStartRegEx() { + if (importConf.getDataFile().isSetSampleStartRegEx() && + !importConf.getDataFile().getSampleStartRegEx().isEmpty()) { + return importConf.getDataFile().getSampleStartRegEx(); + } + throw new IllegalArgumentException("Attribute 'sampleIdRegEx' of not set."); } - public Position getPosition(final String group, final String[] values) throws ParseException { - LOG.trace(String.format("getPosition(group:%s,%s)", - group, - Arrays.toString(values))); - final Column[] cols = getAllColumnsForGroup(group, Type.POSITION); - // combine the values from the different columns - final String[] units = new String[3]; - final double[] posValues = new double[3]; - int epsgCode = -1; - for (final Column c : cols) { - // boolean isCombination = false; // now every position is of type combination - for (final Metadata m : c.getMetadataArray()) { - // check for type combination - // if (m.getKey().equals(Key.TYPE) && m.getValue().equals(Configuration.POSITION_TYPE_COMBINATION)) { - // isCombination = true; - // } - // get parse pattern and parse available values - /*else*/ if (m.getKey().equals(Key.PARSE_PATTERN)) { - String pattern = m.getValue(); - pattern = pattern.replaceAll(Configuration.POSITION_PARSEPATTERN_LATITUDE, "{0}"); - pattern = pattern.replaceAll(Configuration.POSITION_PARSEPATTERN_LONGITUDE, "{1}"); - pattern = pattern.replaceAll(Configuration.POSITION_PARSEPATTERN_ALTITUDE, "{2}"); - pattern = pattern.replaceAll(Configuration.POSITION_PARSEPATTERN_EPSG, "{3}"); - - final MessageFormat mf = new MessageFormat(pattern); - Object[] tokens = null; - try { - tokens = mf.parse(values[c.getNumber()]); - } catch (final ParseException e) { - throw new NumberFormatException(); - } - - if (tokens == null) { - throw new NumberFormatException(); - } - - Object[] latitude, longitude, height; - - if (tokens.length > 0 && tokens[0] != null) { - latitude = parseLat((String) tokens[0]); - posValues[Position.LAT] = (Double) latitude[0]; - units[Position.LAT] = (String) latitude[1]; - } - if (tokens.length > 1 && tokens[1] != null) { - longitude = parseLon((String)tokens[1]); - posValues[Position.LONG] = (Double) longitude[0]; - units[Position.LONG] = (String) longitude[1]; - } - if (tokens.length > 2 && tokens[2] != null) { - height = parseAlt((String)tokens[2]); - posValues[Position.ALT] = (Double) height[0]; - units[Position.ALT] = (String) height[1]; - } - if (tokens.length > 3 && tokens[3] != null) { - epsgCode = Integer.valueOf((String)tokens[3]); - } - } - // get additional information - // LATITUDE - else if (m.getKey().equals(Key.POSITION_LATITUDE)) { - final Object[] latitude = parseLat(m.getValue()); - posValues[Position.LAT] = (Double) latitude[0]; - units[Position.LAT] = (String) latitude[1]; - } - // LONGITUDE - else if (m.getKey().equals(Key.POSITION_LONGITUDE)) { - final Object[] longitude = parseLon(m.getValue()); - posValues[Position.LONG] = (Double) longitude[0]; - units[Position.LONG] = (String) longitude[1]; - } - // ALTITUDE - else if (m.getKey().equals(Key.POSITION_ALTITUDE)) { - final Object[] altitude = parseAlt(m.getValue()); - posValues[Position.ALT] = (Double) altitude[0]; - units[Position.ALT] = (String) altitude[1]; - } - // EPSG - else if (m.getKey().equals(Key.POSITION_EPSG_CODE)) { - epsgCode = Integer.valueOf(m.getValue()); - } - } - - } - return new Position(posValues, units, epsgCode); - } - - private Object[] parseAlt(final String alt) throws ParseException { - LOG.trace(String.format("parseAlt(%s)", - alt)); - double value = Position.VALUE_NOT_SET; - String unit = Position.UNIT_NOT_SET; - - String number; - if (alt.contains("km")) { - unit = "km"; - number = alt.replace("km", ""); - } - else if (alt.contains("mi")) { - unit = "mi"; - number = alt.replace("mi", ""); - } - else if (alt.contains("m")) { - unit = "m"; - number = alt.replace("m", ""); - } - else if (alt.contains("ft")) { - unit = "ft"; - number = alt.replace("ft", ""); - } - else { - number = alt; - } - value = parseToDouble(number); - - return new Object[] {value, unit}; - } - - private Object[] parseLon(final String lon) throws ParseException { - LOG.trace(String.format("parseLon(%s)", - lon)); - double value; - String unit = ""; - - String number; - //TODO handle inputs like degrees/minutes/seconds, n.Br. - if (lon.contains("°")) { - unit = "°"; - final String[] part = lon.split("°"); - number = part[0]; - } else if (lon.contains("m")) { - unit = "m"; - number = lon.replace("m", ""); - } else { - number = lon; - } - value = parseToDouble(number); - - if (unit.equals("")) { - if (value <= 180.0 && value >= -180.0) { - unit = "deg"; - } - else { - unit = "m"; - } + public String getSampleSizeRegEx() { + if (importConf.getDataFile().isSetSampleSizeRegEx() && + !importConf.getDataFile().getSampleSizeRegEx().isEmpty() && + importConf.getDataFile().getSampleSizeRegEx().indexOf("(") >= 0 && + importConf.getDataFile().getSampleSizeRegEx().indexOf(")") > 1) { + return importConf.getDataFile().getSampleSizeRegEx(); } - - final Object[] result = {value, unit}; - return result; + throw new IllegalArgumentException("Attribute 'sampleSizeRegEx' of not set."); } - private Object[] parseLat(final String lat) throws ParseException { - LOG.trace(String.format("parseLat(%s)", - lat)); - double value; - String unit = ""; - - String number; - //TODO handle inputs like degrees/minutes/seconds, n.Br. - if (lat.contains("°")) { - unit = "°"; - final String[] part = lat.split("°"); - number = part[0]; - } else if (lat.contains("m")) { - unit = "m"; - number = lat.replace("m", ""); - } else { - number = lat; + public int getSampleSizeOffset() { + if (importConf.getDataFile().isSetSampleSizeOffset()) { + return importConf.getDataFile().getSampleSizeOffset(); } - value = parseToDouble(number); - - if (unit.equals("")) { - if (value <= 90.0 && value >= -90.0) { - unit = "deg"; - } - else { - unit = "m"; - } - } - - final Object[] result = {value, unit}; - return result; + throw new IllegalArgumentException("Attribute 'sampleSizeOffset' of not set."); } - public double parseToDouble(final String number) throws ParseException{ - LOG.trace(String.format("parseToDouble(%s)", - number)); - final DecimalFormatSymbols symbols = new DecimalFormatSymbols(); - final char dSep = getDecimalSeparator(); - symbols.setDecimalSeparator(dSep); - symbols.setGroupingSeparator(getThousandsSeparator(dSep)); - - Number n; - final DecimalFormat formatter = new DecimalFormat(); - formatter.setDecimalFormatSymbols(symbols); - n = formatter.parse(number); - - return n.doubleValue(); - } - - private char getThousandsSeparator(final char dSep) { - if (dSep == '.') { - return ','; - } else if (dSep == ',') { - return '.'; - } else { - return 0; + public int getSampleDateOffset() { + if (importConf.getDataFile().isSetSampleDateOffset()) { + return importConf.getDataFile().getSampleDateOffset(); } + throw new IllegalArgumentException("Attribute 'sampleDateOffset' of not set."); } - private char getDecimalSeparator() { - return importConf.getCsvMetadata().getDecimalSeparator().charAt(0); - } - - /** - * Returns all columns of the corresponding group - * @param group a {@link java.lang.String String} as group identifier - * @return a Column[] having all the group id - * group or
- */ - public Column[] getAllColumnsForGroup(final String group, final Enum t) { - LOG.trace("getAllColumnsForGroup()"); - if (group == null) { return null; } - final Column[] allCols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); - final ArrayList tmpResultSet = new ArrayList(allCols.length); - for (final Column col : allCols) { - if (col.getType() != null && - col.getType().equals(t) ) { - // we have a position or dateTime - // check the Metadata kvps - if (col.getMetadataArray() != null && col.getMetadataArray().length > 0) { - findGroup: - for (final Metadata meta : col.getMetadataArray()) { - if (meta.getKey().equals(Key.GROUP) && - meta.getValue().equals(group)) { - tmpResultSet.add(col); - break findGroup; - } - } - } - } + public int getSampleDataOffset() { + if (importConf.getDataFile().isSetSampleDataOffset()) { + return importConf.getDataFile().getSampleDataOffset(); } - tmpResultSet.trimToSize(); - Column[] result = new Column[tmpResultSet.size()]; - result = tmpResultSet.toArray(result); - return result; + throw new IllegalArgumentException("Attribute 'sampleDataOffset' of not set."); } - /** - * Returns the group id of the first date time group found in - * CsvMetadata.ColumnAssignments.Column[] - * @return a {@link java.lang.String String} - */ - public String getFirstDateTimeGroup() { - LOG.trace("getFirstDateTimeGroup()"); - final Column[] cols = importConf.getCsvMetadata().getColumnAssignments().getColumnArray(); - for (final Column col : cols) { - if (col.getType().equals(Type.DATE_TIME)){ - // it's DATE_TIME -> get group id from metadata[] - if (col.getMetadataArray() != null && col.getMetadataArray().length > 0) { - for (final Metadata m : col.getMetadataArray()) { - if (m.getKey().equals(Key.GROUP)) { - if (LOG.isDebugEnabled()) { - LOG.debug(String.format("First date time group found: %s", - m.getValue())); - } - return m.getValue(); - } - } - } - } - } - LOG.error("No date time group found in configuration."); - return null; - } - /** - * Returns the uom with the given id or null - * @param idRef - * @return UnitOfMeasurementType instance with - * id == idRef,
or null - */ - public UnitOfMeasurementType getUomById(final String idRef) { - LOG.trace(String.format("getUomById('%s')", - idRef)); - final UnitOfMeasurementType[] uoms = importConf.getAdditionalMetadata().getUnitOfMeasurementArray(); - for (final UnitOfMeasurementType uom : uoms) { - if (uom.getResource().getID().equals(idRef)) { - return uom; - } + public String getSampleDatePattern() { + if (importConf.getDataFile().isSetSampleDatePattern() && + !importConf.getDataFile().getSampleDatePattern().isEmpty()) { + return importConf.getDataFile().getSampleDatePattern(); } - return null; + throw new IllegalArgumentException("Attribute 'sampleDateInfoPattern' of not set."); } - /** - * Checks all columns in CsvMetadata.ColumnAssignments.Column[] and returns - * the id of the first column with Type "UOM" - * @param mVColumnId - * @return the id of the first uom column or -1 if not found - */ - public int getColumnIdForUom(final int mVColumnId) { - LOG.trace(String.format("getColumnIdForUom(%s)", - mVColumnId)); - final Column[] cols = importConf.getCsvMetadata(). - getColumnAssignments().getColumnArray(); - for (final Column col : cols) { - if (col.getType().equals(Type.UOM)) { - return col.getNumber(); - } + public String getSampleDateExtractionRegEx() { + if (importConf.getDataFile().isSetSampleDateExtractionRegEx() && + !importConf.getDataFile().getSampleDateExtractionRegEx().isEmpty() && + importConf.getDataFile().getSampleDateExtractionRegEx().indexOf("(") >= 0 && + importConf.getDataFile().getSampleDateExtractionRegEx().indexOf(")") > 1) { + return importConf.getDataFile().getSampleDateExtractionRegEx(); } - return -1; + throw new IllegalArgumentException("Attribute 'sampleDateExtractionRegEx' of not set."); } - /** - * Returns the op with the given id or null - * @param idRef - * @return - */ - public ObservedPropertyType getObsPropById(final String idRef) { - LOG.trace(String.format("getObsPropById('%s')", - idRef)); - final ObservedPropertyType[] ops = - importConf.getAdditionalMetadata().getObservedPropertyArray(); - for (final ObservedPropertyType op : ops) { - if (op.getResource().getID().equals(idRef)) { - return op; - } - } - return null; - } - /** - * Checks all columns in CsvMetadata.ColumnAssignments.Column[] and returns - * the id of the first column with Type "OBSERVED_PROPERTY" - * @param mVColumnId - * @return the id of the first op column or -1 if not found - */ - public int getColumnIdForOpsProp(final int mVColumnId) { - LOG.trace(String.format("getColumnIdForOpsProp(%s)", - mVColumnId)); - final Column[] cols = importConf.getCsvMetadata(). - getColumnAssignments().getColumnArray(); - for (final Column col : cols) { - if (col.getType().equals(Type.OBSERVED_PROPERTY)) { - return col.getNumber(); - } - } - return -1; + /** + * @return The configured value > 0, if it is set
+ * else -1. + */ + public int getLastModifiedDelta() { + if (importConf.getDataFile().isSetLastModifiedDelta()) { + return importConf.getDataFile().getLastModifiedDelta(); + } + return -1; + } + + /** + * @return number of the line that contains the header information for the + * first time, or
+ * -1 if the optional attribute "headerLine" is not + * set in DataFile element + */ + public int getHeaderLine() { + if (importConf.getDataFile().isSetHeaderLine()) { + return importConf.getDataFile().getHeaderLine().intValue(); + } + return -1; + } + + /** + * @return Name of the data file encoding, or
+ * if not set, "UTF-8" + */ + public String getDataFileEncoding() { + if (importConf.getDataFile().isSetLocalFile() && + importConf.getDataFile().getLocalFile().isSetEncoding() && + !importConf.getDataFile().getLocalFile().getEncoding().isEmpty()) { + return importConf.getDataFile().getLocalFile().getEncoding(); + } + LOG.debug("Using default encoding 'UTF-8'"); + return "UTF-8"; + } + + /** + * @return The {@link ImportStrategy} that could be configured in + * SosImportConfiguration/AdditionalMetadata/Metadata/Key=IMPORT_STRATEGY/Value=The_import_strategy_to_use + *
Default if nothing matching is found: {@link ImportStrategy#SingleObservation} + */ + public ImportStrategy getImportStrategy() { + if (importConf.isSetAdditionalMetadata() && importConf.getAdditionalMetadata().getMetadataArray().length > 0) { + for (int i = 0; i < importConf.getAdditionalMetadata().getMetadataArray().length; i++) { + final Metadata metadata = importConf.getAdditionalMetadata().getMetadataArray(i); + if (metadata.getKey().equals(Key.IMPORT_STRATEGY)) { + if (metadata.getValue().equalsIgnoreCase(ImportStrategy.SweArrayObservationWithSplitExtension.name())) { + return ImportStrategy.SweArrayObservationWithSplitExtension; + } else { + return ImportStrategy.SingleObservation; + } + } + } + } + return ImportStrategy.SingleObservation; + } + + public int getHunkSize() { + if (importConf.isSetAdditionalMetadata() && importConf.getAdditionalMetadata().getMetadataArray().length > 0) { + for (int i = 0; i < importConf.getAdditionalMetadata().getMetadataArray().length; i++) { + final Metadata metadata = importConf.getAdditionalMetadata().getMetadataArray(i); + if (metadata.getKey().equals(Key.HUNK_SIZE)) { + try { + return Integer.parseInt(metadata.getValue()); + } catch (final NumberFormatException nfe) { + LOG.error( + String.format("Value of metadata element with key '%s' could not be parsed to int: '%s'. Ignoring it.", + Key.HUNK_SIZE.toString()), + nfe); + } + } + } + } + return -1; + } + + + public boolean isIgnoreLineRegExSet() { + return importConf.getDataFile().getIgnoreLineRegExArray() != null && + importConf.getDataFile().getIgnoreLineRegExArray().length > 0; } - public Offering getOffering(final Sensor s) { - LOG.trace("getOffering()"); - if( importConf.getSosMetadata().getOffering().isSetGenerate() && - importConf.getSosMetadata().getOffering().getGenerate()) { - return new Offering(s.getName(), s.getUri()); - } else { - final String o = importConf.getSosMetadata().getOffering().getStringValue(); - return new Offering(o,o); + public Pattern[] getIgnoreLineRegExPatterns() { + if (!isIgnoreLineRegExSet()) { + return new Pattern[0]; } - } - - public String getFileName() { - return configFile.getName(); - } - - @Override - public String toString() { - return String.format("Configuration [file=%s]", configFile); - } - - public String getType(final int mVColumnId) { - for (final Column col : importConf.getCsvMetadata().getColumnAssignments().getColumnArray()) { - if (col.getNumber() == mVColumnId) { - for (final Metadata m : col.getMetadataArray()) { - if (m.getKey().equals(Key.TYPE)) { - return m.getValue(); - } - } + final String[] ignoreLineRegExArray = importConf.getDataFile().getIgnoreLineRegExArray(); + final LinkedList patterns = new LinkedList<>(); + for (final String regEx : ignoreLineRegExArray) { + if (regEx != null && !regEx.isEmpty()) { + patterns.add(Pattern.compile(regEx)); } } - return null; - } - - public SensorType getSensorFromAdditionalMetadata() { - LOG.trace("getSensorFromAdditionalMetadata()"); - if (importConf.getAdditionalMetadata() != null && - importConf.getAdditionalMetadata().getSensorArray() != null && - importConf.getAdditionalMetadata().getSensorArray().length == 1) { - return importConf.getAdditionalMetadata().getSensorArray(0); - } - return null; - } - - public boolean isOneMvColumn() { - return (getMeasureValueColumnIds().length == 1); - } - - public String getSosVersion() { - LOG.trace("getSosVersion()"); - return importConf.getSosMetadata().getVersion(); - } - - public String getSosBinding() { - LOG.trace("getSosBinding()"); - if (importConf.getSosMetadata().isSetBinding()) - { - return importConf.getSosMetadata().getBinding(); - } - LOG.info("Optional SosMetadata.Binding not set!"); - return null; - } - - public int getExpectedColumnCount() { - return importConf.getCsvMetadata().getColumnAssignments().sizeOfColumnArray(); - } - - public Pattern getLocaleFilePattern() { - return localeFilePattern; - } - - public String getRegExDateInfoInFileName() { - return importConf.getDataFile().getRegExDateInfoInFileName(); - } - - public String getDateInfoPattern() { - return importConf.getDataFile().getDateInfoPattern(); - } - - public boolean isDateInfoExtractionSetupValid() { - return importConf.getDataFile().isSetRegExDateInfoInFileName() && - importConf.getDataFile().isSetRegExDateInfoInFileName() && - !getRegExDateInfoInFileName().isEmpty() && - importConf.getDataFile().isSetDateInfoPattern() && - getRegExDateInfoInFileName().indexOf("(") >= 0 && - getRegExDateInfoInFileName().indexOf(")") > 1 && - !getDateInfoPattern().isEmpty(); - } - - public boolean isUseDateInfoFromFileModificationSet() { - return importConf.getDataFile().isSetUseDateFromLastModifiedDate() && - importConf.getDataFile().getUseDateFromLastModifiedDate(); - } - - /** - * @return The configured value > 0, if it is set
- * else -1. - */ - public int getLastModifiedDelta() { - if (importConf.getDataFile().isSetLastModifiedDelta()) { - return importConf.getDataFile().getLastModifiedDelta(); - } - return -1; + return patterns.toArray(new Pattern[patterns.size()]); } - /** - * @return number of the line that contains the header information for the - * first time, or
- * -1 if the optional attribute "headerLine" is not - * set in DataFile element - */ - public int getHeaderLine() { - if (importConf.getDataFile().isSetHeaderLine()) { - return importConf.getDataFile().getHeaderLine().intValue(); - } - return -1; + public boolean isInsertSweArrayObservationTimeoutBufferSet() { + return importConf.getSosMetadata().isSetInsertSweArrayObservationTimeoutBuffer(); } - /** - * @return Name of the data file encoding, or
- * if not set, "UTF-8" - */ - public String getDataFileEncoding() { - if (importConf.getDataFile().isSetLocalFile() && - importConf.getDataFile().getLocalFile().isSetEncoding() && - !importConf.getDataFile().getLocalFile().getEncoding().isEmpty()) { - return importConf.getDataFile().getLocalFile().getEncoding(); + public int getInsertSweArrayObservationTimeoutBuffer() { + if (isInsertSweArrayObservationTimeoutBufferSet()) { + return importConf.getSosMetadata().getInsertSweArrayObservationTimeoutBuffer(); } - LOG.debug("Using default encoding 'UTF-8'"); - return "UTF-8"; + throw new IllegalArgumentException("Attribute 'insertSweArrayObservationTimeoutBuffer' of not set."); } - /** - * @return The {@link ImportStrategy} that could be configured in - * SosImportConfiguration/AdditionalMetadata/Metadata/Key=IMPORT_STRATEGY/Value=The_import_strategy_to_use - *
Default if nothing matching is found: {@link ImportStrategy#SingleObservation} - */ - public ImportStrategy getImportStrategy() { - if (importConf.isSetAdditionalMetadata() && importConf.getAdditionalMetadata().getMetadataArray().length > 0) { - for (int i = 0; i < importConf.getAdditionalMetadata().getMetadataArray().length; i++) { - final Metadata metadata = importConf.getAdditionalMetadata().getMetadataArray(i); - if (metadata.getKey().equals(Key.IMPORT_STRATEGY)) { - if (metadata.getValue().equalsIgnoreCase(ImportStrategy.SweArrayObservationWithSplitExtension.name())) { - return ImportStrategy.SweArrayObservationWithSplitExtension; - } else { - return ImportStrategy.SingleObservation; - } - } - } - } - return ImportStrategy.SingleObservation; - } - public int getHunkSize() { - if (importConf.isSetAdditionalMetadata() && importConf.getAdditionalMetadata().getMetadataArray().length > 0) { - for (int i = 0; i < importConf.getAdditionalMetadata().getMetadataArray().length; i++) { - final Metadata metadata = importConf.getAdditionalMetadata().getMetadataArray(i); - if (metadata.getKey().equals(Key.HUNK_SIZE)) { - try { - return Integer.parseInt(metadata.getValue()); - } catch (final NumberFormatException nfe) { - LOG.error( - String.format("Value of metadata element with key '%s' could not be parsed to int: '%s'. Ignoring it.", - Key.HUNK_SIZE.toString()), - nfe); - } - } - } - } - return -1; - } } diff --git a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/DataFile.java b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/DataFile.java index 36df1228..2768db1c 100644 --- a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/DataFile.java +++ b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/DataFile.java @@ -409,14 +409,14 @@ public Timestamp getTimeStamp(final int mVColumn, final String[] values) throws } enrichTimestampWithColumnMetadata(ts,column); } - if (configuration.isDateInfoExtractionSetupValid()) { - ts.enrichByFilename( + if (configuration.isDateInfoExtractionFromFileNameSetupValid()) { + ts.enrich( file.getName(), configuration.getRegExDateInfoInFileName(), configuration.getDateInfoPattern()); } if (configuration.isUseDateInfoFromFileModificationSet()) { - ts.enrichByFileModificationDate(file.lastModified(), configuration.getLastModifiedDelta()); + ts.enrich(file.lastModified(), configuration.getLastModifiedDelta()); } return ts; } diff --git a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/SensorObservationService.java b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/SensorObservationService.java index 5de72132..adb67748 100644 --- a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/SensorObservationService.java +++ b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/SensorObservationService.java @@ -32,6 +32,7 @@ import java.io.IOException; import java.io.InputStreamReader; import java.io.UnsupportedEncodingException; +import java.net.MalformedURLException; import java.net.URL; import java.text.ParseException; import java.util.ArrayList; @@ -46,6 +47,8 @@ import java.util.List; import java.util.Map; import java.util.Set; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import net.opengis.sos.x10.InsertObservationResponseDocument; import net.opengis.sos.x10.InsertObservationResponseDocument.InsertObservationResponse; @@ -120,23 +123,63 @@ public final class SensorObservationService { private String[] headerLine; - private final ImportStrategy importStrategy; + private boolean isSampleBasedDataFile = false; // Identified on localhost on development system // Default value: 5000 // Max possible value: 12500 private int hunkSize = 5000; - public SensorObservationService(final URL sosUrl, - final String version, - final String binding, - final ImportStrategy importStrategy, - final int hunkSize) throws ExceptionReport, OXFException { - LOG.trace(String.format("SensorObservationService(%s)", sosUrl)); - this.sosUrl = sosUrl; - sosVersion = version; - sosBinding = getBinding(binding); - this.importStrategy = importStrategy; + private final Configuration config; + + // stores the Timestamp of the last insertObservations + // (required for handling sample based files) + private Timestamp lastTimestamp = null; + + // The date information of the current sample + private Timestamp sampleDate; + + private Pattern sampleIdPattern; + + private Pattern sampleSizePattern; + + private boolean isInSample; + + private int sampleSize; + + private int sampleSizeOffset; + + private int sampleDateOffset; + + private int sampleOffsetDifference; + + private int sampleDataOffset; + + private int lineCounter; + + private final int[] ignoredColumns; + + private Pattern[] ignorePatterns; + + // adding 25s + private int sweArrayObservationTimeOutBuffer = 25000; + + public SensorObservationService(final Configuration config) throws ExceptionReport, OXFException, MalformedURLException { + LOG.trace(String.format("SensorObservationService(%s)", config.toString())); + this.config = config; + sosUrl = config.getSosUrl(); + sosVersion = config.getSosVersion(); + sosBinding = getBinding(config.getSosBinding()); + isSampleBasedDataFile = config.isSamplingFile(); + ignoredColumns = config.getIgnoredColumnIds(); + if (isSampleBasedDataFile) { + sampleIdPattern = Pattern.compile(config.getSampleStartRegEx()); + sampleSizePattern = Pattern.compile(config.getSampleSizeRegEx()); + sampleSizeOffset = config.getSampleSizeOffset(); + sampleDateOffset = config.getSampleDateOffset(); + sampleOffsetDifference = Math.abs(sampleDateOffset - sampleSizeOffset); + sampleDataOffset = config.getSampleDataOffset(); + } if (sosBinding == null) { sosWrapper = SosWrapperFactory.newInstance(sosUrl.toString(),sosVersion); } else { @@ -153,8 +196,19 @@ public SensorObservationService(final URL sosUrl, if (sosVersion.equals("2.0.0")) { offerings = new HashMap(); } - if (hunkSize > 0) { - this.hunkSize = hunkSize; + if (config.getHunkSize() > 0) { + hunkSize = config.getHunkSize(); + } + if (config.isIgnoreLineRegExSet()) { + ignorePatterns = config.getIgnoreLineRegExPatterns(); + } + if (config.isInsertSweArrayObservationTimeoutBufferSet()) { + sweArrayObservationTimeOutBuffer = config.getInsertSweArrayObservationTimeoutBuffer(); + } + if (isSampleBasedDataFile && config.getImportStrategy().equals(ImportStrategy.SweArrayObservationWithSplitExtension)) { + LOG.info("Using {}ms timeout buffer during insert observation requests. " + + "Change if required.", + sweArrayObservationTimeOutBuffer); } } @@ -193,8 +247,7 @@ public boolean isTransactional() { } final OperationsMetadata opMeta = serviceDescriptor.getOperationsMetadata(); LOG.debug(String.format("OperationsMetadata found: %s", opMeta)); - // check for RegisterSensor and InsertObservationOperation - // TODO implement version specific + // check for (Insert|Register)Sensor and InsertObservationOperation if ((opMeta.getOperationByName(SOSAdapter.REGISTER_SENSOR) != null || opMeta.getOperationByName(SOSAdapter.INSERT_SENSOR) != null) && @@ -208,12 +261,12 @@ public boolean isTransactional() { return false; } - public List importData(final DataFile dataFile) throws IOException, OXFException, XmlException, IllegalArgumentException { + public List importData(final DataFile dataFile) throws IOException, OXFException, XmlException, IllegalArgumentException, ParseException { LOG.trace("importData()"); // 0 Get line final CSVReader cr = dataFile.getCSVReader(); String[] values; - int lineCounter = dataFile.getFirstLineWithData(); + lineCounter = dataFile.getFirstLineWithData(); if (dataFile.getHeaderLine() > -1 && headerLine == null) { headerLine = readHeaderLine(dataFile); } @@ -225,23 +278,25 @@ public List importData(final DataFile dataFile) throws IOExce LOG.error("No measured value columns found in configuration"); return null; } - skipAlreadyReadLines(cr, lineCounter); - switch (importStrategy) { + skipLines(cr, lastLine); + switch (config.getImportStrategy()) { case SingleObservation: long startReadingFile = System.currentTimeMillis(); // for each line while ((values = cr.readNext()) != null) { - if (isNotEmpty(values) && isSizeValid(dataFile, values) && !isHeaderLine(values)) { + if (!isLineIgnorable(values) && isNotEmpty(values) && isSizeValid(dataFile, values) && !isHeaderLine(values)) { LOG.debug(String.format("Handling CSV line #%d: %s",lineCounter+1,Arrays.toString(values))); - final InsertObservation[] ios = getInsertObservations(values,mVCols,dataFile,lineCounter); + final InsertObservation[] ios = getInsertObservations(values,mVCols,dataFile); numOfObsTriedToInsert += ios.length; insertObservationsForOneLine(ios,values,dataFile); LOG.debug(Feeder.heapSizeInformation()); } else { LOG.trace(String.format("\t\tSkip CSV line #%d: %s",(lineCounter+1),Arrays.toString(values))); } - lastLine++; lineCounter++; + if (lineCounter % 10000 == 0) { + LOG.info("Processed line {}.",lineCounter); + } } long finishedImportData = System.currentTimeMillis(); LOG.debug("Timing:\nStart File: {}\nFinished importing: {}", @@ -254,10 +309,20 @@ public List importData(final DataFile dataFile) throws IOExce startReadingFile = System.currentTimeMillis(); TimeSeriesRepository timeSeriesRepository = new TimeSeriesRepository(mVCols.length); int currentHunk = 0; + int sampleStartLine = lineCounter; while ((values = cr.readNext()) != null) { - if (isNotEmpty(values) && isSizeValid(dataFile, values) && !isHeaderLine(values)) { + // if it is a sample based file, I need to get the following information + // * date information (depends on last timestamp because of + if (isSampleBasedDataFile && !isInSample && isSampleStart(values)) { + sampleStartLine = lineCounter; + getSampleMetaData(cr); + isInSample = true; + skipLines(cr, sampleDataOffset-1-(lineCounter-sampleStartLine)); + continue; + } + if (!isLineIgnorable(values) && isNotEmpty(values) && isSizeValid(dataFile, values) && !isHeaderLine(values)) { LOG.debug(String.format("Handling CSV line #%d: %s",lineCounter+1,Arrays.toString(values))); - final InsertObservation[] ios = getInsertObservations(values,mVCols,dataFile,lineCounter); + final InsertObservation[] ios = getInsertObservations(values,mVCols,dataFile); timeSeriesRepository.addObservations(ios); numOfObsTriedToInsert += ios.length; LOG.debug(Feeder.heapSizeInformation()); @@ -271,17 +336,31 @@ public List importData(final DataFile dataFile) throws IOExce } else { LOG.trace(String.format("\t\tSkip CSV line #%d: %s",(lineCounter+1),Arrays.toString(values))); } - lastLine++; lineCounter++; + if (lineCounter % 10000 == 0) { + LOG.info("Processed line {}.",lineCounter); + } + if (isSampleBasedDataFile) { + LOG.debug("SampleFile: {}; isInSample: {}; lineCounter: {}; sampleStartLine: {}; sampleSize: {}; sampleDataOffset: {}", + isSampleBasedDataFile, + isInSample, + lineCounter, + sampleStartLine, + sampleSize, + sampleDataOffset); + } + if (isSampleBasedDataFile && isInSample && isSampleEndReached(sampleStartLine)) { + isInSample = false; + LOG.debug("Current sample left"); + } } if (!timeSeriesRepository.isEmpty()) { insertTimeSeries(timeSeriesRepository); } - final long finishedReadingFile = System.currentTimeMillis(); + lastLine = lineCounter+1; finishedImportData = System.currentTimeMillis(); - LOG.debug("Timing:\nStart File: {}\nFinished File/Start importing: {}\nFinished importing: {}", + LOG.debug("Timing:\nStart File: {}\nFinished importing: {}", new Date(startReadingFile).toString(), - new Date(finishedReadingFile).toString(), new Date(finishedImportData).toString()); } @@ -291,14 +370,83 @@ public List importData(final DataFile dataFile) throws IOExce return failedInsertObservations; } - private void skipAlreadyReadLines(final CSVReader cr, - int lineCounter) throws IOException { + private boolean isLineIgnorable(final String[] values) { + if (ignorePatterns == null || ignorePatterns.length > 0) { + final String line = restoreLine(values); + for (final Pattern pattern : ignorePatterns) { + if (pattern.matcher(line).matches()) { + return true; + } + } + } + return false; + } + + private void getSampleMetaData(final CSVReader cr) throws IOException, ParseException { + if (sampleDateOffset < sampleSizeOffset) { + skipLines(cr,sampleDateOffset-1); + sampleDate = parseSampleDate(cr.readNext()); + lineCounter++; + skipLines(cr,sampleOffsetDifference-1); + sampleSize = parseSampleSize(cr.readNext()); + lineCounter++; + } else { + skipLines(cr,sampleSizeOffset-1); + sampleSize = parseSampleSize(cr.readNext()); + lineCounter++; + skipLines(cr,sampleOffsetDifference-1); + sampleDate = parseSampleDate(cr.readNext()); + lineCounter++; + } + + } + + private int parseSampleSize(final String[] values) throws ParseException { + final String lineToParse = restoreLine(values); + final Matcher matcher = sampleSizePattern.matcher(lineToParse); + if (matcher.matches() && matcher.groupCount() == 1) { + final String dateInformation = matcher.group(1); + return Integer.parseInt(dateInformation); + // TODO handle NumberformatException + } + throw new ParseException(String.format("Could not extract sampleSize from '%s' using regular expression '%s' (Offset is always 42).", + lineToParse, + sampleSizePattern.pattern()),42); + } + + private String restoreLine(final String[] values) { + final StringBuffer sb = new StringBuffer(); + for (int i = 0; i < values.length; i++) { + sb.append(values[i]); + if (i != values.length-1) { + sb.append(config.getCsvSeparator()); + } + } + return sb.toString(); + } + + private Timestamp parseSampleDate(final String[] values) throws ParseException { + final String dateInfoPattern = config.getSampleDatePattern(); + final String regExToExtractDateInfo = config.getSampleDateExtractionRegEx(); + final String timestampInformation = restoreLine(values); + return new Timestamp().enrich(timestampInformation, regExToExtractDateInfo, dateInfoPattern); + } + + public boolean isSampleEndReached(final int sampleStartLine) { + return sampleStartLine + sampleSize + sampleDataOffset - 1 == lineCounter; + } + + private boolean isSampleStart(final String[] values) { + return sampleIdPattern.matcher(restoreLine(values)).matches(); + } + + private void skipLines(final CSVReader cr, + int skipCount) throws IOException { // get the number of lines to skip (coming from already read lines) String[] values; - int skipCount = lastLine; while (skipCount > 0) { values = cr.readNext(); - LOG.trace(String.format("\t\tSkip CSV line #%d: %s",(lineCounter+1),Arrays.toString(values))); + LOG.trace(String.format("\t\tSkip CSV line #%d: %s",(lineCounter+1),restoreLine(values))); skipCount--; lineCounter++; } @@ -337,12 +485,12 @@ private boolean isSizeValid(final DataFile dataFile, return true; } - private boolean isNotEmpty(final String[] values) - { + private boolean isNotEmpty(final String[] values) { if (values != null && values.length > 0) { - for (final String value : values) { - if (value == null || value.isEmpty()) { - LOG.error("Current line '{}' contains empty values . Skipping this line!", Arrays.toString(values)); + for (int i = 0; i < values.length; i++) { + final String value = values[i]; + if (!isColumnIgnored(i) && (value == null || value.isEmpty())) { + LOG.debug("Current line '{}' contains empty values . Skipping this line!", Arrays.toString(values)); return false; } } @@ -351,10 +499,18 @@ private boolean isNotEmpty(final String[] values) return false; } + private boolean isColumnIgnored(final int i) { + for (final int ignoredColumn : ignoredColumns) { + if (i == ignoredColumn) { + return true; + } + } + return false; + } + private InsertObservation[] getInsertObservations(final String[] values, final int[] mVColumns, - final DataFile df, - final int currentLine){ + final DataFile df){ LOG.trace("getInsertObservations()"); if (mVColumns == null || mVColumns.length == 0) { LOG.error("Method called with bad arguments: values: {}, mVColumns: {}", Arrays.toString(values), Arrays.toString(mVColumns)); @@ -394,6 +550,13 @@ private InsertObservation getInsertObservationForColumnIdFromValues(final int mV // TODO implement using different templates in later version depending on the class of value // TIMESTAMP final Timestamp timeStamp = dataFile.getTimeStamp(mVColumnId,values); + if (isSampleBasedDataFile) { + if (lastTimestamp != null && timeStamp.before(lastTimestamp)) { + sampleDate.applyDayDelta(1); + } + lastTimestamp = new Timestamp().enrich(timeStamp); + timeStamp.enrich(sampleDate); + } LOG.debug("Timestamp: {}", timeStamp); // UOM CODE final UnitOfMeasurement uom = dataFile.getUnitOfMeasurement(mVColumnId,values); @@ -546,7 +709,13 @@ private String insertSweArrayObservation(final org.n52.oxf.sos.request.InsertObs try { try { + final int connectionTimeout = sosWrapper.getConnectionTimeout(); + final int readTimeout = sosWrapper.getReadTimeout(); + sosWrapper.setConnectionTimeOut(connectionTimeout + sweArrayObservationTimeOutBuffer); + sosWrapper.setReadTimeout(readTimeout + sweArrayObservationTimeOutBuffer); opResult = sosWrapper.doInsertObservation(sweArrayObservation); + sosWrapper.setConnectionTimeOut(connectionTimeout); + sosWrapper.setReadTimeout(readTimeout); if (sosVersion.equals("1.0.0")) { try { final InsertObservationResponse response = InsertObservationResponseDocument.Factory.parse(opResult.getIncomingResultAsStream()).getInsertObservationResponse(); diff --git a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/model/Timestamp.java b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/model/Timestamp.java index f60ea61b..8d64b5fd 100644 --- a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/model/Timestamp.java +++ b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/model/Timestamp.java @@ -34,10 +34,11 @@ /** * @author Eike Hinderk Jürrens + * + * TODO switch to Joda time or java 8 to get time zone problems fixed */ public class Timestamp { - private static final String DEFAULT_PATTERN = "yyyy-MM-dd'T'HH:mm:ssX"; private static final int millisPerDay = 1000 * 60 * 60 * 24; private short year = Short.MIN_VALUE; private byte month = Byte.MIN_VALUE; @@ -49,7 +50,7 @@ public class Timestamp { @Override public String toString() { - // yyyy-mm-ddThh:mm:ss+hh:mm+zz:zz => 31 chars + // yyyy-MM-ddTHH:mm:ss+hh:mm => 31 chars final StringBuffer ts = new StringBuffer(31); if (year != Short.MIN_VALUE) { ts.append(year); @@ -123,7 +124,7 @@ public Timestamp set(final long dateToSet) { } /** - * @param fileName the filename that might contain additional information + * @param timestampInformation the filename that might contain additional information * for the {@link Timestamp} * @param regExToExtractFileInfo * @param dateInfoPattern @@ -135,19 +136,19 @@ public Timestamp set(final long dateToSet) { * @throws IndexOutOfBoundsException in the case of no group is found using * the value of the Datafile attribute "regExDateInfoInFileName". */ - public Timestamp enrichByFilename( - final String fileName, + public Timestamp enrich( + final String timestampInformation, final String regExToExtractFileInfo, final String dateInfoPattern) throws ParseException { - if (fileName == null || fileName.isEmpty() || + if (timestampInformation == null || timestampInformation.isEmpty() || regExToExtractFileInfo == null || regExToExtractFileInfo.isEmpty() || dateInfoPattern == null || dateInfoPattern.isEmpty()) { return this; } final Pattern pattern = Pattern.compile(regExToExtractFileInfo); - final Matcher matcher = pattern.matcher(fileName); - if (matcher.matches()) { + final Matcher matcher = pattern.matcher(timestampInformation); + if (matcher.matches() && matcher.groupCount() == 1) { final SimpleDateFormat sdf = new SimpleDateFormat(dateInfoPattern); final String dateInformation = matcher.group(1); final GregorianCalendar cal = new GregorianCalendar(); @@ -179,13 +180,63 @@ public Timestamp enrichByFilename( } protected Date toDate() { + final String datePattern = getDatePattern(); try { - return new SimpleDateFormat(DEFAULT_PATTERN).parse(toString()); + return new SimpleDateFormat(datePattern).parse(toString()); } catch (final ParseException e) { throw new RuntimeException("Could not execute toDate()",e); } } + private String getDatePattern() { + // "yyyy-MM-dd'T'HH:mm:ssX"; + final StringBuffer ts = new StringBuffer(31); + if (year != Short.MIN_VALUE) { + ts.append("yyyy"); + if (month != Byte.MIN_VALUE) { + ts.append("-"); + } + } + if (month != Byte.MIN_VALUE) { + ts.append("MM"); + if (day != Byte.MIN_VALUE) { + ts.append("-"); + } + } + if (day != Byte.MIN_VALUE) { + ts.append("dd"); + } + if ( (year != Short.MIN_VALUE || month != Byte.MIN_VALUE || day != Byte.MIN_VALUE ) + && (hour != Byte.MIN_VALUE || minute != Byte.MIN_VALUE || seconds != Byte.MIN_VALUE)) { + ts.append("'T'"); + } + if (hour != Byte.MIN_VALUE) { + ts.append("HH"); + if (minute != Byte.MIN_VALUE) { + ts.append(":"); + } + } + if (minute != Byte.MIN_VALUE) { + ts.append("mm:"); + } else if (hour != Byte.MIN_VALUE) { + ts.append("mm:"); + } + if (seconds != Byte.MIN_VALUE ) { + ts.append("ss"); + } else if (minute != Byte.MIN_VALUE && hour != Byte.MIN_VALUE) { + ts.append("ss"); + } + if (timezone != Byte.MIN_VALUE && + (hour != Byte.MIN_VALUE || minute != Byte.MIN_VALUE || seconds != Byte.MIN_VALUE)) { + ts.append("X"); + } + final String datePattern = ts.toString(); + if (datePattern.isEmpty()) { + return "yyyy-MM-dd'T'HH:mm:ssX"; + } + return datePattern; + } + public boolean after(final Timestamp timeStamp) { if (timeStamp == null) { throw new IllegalArgumentException("parameter timeStamp is mandatory."); @@ -204,7 +255,7 @@ public boolean before(final Timestamp timeStamp) { * @param lastModified long * @param lastModifiedDelta -1, if it should be ignored, else > 0. */ - public Timestamp enrichByFileModificationDate(long lastModified, + public Timestamp enrich(long lastModified, final int lastModifiedDelta) { final GregorianCalendar cal = new GregorianCalendar(); if (lastModifiedDelta > 0) { @@ -217,6 +268,41 @@ public Timestamp enrichByFileModificationDate(long lastModified, return this; } + public Timestamp applyDayDelta(final int daysToAdd) { + final Timestamp tmp = new Timestamp().set(toDate().getTime() + (daysToAdd * millisPerDay)); + setYear(tmp.getYear()); + setMonth(tmp.getMonth()); + setDay(tmp.getDay()); + return this; + } + + public Timestamp enrich(final Timestamp other) { + if (other != null) { + if (other.getYear() > Short.MIN_VALUE) { + setYear(other.getYear()); + } + if (other.getMonth() > Byte.MIN_VALUE) { + setMonth(other.getMonth()); + } + if (other.getDay() > Byte.MIN_VALUE) { + setDay(other.getDay()); + } + if (other.getHour() > Byte.MIN_VALUE) { + setHour(other.getHour()); + } + if (other.getMinute() > Byte.MIN_VALUE) { + setMinute(other.getMinute()); + } + if (other.getSeconds() > Byte.MIN_VALUE) { + setSeconds(other.getSeconds()); + } + if (other.getTimezone() > Byte.MIN_VALUE) { + setTimezone(other.getTimezone()); + } + } + return this; + } + public void setYear(final short year) { this.year = year; } @@ -272,4 +358,5 @@ public byte getSeconds() { public byte getTimezone() { return timezone; } + } diff --git a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/task/OneTimeFeeder.java b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/task/OneTimeFeeder.java index 26d2a3c1..535903ab 100644 --- a/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/task/OneTimeFeeder.java +++ b/52n-sos-importer-feeder/src/main/java/org/n52/sos/importer/feeder/task/OneTimeFeeder.java @@ -30,7 +30,7 @@ import java.io.PrintWriter; import java.net.MalformedURLException; import java.net.SocketException; -import java.net.URL; +import java.text.ParseException; import java.util.List; import java.util.Scanner; @@ -40,7 +40,6 @@ import org.n52.oxf.OXFException; import org.n52.oxf.ows.ExceptionReport; import org.n52.sos.importer.feeder.Configuration; -import org.n52.sos.importer.feeder.Configuration.ImportStrategy; import org.n52.sos.importer.feeder.DataFile; import org.n52.sos.importer.feeder.SensorObservationService; import org.n52.sos.importer.feeder.model.requests.InsertObservation; @@ -157,18 +156,10 @@ public void run() { if (dataFile.isAvailable()) { try { // check SOS - final URL sosURL = config.getSosUrl(); - final String sosVersion = config.getSosVersion(); - final String sosBinding = config.getSosBinding(); - final ImportStrategy importStrategy = config.getImportStrategy(); - final int hunkSize = config.getHunkSize(); SensorObservationService sos = null; + final String sosURL = config.getSosUrl().toString(); try { - sos = new SensorObservationService(sosURL, - sosVersion, - sosBinding, - importStrategy, - hunkSize); + sos = new SensorObservationService(config); } catch (final ExceptionReport er) { LOG.error("SOS " + sosURL + " is not available. Please check the configuration!", er); } catch (final OXFException oxfe) { @@ -225,6 +216,10 @@ public void run() { log(e); } catch (final XmlException e) { log(e); + } catch (final ParseException e) { + log(e); + } catch (final IllegalArgumentException e) { + log(e); } } } diff --git a/52n-sos-importer-feeder/src/main/resources/logback.xml b/52n-sos-importer-feeder/src/main/resources/logback.xml index d529e8a4..8bf3eeaf 100644 --- a/52n-sos-importer-feeder/src/main/resources/logback.xml +++ b/52n-sos-importer-feeder/src/main/resources/logback.xml @@ -39,7 +39,7 @@ - + diff --git a/52n-sos-importer-feeder/src/test/java/org/n52/sos/importer/feeder/model/TimestampTest.java b/52n-sos-importer-feeder/src/test/java/org/n52/sos/importer/feeder/model/TimestampTest.java index 4905c279..e8cf54ef 100644 --- a/52n-sos-importer-feeder/src/test/java/org/n52/sos/importer/feeder/model/TimestampTest.java +++ b/52n-sos-importer-feeder/src/test/java/org/n52/sos/importer/feeder/model/TimestampTest.java @@ -32,9 +32,7 @@ import org.junit.Before; import org.junit.Test; -/** - * - */ + public class TimestampTest { private static final int millisPerDay = 1000 * 60 * 60 * 24; @@ -80,58 +78,60 @@ public void createTimestamp() throws Exception } - @Test - public final void shouldGetAdditionalTimestampValuesFromFileName() throws ParseException { + @Test public final void + shouldGetAdditionalTimestampValuesFromFileName() + throws ParseException { final String fileName = "test-sensor_20140615.csv"; final Timestamp ts = new Timestamp(); - ts.enrichByFilename(fileName,"test-sensor_(\\d{8})\\.csv","yyyyMMdd"); + ts.enrich(fileName,"test-sensor_(\\d{8})\\.csv","yyyyMMdd"); assertThat(ts.toString(), is("2014-06-15")); } - @Test - public final void shouldReturnSameValueIfParametersAreInvalid() throws ParseException { + @Test public final void + shouldReturnSameValueIfParametersAreInvalid() + throws ParseException { Timestamp ts = new Timestamp(); - ts.enrichByFilename(null, null, null); + ts.enrich(null, null, null); assertThat(ts.toString(), is("")); ts = new Timestamp(); - ts.enrichByFilename("", null, null); + ts.enrich("", null, null); assertThat(ts.toString(), is("")); ts = new Timestamp(); - ts.enrichByFilename("-", null, null); + ts.enrich("-", null, null); assertThat(ts.toString(), is("")); ts = new Timestamp(); - ts.enrichByFilename("-", "", null); + ts.enrich("-", "", null); assertThat(ts.toString(), is("")); ts = new Timestamp(); - ts.enrichByFilename("-", "-", null); + ts.enrich("-", "-", null); assertThat(ts.toString(), is("")); ts = new Timestamp(); - ts.enrichByFilename("-", "-", ""); + ts.enrich("-", "-", ""); assertThat(ts.toString(), is("")); } - @Test - public final void shouldEnrichWithLastModificationDate() { + @Test public final void + shouldEnrichWithLastModificationDate() { final long lastModified = getCurrentTimeMillisTimestampCompatible(); - timestamp.enrichByFileModificationDate(lastModified, -1); + timestamp.enrich(lastModified, -1); final Timestamp expected = new Timestamp().set(lastModified); assertThat(timestamp.getYear(), is(expected.getYear())); assertThat(timestamp.getMonth(), is(expected.getMonth())); assertThat(timestamp.getDay(), is(expected.getDay())); } - @Test - public final void shouldEnrichWithLastModificationDateWithLastModifiedDayDelta() { + @Test public final void + shouldEnrichWithLastModificationDateWithLastModifiedDayDelta() { final long lastModified = getCurrentTimeMillisTimestampCompatible(); final int lastModifiedDelta = 2; - timestamp.enrichByFileModificationDate(lastModified, lastModifiedDelta); + timestamp.enrich(lastModified, lastModifiedDelta); final long expectedMillis = lastModified - (lastModifiedDelta * millisPerDay); final Timestamp expected = new Timestamp().set(expectedMillis); assertThat(timestamp.getYear(), is(expected.getYear())); @@ -139,18 +139,56 @@ public final void shouldEnrichWithLastModificationDateWithLastModifiedDayDelta() assertThat(timestamp.getDay(), is(expected.getDay())); } - @Test - public final void shouldEnrichWithLastModificationDateWithLastModifiedDayDeltaWithYearChange() { + @Test public final void + shouldEnrichWithLastModificationDateWithLastModifiedDayDeltaWithYearChange() { final long lastModified = 0; final int lastModifiedDelta = 2; final long expectedMillis = lastModified - (lastModifiedDelta * millisPerDay); - timestamp.enrichByFileModificationDate(lastModified, lastModifiedDelta); + timestamp.enrich(lastModified, lastModifiedDelta); final Timestamp expected = new Timestamp().set(expectedMillis); assertThat(timestamp.getYear(), is(expected.getYear())); assertThat(timestamp.getMonth(), is(expected.getMonth())); assertThat(timestamp.getDay(), is(expected.getDay())); } + @Test public final void + shouldEnrichDateInformationFromOtherTimeStamp() { + final Timestamp other = new Timestamp().set(getCurrentTimeMillisTimestampCompatible()); + timestamp.set(0).enrich(other); + + assertThat(timestamp.getYear(), is(other.getYear())); + assertThat(timestamp.getMonth(), is(other.getMonth())); + assertThat(timestamp.getDay(), is(other.getDay())); + } + + @Test public void + shouldAddDayDelta() { + timestamp.set(0).applyDayDelta(2); + + assertThat(timestamp.getYear(), is((short)1970)); + assertThat(timestamp.getMonth(), is((byte)1)); + assertThat(timestamp.getDay(), is((byte)3)); + + timestamp.set(0).applyDayDelta(-2); + + assertThat(timestamp.getYear(), is((short)1969)); + assertThat(timestamp.getMonth(), is((byte)12)); + assertThat(timestamp.getDay(), is((byte)30)); + } + + @Test public void + shouldWorkWithoutTimezonesAddDayDelta() throws ParseException { + timestamp.enrich(new Timestamp().enrich("1970-01-01", "(\\d{4}-\\d{2}-\\d{2})", "yyyy-MM-dd").applyDayDelta(1)); + + assertThat(timestamp.getYear(), is((short)1970)); + assertThat(timestamp.getMonth(), is((byte)1)); + assertThat(timestamp.getDay(), is((byte)2)); + assertThat(timestamp.getSeconds(), is(Byte.MIN_VALUE)); + assertThat(timestamp.getMinute(), is(Byte.MIN_VALUE)); + assertThat(timestamp.getHour(), is(Byte.MIN_VALUE)); + assertThat(timestamp.getTimezone(), is(Byte.MIN_VALUE)); + } + private long getCurrentTimeMillisTimestampCompatible() { // Timestamp is not storing milliseconds now => remove them return (System.currentTimeMillis() / 1000) * 1000; diff --git a/RELEASE_NOTES b/RELEASE_NOTES index 214501f4..2ce41be1 100644 --- a/RELEASE_NOTES +++ b/RELEASE_NOTES @@ -26,6 +26,42 @@ Features: * "useDateFromLastModifiedDate" for enabling this feature * "lastModifiedDelta" for moving the date n days back (this attribute is OPTIONAL for this feature, too.) + * Ignore lines with regular expressions feature: + 0..infinity elements can be added to the element. + Each element will be used as regular expression and applied to each line of + the data file before parsing. + * Handling of data files containing several sample runs. A sample run contains + additional metadata like its size (number of performed measurements) and a + date. + The required attributes are: + * "sampleStartRegEx" - the start of a new sample (MUST match the whole + line). + * "sampleDateOffset" - the offset of the line containing the date of the + sample from the start line. + * "sampleDateExtractionRegEx" - the regular expression to extract the date + information from the line containing the + date information of the current sample. The + expression MUST result in ONE group. This + group will be parsed to a java.util.Date + using "sampleDatePattern" attribute. + * "sampleDatePattern" - the pattern used to parse the date information of + the current pattern. + * "sampleDataOffset" - the offset in lines from sample beginning till the + first lines with data. + * "sampleSizeOffset" - the offset in lines from sample beginning till the + line containing the sample size in lines with data. + * "sampleSizeRegEx" - the regular expression to extract the sample size. + The regular expression MUST result in ONE group + which contains an integer value. + * Setting of timeout buffer for the insertion of SweArrayObservations: + With the attribute "insertSweArrayObservationTimeoutBuffer" of 1s more connect and socket timeout. + The size of this value is related to the set-up of the SOS server, importer, + and the HUNK_SIZE value. + The current OX-F SimpleHttpClient implementation uses a default value of 5s, + hence setting this to 25,000 results in 30s connection and socket timeout. Changes: --------