Feeder fails to read FoI position

[lieberjosh]

Successfully created a configuration file from a .csv file of 4 humidity observations per feature of interest with lat-lon coordinates. Feeder fails, though, with a null position and then a null pointer exception reading the LongitudeUnit. Any suggestions? This happens with builds under both Java 7 and 8, both remotely and locally to the SOS service.
DATA_Obs_160907_1sensor_csv.txt
DATA_Obs_160907_1sensor.csv.52n-sos-import-config_xml.txt

java -jar 52n-sos-importer-feeder-bin.jar -c ~/tmp/FCP1_-RBG_DATA_Obs_160907_1sensor.csv.52n-sos-import-config.xml
10:56:45.177 [main] INFO Feeder - Application started
Implementation-Vendor: N/A
Implementation-Title: Java Runtime Environment
Implementation-Version: 1.7.0_111
Implementation-Vendor-Id: com.sun
Specification-Vendor: Oracle Corporation
Manifest-Version: 1.0
Specification-Title: Java Platform API Specification
Created-By: 1.7.0_0 (Oracle Corporation)
Specification-Version: 1.7
Extension-Name: javax.crypto
HeapSize Information: max: 878MB; total now: 59MB; free now: 47MB; used now: 11MB
os.name: Linux; os.arch: amd64; os.version: 4.5.5-x86_64-linode69
10:56:45.715 [OneTimeFeeder] INFO OneTimeFeeder - Starting feeding data from file via configuration 'FCP1-RBG_DATA_Obs_160907_1sensor.csv.52n-sos-import-config.xml' to SOS instance
10:56:45.745 [OneTimeFeeder] TRACE SensorObservationService - SensorObservationService(Configuration [file=/home/server/tmp/FCP1-_RBG_DATA_Obs_160907_1sensor.csv.52n-sos-import-config.xml])
10:56:46.025 [OneTimeFeeder] DEBUG SimpleHttpClient - executing POST method to 'http://serv.tumblingwalls.com/52n-sos-webapp/service'.
10:56:47.443 [OneTimeFeeder] DEBUG SensorObservationService - OperationsMetadata found: org.n52.oxf.ows.capabilities.OperationsMetadata@5ed7443d
10:56:47.444 [OneTimeFeeder] DEBUG SensorObservationService - Found all required operations: (RegisterSensor|InsertSensor), InsertObservation
10:56:47.448 [OneTimeFeeder] DEBUG OneTimeFeeder - Check counter file '/home/server/.SOSImporter/b29600c2497f5fd23ef620bf97fd7819'.
10:56:47.449 [OneTimeFeeder] DEBUG OneTimeFeeder - Counter file does not exist.
10:56:47.449 [OneTimeFeeder] TRACE SensorObservationService - importData()
10:56:47.469 [OneTimeFeeder] DEBUG SensorObservationService - Handling CSV line #2: [SE10_0AB_1, Sensor1, 51.487902, 0.004408, 0.71, Rel_Humidity, Percent, 3/14/2016 1.30 PM, , , �]
10:56:47.470 [OneTimeFeeder] DEBUG SensorObservationService - Parsing measured value column 4
10:56:47.474 [OneTimeFeeder] DEBUG SensorObservationService - Sensor: Sensor [name=Sensor1, uri=Sensor1]
10:56:47.478 [OneTimeFeeder] DEBUG SensorObservationService - Feature of Interest: FeatureOfInterest [p=null, name=SE10_0AB_1, uri=SE10_0AB_1]
10:56:47.499 [OneTimeFeeder] DEBUG SensorObservationService - Value: 0.71
10:56:47.516 [OneTimeFeeder] DEBUG SensorObservationService - Timestamp: 2016-03-14T13:30:00+01:00
10:56:47.519 [OneTimeFeeder] DEBUG SensorObservationService - UomCode: 'UnitOfMeasurement [name=Percent, uri=Percent]'
10:56:47.524 [OneTimeFeeder] DEBUG SensorObservationService - ObservedProperty: ObservedProperty [name=Rel_Humidity, uri=Rel_Humidity]
10:56:47.527 [OneTimeFeeder] DEBUG SensorObservationService - Offering: Offering [name=Sensor1, uri=Sensor1]
10:56:47.529 [OneTimeFeeder] DEBUG SensorObservationService - Found '1' Observed Properties for Sensor 'Sensor1': '[ObservedProperty [name=Rel_Humidity, uri=Rel_Humidity]]'
10:56:47.530 [OneTimeFeeder] DEBUG SensorObservationService - Found '1' Measured value types for observed properties of sensor 'Sensor1': '{ObservedProperty [name=Rel_Humidity, uri=Rel_Humidity]=NUMERIC}'.
10:56:47.530 [OneTimeFeeder] DEBUG SensorObservationService - Found '1' units of measurement for observed properties of sensor 'Sensor1': '{ObservedProperty [name=Rel_Humidity, uri=Rel_Humidity]=Percent}'.
Exception in thread "OneTimeFeeder" java.lang.NullPointerException
at org.n52.sos.importer.feeder.model.requests.RegisterSensor.getLongitudeUnit(RegisterSensor.java:107)
at org.n52.sos.importer.feeder.util.DescriptionBuilder.createSML(DescriptionBuilder.java:79)
at org.n52.sos.importer.feeder.SensorObservationService.createInsertSensorParametersFromRS(SensorObservationService.java:1018)
at org.n52.sos.importer.feeder.SensorObservationService.registerSensor(SensorObservationService.java:940)
at org.n52.sos.importer.feeder.SensorObservationService.insertObservationsForOneLine(SensorObservationService.java:667)
at org.n52.sos.importer.feeder.SensorObservationService.importData(SensorObservationService.java:304)
at org.n52.sos.importer.feeder.task.OneTimeFeeder.run(OneTimeFeeder.java:203)
at java.lang.Thread.run(Thread.java:745)

[EHJ-52n]

Dear @lieberjosh,
take a look at your datafile line 8 for example: SE10_0BT_2,Sensor1,51.494373�,0.012119,0.73,Rel_Humidity,Percent,5/14/2016 1.33 PM,,,

The char � should not be there, I think. Does the import work, if the line is clean?

Please ensure, that your data file is clean or the encoding is matching. Take a look at the documentation (search for encoding). The encoding is an optional tag child of the LocalFile configuration.

Do this solve your issue?

[lieberjosh]

Sadly no. The extraneous character crept in while I was cleaning up the file to post, but I removed it, made sure all lat-lon values were valid numbers, saved the file in UTF-8, but I get the same (lack of) result:

07:52:04.784 [OneTimeFeeder] DEBUG SensorObservationService - Feature of Interest: FeatureOfInterest [p=null, name=SE10_0AB_1, uri=SE10_0AB_1]
...
Exception in thread "OneTimeFeeder" java.lang.NullPointerException
at org.n52.sos.importer.feeder.model.requests.RegisterSensor.getLongitudeUnit(RegisterSensor.java:107)
DATA_Obs_160907_1sensor4obs.csv.52n-sos-import-config_xml.txt
FCP1_-_RBG_DATA_Obs_160907_1sensor_csv.txt

[EHJ-52n]

Dear @lieberjosh ,
I tested your configuration and datafile. That the configuration is not matching the datafile in terms of number of columns.

11:53:54.721 [main] ERROR SensorObservationService - Number of Expected columns '8' does not match number of found columns '11' -> Cancel import!

Please add for the 3 not used columns the following configuration snippet to your config file:

[...]
  <Column>
    <Number>8</Number>
    <Type>DO_NOT_EXPORT</Type>
  </Column>
  <Column>
    <Number>9</Number>
    <Type>DO_NOT_EXPORT</Type>
  </Column>
  <Column>
    <Number>10</Number>
    <Type>DO_NOT_EXPORT</Type>
  </Column>
</ColumnAssignments>

Did you create the configuration manually or using the wizard. The NPE is caused because the configuration doesn't tell the feeder that the foi is part of each line and not contained in the configuration xml itself.

[EHJ-52n]

It is a problem with the feeder logic. Will be fixed with the next commits.

[EHJ-52n]

@lieberjosh

Could you please verify, if this version of the feeder is working with your updated configuration (see #57 (comment)):

https://52north.org/delivery/SensorWeb/sos-importer/issue-057/52n-sos-importer-feeder-bin.jar (42MB)

If you want to build from source, please checkout this branch:

EHJ-52n/sos-importer/fix/read-foi-position

[debard]

Hi,
thank you for the update of the version for feeder. I have the same problem and i run with the new version. The error is fixed. The FOI position is readed correctly in my csv data without error.
Thank a lot
Regards