bad srv and subsrv in logs according with SQL data

[AlvaroVega]

Bad srv /susrv in logs

time=2020-11-25T08:37:16.224Z | lvl=ERROR | corr=6c336856-2ef9-11eb-8a73-fa163e83ea20; cbnotif=1 | trans=368427b4-dc5b-4c28-ba03-3d06b4ef15e8 | srv=SSSSSS1 | subsrv=/ssssssssssss1 | op=processNewBatches | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.NGSISink[610] : BadContextData (SQLException). POSTGRESQL Data insertion error. Query: insert into parking_onstreetparking (recvTime,fiwareServicePath,entityId,entityType,Active,Active_md,TimeInstant,TimeInstant_md,address,address_md,allowedVehicleType,allowedVehicleType_md,areBordersMarked,areBordersMarked_md,areaServed,areaServed_md,availableSpotNumber,availableSpotNumber_md,capacityPercent,capacityPercent_md,category,category_md,chargeType,chargeType_md,description,description_md,displayCapacity,displayCapacity_md,displayCapacityPercent,displayCapacityPercent_md,displayNumOccupied,displayNumOccupied_md,displayOccupiedPercent,displayOccupiedPercent_md,extraSpotNumberPMR,extraSpotNumberPMR_md,location,location_md,name,name_md,numOccupied,numOccupied_md,occupancyDetectionType,occupancyDetectionType_md,occupiedPercent,occupiedPercent_md,permitActiveHours,permitActiveHours_md,requiredPermit,requiredPermit_md,totalSpotNumber,totalSpotNumber_md) values ('2020-11-25 08:37:13.979','/ssssssssssss2','QuatreCamins','OnStreetParking',TRUE,'[]','2020-11-25T08:37:13.967Z','[]','Crta. Sant Boi','[]','anyVehicle','[]',TRUE,'[]','FGC','[]',71,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:36:48.233Z"}]',75,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:37:13.967Z"}]','free','[]','free','[]','Aparcamiento de Quatre Camins','[]',71,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:36:48.560Z"}]',46,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:36:48.727Z"}]',83,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:36:48.783Z"}]',54,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:36:48.675Z"}]',3,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:37:13.856Z"}]',ST_GeomFromGeoJSON('{"type":"Point","coordinates":[2.00174,41.40672]}'),'[{"name":"location","type":"string","value":"WGS84"}]','Quatre Camins','[]',83,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:36:48.618Z"}]','induction','[]',25,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:37:13.914Z"}]','{}','[]','noPermitNeeded','[]',154,'[{"name":"TimeInstant","type":"DateTime","value":"2020-11-25T08:36:48.233Z"}]') (ERROR: column "extraspotnumberpmr" of relation "parking_onstreetparking" does not exist Position: 580). Stack trace: [com.telefonica.iot.cygnus.backends.sql.SQLBackendImpl.insertContextData(SQLBackendImpl.java:249), com.telefonica.iot.cygnus.sinks.NGSIPostgisSink.persistAggregation(NGSIPostgisSink.java:466), com.telefonica.iot.cygnus.sinks.NGSIPostgisSink.persistBatch(NGSIPostgisSink.java:397), com.telefonica.iot.cygnus.sinks.NGSISink.processNewBatches(NGSISink.java:597), com.telefonica.iot.cygnus.sinks.NGSISink.process(NGSISink.java:373), org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68), org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147), java.lang.Thread.run(Thread.java:748)]

[AlvaroVega]

One batch is able to process/store events from several services and subservices at the same time.
Maybe there is no sense in set service and subservice for batch log:

fiware-cygnus/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/sinks/NGSISink.java

Lines 567 to 571 in 6c442f5

	MDC.put(CommonConstants.LOG4J_SVC,
	ngsiEvent.getHeaders().get(CommonConstants.HEADER_FIWARE_SERVICE));
	MDC.put(CommonConstants.LOG4J_SUBSVC,
	ngsiEvent.getHeaders().get(CommonConstants.HEADER_FIWARE_SERVICE_PATH));

and we can remove above lines @IvanHdzC ?

[IvanHdzC]

One batch is able to process/store events from several services and subservices at the same time.
Maybe there is no sense in set service and subservice for batch log:

fiware-cygnus/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/sinks/NGSISink.java

Lines 567 to 571 in 6c442f5

	MDC.put(CommonConstants.LOG4J_SVC,
	ngsiEvent.getHeaders().get(CommonConstants.HEADER_FIWARE_SERVICE));
	MDC.put(CommonConstants.LOG4J_SUBSVC,
	ngsiEvent.getHeaders().get(CommonConstants.HEADER_FIWARE_SERVICE_PATH));

and we can remove above lines @IvanHdzC ?

Would it be a good idea to set those headers as N/A? I'm not sure if they can be null.

[fgalan]

PR #1985, now merged.

Maybe the issue can be closed?