Cygnus includes extra quotes when posting to CKAN

[RobertoGallegoDelgado]

We have detected that Cygnus includes a double quote three times when the Context Broker data is of type Text (and therefore has a double quote) and it persists this data in CKAN.

When an attribute is in Context Broker and is of type Text, it is displayed in Context Broker with a double quote. Quad is of type Number, Context Broker shows it without double quote. Example:
kpi_product_1,"KeyPerformanceIndicator","daily","[""quantitative""]","2020-01-03T12:19:53.448Z","2020-05-14T07:07:38.795Z",0,"product statistics","{""name"":""Restaurante X""}","Pasta con tomate y orégano",

• "KeyPerformanceIndicator" is Text type
• 0 is Number type

When this data is published in CKAN, the Text type attributes are exported (in CSV format) as """XXX"""
For example, the above example would display as:

"""KeyPerformanceIndicator"""

The result when viewing and downloading the resources in CKAN is an entity like this:

121,2020-06-22T09:14:08,"""/envAirQualityRESPIRA3""","""RESPIRA""","""RESPIRA:RESPIRA_240AC4A3EBF8""",2020-06-22T09:07:44,"[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-16T00:10:13.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]","""0""","[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-16T00:10:13.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]","""5.25""","[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-16T00:10:13.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]","""4.94""","[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-16T00:10:13.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]","""5.25""","[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-16T00:10:13.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]","""5.25""","[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-16T00:10:13.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]","""44.64""","[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-16T00:10:13.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]","""27.06""","[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-16T00:10:13.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]","""0.55""","[{""type"": ""DateTime"", ""name"": ""dateCreated"", ""value"": ""2019-11-30T13:38:12.00Z""}, {""type"": ""DateTime"", ""name"": ""dateModified"", ""value"": ""2020-06-22T09:07:44.00Z""}]"

We request that the publication in CKAN be the same as that shown in Context Broker, without triple double quotes.

[AlvaroVega]

for a event like:

({"subscriptionId":"5f475ac02d7063421980836a","originator":"localhost","contextResponses":[{"contextElement":{"type":"RESPIRA","isPattern":"false","id":"bada1","attributes":[{"name":"TimeInstant","type":"DateTime","value":"2020-07-01T20:32:06.000Z","metadatas":[{"name":"dateCreated","type":"DateTime","value":"2020-01-10T10:08:33.000Z"},{"name":"dateModified","type":"DateTime","value":"2020-07-08T10:55:38.000Z"}]},{"name":"airQualityIndex","type":"Number","value":28,"metadatas":[{"name":"dateCreated","type":"DateTime","value":"2020-01-10T10:08:33.000Z"},{"name":"dateModified","type":"DateTime","value":"2020-07-08T10:55:38.000Z"}]}]},"statusCode":{"code":"200","reasonPhrase":"OK"}}]})

a log obtained is

time=2020-09-01T07:58:13.743Z | lvl=INFO | corr=de0761da-ec28-11ea-8a31-fa163ec74401 | trans=2ccef30f-ecfc-4c73-8199-7310a08ae19a | srv=dipbad | subsrv=/envAirQualityRESPIRA | op=persistAggregation | comp=Cygnus | msg=com.telefonica.iot.cygnus.sinks.NGSICKANSink[368] : [valencia-sink-ckanvalencia] Persisting data at NGSICKANSink (orgName=diputacion-de-badajoz, pkgName=diputacion-de-badajoz_calidad-del-aire-durante-el-2020, resName=indicesdecalidaddelaire_respira, data={"recvTime":"2020-09-01T07:58:09.707Z","fiwareServicePath":"/envAirQualityRESPIRA","entityId":"bada1","entityType":"RESPIRA","TimeInstant":"2020-07-01T20:32:06.000Z","TimeInstant_md":"[{\"name\":\"dateCreated\",\"type\":\"DateTime\",\"value\":\"2020-01-10T10:08:33.000Z\"},{\"name\":\"dateModified\",\"type\":\"DateTime\",\"value\":\"2020-07-08T10:55:38.000Z\"}]","airQualityIndex":28,"airQualityIndex_md":"[{\"name\":\"dateCreated\",\"type\":\"DateTime\",\"value\":\"2020-01-10T10:08:33.000Z\"},{\"name\":\"dateModified\",\"type\":\"DateTime\",\"value\":\"2020-07-08T10:55:38.000Z\"}]"})

[manucarrace]

The requests cygnus is making:

POST /api/3/action/datastore_upsert HTTP/1.1
Authorization: 3d20cdae-44ad-4eeb-a8b7-8de485140133
Content-Type: application/json; charset=utf-8
Content-Length: 698
Host: 10.0.0.2:5055
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.3.6 (java 1.5)

{ "resource_id": "cefef520-66a9-4616-b6e2-d0730545c47b", "records": [ {"recvTime":"2020-09-01T08:24:10.602Z","fiwareServicePath":"/envAirQualityRESPIRA","entityId":"bada1","entityType":"RESPIRA","TimeInstant":"2020-07-01T20:32:06.000Z","TimeInstant_md":"[{\"name\":\"dateCreated\",\"type\":\"DateTime\",\"value\":\"2020-01-10T10:08:33.000Z\"},{\"name\":\"dateModified\",\"type\":\"DateTime\",\"value\":\"2020-07-08T10:55:38.000Z\"}]","airQualityIndex":30,"airQualityIndex_md":"[{\"name\":\"dateCreated\",\"type\":\"DateTime\",\"value\":\"2020-01-10T10:08:33.000Z\"},{\"name\":\"dateModified\",\"type\":\"DateTime\",\"value\":\"2020-07-08T10:55:38.000Z\"}]"} ], "method": "insert", "force": "true" }HTTP/1.1 200 OK

[AlvaroVega]

Reproduced in cygnus 2.3.0

[fgalan]

Solved by PR #1919 ? Should the issue be closed?

[AlvaroVega]

Still not close this issue due to #1919 (comment)

[fgalan]

After PR #1923 maybe this issue is now closeable.