Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task/handle textunrestricted attr #2128

Merged
merged 22 commits into from
Feb 3, 2022
Merged

Task/handle textunrestricted attr #2128

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
585a309
handle TextUnrestricted attr value
AlvaroVega Feb 3, 2022
338ab82
update test
AlvaroVega Feb 3, 2022
b7e5017
update test
AlvaroVega Feb 3, 2022
27705a1
get String when textUnrestricted
AlvaroVega Feb 3, 2022
45f46d4
get String when textUnrestricted
AlvaroVega Feb 3, 2022
4d8cb9e
apply for row aggregator
AlvaroVega Feb 3, 2022
2f1c304
use getEscapedString
AlvaroVega Feb 3, 2022
ebaabbe
update cnr
AlvaroVega Feb 3, 2022
b8eb08b
fix escape
AlvaroVega Feb 3, 2022
df6e57c
fix escape
AlvaroVega Feb 3, 2022
917a345
do not concat quotationMark
AlvaroVega Feb 3, 2022
34cbf66
add other quotationMark case
AlvaroVega Feb 3, 2022
78de604
Create unrestricted_type.md
AlvaroVega Feb 3, 2022
f4bdb5e
Update README.md
AlvaroVega Feb 3, 2022
04c5075
update doc
AlvaroVega Feb 3, 2022
246f328
Update doc/cygnus-ngsi/flume_extensions_catalogue/ngsi_rest_handler.md
AlvaroVega Feb 3, 2022
ba66dec
Delete unrestricted_type.md
AlvaroVega Feb 3, 2022
81554eb
Update README.md
AlvaroVega Feb 3, 2022
a997c26
ADD entry to ToC in .md
fgalan Feb 3, 2022
e0a8b63
Update CHANGES_NEXT_RELEASE
AlvaroVega Feb 3, 2022
626fd78
Update doc/cygnus-ngsi/flume_extensions_catalogue/ngsi_rest_handler.md
AlvaroVega Feb 3, 2022
08534b0
Update NGSIGenericAggregator.java
AlvaroVega Feb 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES_NEXT_RELEASE
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
- [cygnus-commons] Check TextUnrestricted type to escape character ' as '' (#2125)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe something about this should be done in documenation?

Not sure in which .md. Do we have one about NGSIHandler? (in a similar way we have per-sink documents)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in f4bdb5e

AlvaroVega marked this conversation as resolved.
Show resolved Hide resolved
- [cygnus-ngsi][KafkaSink] Upgrade libthrift dependency from 0.12.0 to 0.14.1 due to github vulnerability report
- [cygnus-commons] Upgrade postgresql dependency from 42.2.22 to 42.2.25 due to github vulnerability report
20 changes: 20 additions & 0 deletions cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/aggregation/NGSIGenericAggregator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -757,6 +757,26 @@ public void setEnableNameMappings(boolean enableNameMappings) {
this.enableNameMappings = enableNameMappings;
} //setEnableNameMappings

/**
* getEscapedString
*
* @param JsonElement an UnrestrictedString
* @param String a quotationMark to escape
* @return the escaped string
*/
public String getEscapedString(JsonElement value, String quotationMark) {
String escaped = value.toString();
switch (quotationMark) {
case "'":
escaped = escaped.replaceAll("'", "''");
break;
case "\"":
escaped = escaped.replaceAll("\"", "\"\"");
break;
}
return escaped;
}

/**
* Aggregate declaration for child classes.
*
Expand Down
2 changes: 2 additions & 0 deletions ...ngsi/src/main/java/com/telefonica/iot/cygnus/aggregation/NGSIGenericColumnAggregator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,8 @@ public void aggregate(NGSIEvent event) {
} catch (Exception e) {
LOGGER.error("[" + getName() + "] Processing context attribute (name=" + attrValue.toString());
}
} else if (attrType.equals("TextUnrestricted")) {
attrValue = jsonParser.parse(getEscapedString(attrValue, "'"));
}
// Check if the attribute already exists in the form of 2 columns (one for metadata); if not existing,
// add an empty value for all previous rows
Expand Down
3 changes: 3 additions & 0 deletions ...us-ngsi/src/main/java/com/telefonica/iot/cygnus/aggregation/NGSIGenericRowAggregator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ public void aggregate(NGSIEvent event) {
JsonArray jsonAttrMetadata = (JsonArray) jsonParser.parse(attrMetadata);
LOGGER.debug("[" + getName() + "] Processing context attribute (name=" + attrName + ", type="
+ attrType + ")");
if (attrType.equals("TextUnrestricted")) {
attrValue = jsonParser.parse(getEscapedString(attrValue, "'"));
}
// aggregate the attribute information
aggregation.get(NGSIConstants.RECV_TIME_TS).add(new JsonPrimitive(Long.toString(recvTimeTs)));
aggregation.get(NGSIConstants.RECV_TIME).add(new JsonPrimitive(recvTime));
Expand Down
8 changes: 4 additions & 4 deletions cygnus-ngsi/src/test/java/com/telefonica/iot/cygnus/sinks/NGSIMySQLSinkTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1287,8 +1287,8 @@ private NotifyContextRequest.ContextElement createContextElement() {
contextAttribute1.setContextMetadata(metadata);
NotifyContextRequest.ContextAttribute contextAttribute2 = new NotifyContextRequest.ContextAttribute();
contextAttribute2.setName("someName2");
contextAttribute2.setType("someType2");
contextAttribute2.setContextValue(new JsonPrimitive("someValue2"));
contextAttribute2.setType("TextUnrestricted");
contextAttribute2.setContextValue(new JsonPrimitive("someValue'2"));
contextAttribute2.setContextMetadata(null);
ArrayList<NotifyContextRequest.ContextAttribute> attributes = new ArrayList<>();
attributes.add(contextAttribute1);
Expand Down Expand Up @@ -1352,7 +1352,7 @@ public void testNativeTypeColumnBatch() throws CygnusBadConfiguration, CygnusRun
for (NGSIEvent event : events) {
aggregator.aggregate(event);
}
String correctBatch = "('2016-04-20 07:19:55.801','somePath','someId','someType',2,'[]',TRUE,'[]','2016-09-21T01:23:00.00Z','[]','{\"type\": \"Point\",\"coordinates\": [-0.036177,39.986159]}','[]','{\"String\": \"string\"}','[]','foo','[]','','[]',NULL,NULL,NULL,NULL),('2016-04-20 07:19:55.801','somePath','someId','someType',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,'-3.7167, 40.3833','[{\"name\":\"location\",\"type\":\"string\",\"value\":\"WGS84\"}]','someValue2','[]')";
String correctBatch = "('2016-04-20 07:19:55.801','somePath','someId','someType',2,'[]',TRUE,'[]','2016-09-21T01:23:00.00Z','[]','{\"type\": \"Point\",\"coordinates\": [-0.036177,39.986159]}','[]','{\"String\": \"string\"}','[]','foo','[]','','[]',NULL,NULL,NULL,NULL),('2016-04-20 07:19:55.801','somePath','someId','someType',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,'-3.7167, 40.3833','[{\"name\":\"location\",\"type\":\"string\",\"value\":\"WGS84\"}]','someValue''2','[]')";
String valuesForInsert = SQLQueryUtils.getValuesForInsert(aggregator.getAggregationToPersist(), aggregator.isAttrNativeTypes());
if (valuesForInsert.equals(correctBatch)) {
System.out.println(getTestTraceHead("[NGSIMySQKSink.testNativeTypesColumnBatch]")
Expand Down Expand Up @@ -1397,7 +1397,7 @@ public void testNativeTypeRowBatch() throws CygnusBadConfiguration, CygnusRuntim
for (NGSIEvent event : events) {
aggregator.aggregate(event);
} // for
String correctBatch = "('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someNumber','number','2','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','somneBoolean','Boolean','true','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someDate','DateTime','2016-09-21T01:23:00.00Z','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someGeoJson','geo:json','{\"type\": \"Point\",\"coordinates\": [-0.036177,39.986159]}','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someJson','json','{\"String\": \"string\"}','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someString','string','foo','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someString2','string','','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someName1','someType1','-3.7167, 40.3833','[{\"name\":\"location\",\"type\":\"string\",\"value\":\"WGS84\"}]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someName2','someType2','someValue2','[]')";
String correctBatch = "('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someNumber','number','2','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','somneBoolean','Boolean','true','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someDate','DateTime','2016-09-21T01:23:00.00Z','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someGeoJson','geo:json','{\"type\": \"Point\",\"coordinates\": [-0.036177,39.986159]}','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someJson','json','{\"String\": \"string\"}','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someString','string','foo','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someString2','string','','[]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someName1','someType1','-3.7167, 40.3833','[{\"name\":\"location\",\"type\":\"string\",\"value\":\"WGS84\"}]'),('1461136795801','2016-04-20 07:19:55.801','somePath','someId','someType','someName2','TextUnrestricted','someValue''2','[]')";
String valuesForInsert = SQLQueryUtils.getValuesForInsert(aggregator.getAggregationToPersist(), aggregator.isAttrNativeTypes());
if (valuesForInsert.equals(correctBatch)) {
System.out.println(getTestTraceHead("[NGSIMySQKSink.testNativeTypesRowBatch]")
Expand Down