Oracle Sink (11g and 12c)

[AlvaroVega]

issue #2195

This new sinks is based on SQL and implements:

• Column support (no upsert mode support for lastdata, just insert)
• Geo:Point support
• Geo:json support (uses oracle 12c locator (spatial) feature https://docs.oracle.com/database/121/SPATL/sdo_locator.htm#SPATL340)
• Doc about new sink: https://github.com/telefonicaid/fiware-cygnus/blob/task/oracle_sink/doc/cygnus-ngsi/flume_extensions_catalogue/ngsi_oracle_sink.md
• Uses ojdbc6 : Oracle JDBC Driver compatible with JDK6, JDK7, and JDK8 which implements JDBC 4.0 spec (https://docs.oracle.com/cd/B28359_01/java.111/b31224/jdbcvers.htm#BCFFIFAD) valid for oracle 11g and oracle 12c (https://docs.oracle.com/en/database/oracle/oracle-database/12.2/jjdbc/JDBC-standards-support.html#GUID-0ABBC622-2FC8-4713-810F-F1417200C361)

Tested using this instance: Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production

iot-oracle:
  container_name: iot-oracle
  hostname: iot-oracle
  image: oracleinanutshell/oracle-xe-11g:latest
  ports:
   - 1521:1521
   - 5500:8080
  environment:
   - ORACLE_ALLOW_REMOTE=true
   - ORACLE_DISABLE_ASYNCH_IO=true
   - ORACLE_ENABLE_XDB=true
   - NLS_TIMESTAMP_FORMAT=YYYY-MM-DD HH24:MI:SS.FF6
   - NLS_TIMESTAMP_TZ_FORMAT=YYYY-MM-DD"T"HH24:MI:SS.FF6 TZR
   - NLS_DATE_FORMAT=YYYY-MM-DD HH24:MI:SS
   - NLS_LANG=SPANISH_SPAIN.UTF8   
   #system/oracle
   # databasename: xe
  log_driver: json-file
  log_opt:
    max-size: "250m"

Typical oracle-sink config:

# # NGSIOracleSink configuration
cygnus-ngsi.sinks.oracle-sink.type = com.telefonica.iot.cygnus.sinks.NGSIOracleSQLSink
cygnus-ngsi.sinks.oracle-sink.channel = oracle-channel
#cygnus-ngsi.sinks.oracle-sink.enable_encoding = false
cygnus-ngsi.sinks.oracle-sink.enable_grouping = false
cygnus-ngsi.sinks.oracle-sink.enable_name_mappings = true
#cygnus-ngsi.sinks.oracle-sink.enable_lowercase = false
cygnus-ngsi.sinks.oracle-sink.data_model = dm-by-entity-type
cygnus-ngsi.sinks.oracle-sink.oracle_host = iot-oracle
cygnus-ngsi.sinks.oracle-sink.oracle_port = 1521
cygnus-ngsi.sinks.oracle-sink.oracle_database = xe
cygnus-ngsi.sinks.oracle-sink.oracle_username = system
cygnus-ngsi.sinks.oracle-sink.oracle_password = oracle
cygnus-ngsi.sinks.oracle-sink.attr_persistence = column
cygnus-ngsi.sinks.oracle-sink.attr_native_types = true
cygnus-ngsi.sinks.oracle-sink.batch_size = 1
cygnus-ngsi.sinks.oracle-sink.batch_timeout = 10

New optional sink configuration options:

cygnus-ngsi.sinks.oracle-sink.nls_timestamp_format=YYYY-MM-DD HH24:MI:SS.FF6
cygnus-ngsi.sinks.oracle-sink.nls_timestamp_tz_format=YYYY-MM-DDTHH24:MI:SS.FF6 TZR
cygnus-ngsi.sinks.oracle-sink.oracle_locator=false 

Example of table creation for column and dm-by-entity-type:

CREATE TABLE thing (
    entityid varchar2(40) NOT NULL,
    entitytype varchar2(40) NOT NULL,
    recvtime timestamp,
    timeinstant timestamp with time zone,
    timeinstant_md varchar2(40),
    fiwareservicepath varchar2(40) NOT NULL,
    st1 varchar2(40),
    st1_md varchar2(40),
    st2 varchar2(40),
    st2_md varchar2(40),
    location SDO_GEOMETRY,
    location_md varchar2(40),
    CONSTRAINT tmp1_pk PRIMARY KEY (entityid, entitytype, recvtime)
);

[fgalan]

No "Last Data Functionality" sub-bullet? Does that means that lastdata is not yet implemented in the OracleSink?

[fgalan]

In the ngsi_oracle_sink.md file I see:

| last\_data\_mode | no | insert | <i>insert</i>  to set last data mode. Check this [link](./last_data_function.md) for more details. In oracle sink <i>both</i> and <i>upsert</i> are not avaiable |

so maybe it's just a typo here.

[AlvaroVega]

That's: Column support (no upsert mode support for lastdata, just insert)

[fgalan]

Clarified at skype. NTC

[fgalan]

LGTM