[cygnus-row] errors when use "." in resource or organization

[iariasleon]

In resources, CKAN, mysql, as Hadoop; is created the element, but "." is changed by "_", example:

resource = modelogw.assetgw-device
http://130.206.83.8/dataset/test_default_6_fiware-test/resource/ce0be84e-1098-4bf0-bdbd-0c5467e7f474

organization:
ckan:

time=2014-09-22T14:30:13.752CEST | lvl=DEBUG | trans=1411388427-767-0000000021 | function=doCKANRequest | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.backends.ckan.CKANBackendImpl[781] : CKAN response (2484 bytes): HTTP/1.1 409 Conflict
time=2014-09-22T14:30:13.753CEST | lvl=DEBUG | trans=1411388427-767-0000000021 | function=doCKANRequest | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.backends.ckan.CKANBackendImpl[784] : response payload: {"help": "Create a new organization.\n\n    You must be authorized to create organizations.\n\n    Plugins may change the parameters of this function depending on the value\n    of the ``type`` parameter, see the ``IGroupForm`` plugin interface.\n\n    :param name: the name of the organization, a string between 2 and 100 characters\n        long, containing only lowercase alphanumeric characters, ``-`` and\n        ``_``\n    :type name: string\n    :param id: the id of the organization (optional)\n    :type id: string\n    :param title: the title of the organization (optional)\n    :type title: string\n    :param description: the description of the organization (optional)\n    :type description: string\n    :param image_url: the URL to an image to be displayed on the organization's page\n        (optional)\n    :type image_url: string\n    :param state: the current state of the organization, e.g. ``'active'`` or\n        ``'deleted'``, only active organizations show up in search results and\n        other lists of organizations, this parameter will be ignored if you are not\n        authorized to change the state of the organization (optional, default:\n        ``'active'``)\n    :type state: string\n    :param approval_status: (optional)\n    :type approval_status: string\n    :param extras: the organization's extras (optional), extras are arbitrary\n        (key: value) metadata items that can be added to organizations, each extra\n        dictionary should have keys ``'key'`` (a string), ``'value'`` (a\n        string), and optionally ``'deleted'``\n    :type extras: list of dataset extra dictionaries\n    :param packages: the datasets (packages) that belong to the organization, a list\n        of dictionaries each with keys ``'name'`` (string, the id or name of\n        the dataset) and optionally ``'title'`` (string, the title of the\n        dataset)\n    :type packages: list of dictionaries\n    :param users: the users that belong to the organization, a list of dictionaries\n        each with key ``'name'`` (string, the id or name of the user) and\n        optionally ``'capacity'`` (string, the capacity in which the user is\n        a member of the organization)\n    :type users: list of dictionaries\n\n    :returns: the newly created organization\n    :rtype: dictionary\n\n    ", "success": false, "error": {"name": ["Url must be purely lowercase alphanumeric (ascii) characters and these symbols: -_"], "__type": "Validation Error"}}
time=2014-09-22T14:30:13.753CEST | lvl=ERROR | trans=1411388427-767-0000000021 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[161] : Runtime error (Don't know how to treat the response code. Possibly the organization already exists in a deleted state (respCode=409, orgName=org601.test))

In MySQL:

time=2014-09-22T17:30:47.327CEST | lvl=DEBUG | trans=1411399088-851-0000000025 | function=createDatabase | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.backends.mysql.MySQLBackend[85] : Executing MySQL query (create database if not exists cygnus_org601.test)
time=2014-09-22T17:30:47.328CEST | lvl=ERROR | trans=1411399088-851-0000000025 | function=process | comp=Cygnus | msg=es.tid.fiware.fiwareconnectors.cygnus.sinks.OrionSink[161] : Runtime error (You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.test' at line 1)

[frbattid]

Let's try to summarize how Cygnus manages the non alphanumeric characters sink by sink, and try to homegenize all the approaches.

Let's see the problem from the data persistence point of view:

• HDFSSink (row)
  • Notified entityId and entityType are encoded by replacing all the non alphanumeric characters by '_', then they are persisted within HDFS files. This replacement is not really necessary.
• HDFSSink (column)
  • Nothing is persisted regarding the entityId norentityType.
• CKANSink (row)
  • Notified entityId and entityType are encoded by replacing all the non alphanumeric characters by '_', then they are persisted within CKAN resources. This replacement is not really necessary.
• CKANSink (column)
  • Nothing is persisted regarding the entityId norentityType
• MySQLSink (row)
  • Notified entityId and entityType are encoded by replacing all the non alphanumeric characters by '_', then they are persisted within MySQL tables. This replacement is not really necessary.
• MySQLSink (column)
  • Nothing is persisted regarding the entityId norentityType

Now, let's see the problem from the structure* point of view:

• organization building: two problems:
  • It is not encoded, thus the problems described by @iariasleon in this issue arise.
  • Is is converted into lower case (why?), thus the problems described by @iariasleon at https://github.com/telefonicaid/fiware-connectors/issues/197 arise.
• destination building: by using the matching rules table, this should not be a problem anymore (the user is responsible for defining the destination) except for the default destination building. This default destination should have into account:
  • The union of the forbidden characters for all the storages. Currently, no encoding is performed on entityId norentityType.
  • The concatenation character joining entityId and entityType. Currently, this is '-', which in the end must be replace by '_' in MySQL because '-' is not accepted.

(*)
HDFS structure == /user/default_user/organization/destination_folder/destination_file.txt
CKAN structure == organization -> defaultPackage -> destination resource -> datastore
MySQL structure == organization database -> destination table

[frbattid]

Fixed in PR https://github.com/telefonicaid/fiware-connectors/pull/223