Fixing JSON representation format for entities and attributes in APIv2

[kzangeli]

This issue implements a set of changes of the entities/attributes representation format discussed during September 2015 by

• @jmcanterafonseca,
• @fgalan,
• @mrutid,
• @kzangeli and
• @antoniojoseln

See http://telefonicaid.github.io/fiware-orion/api/v2/ for details (yet to be added).

Parts to implement:

1. Creation/Update of Entities
2. Rendering of Entity
3. Creation/Update of Attribute
4. Rendering of Attribute
5. Update of Attribute Value
6. Rendering of Attribute Value
7. Selector to make the broker accept different input payloads for /v2/entities
8. Make attribute::value non-mandatory - if not present, value is NULL
9. Modify input format of strings in 5. Update of Attribute Value, not to need citation-marks (not finally needed)
10. (Introduced by FGM at 26/10/2015) Replace op=append by options=[...,]append[,...] usage (I have looked to feature/1259_new_json_for_v2 branch and it seems that the old way is still in use)

1. Creation/Update of Entities

1A) POST /v2/entities
1B) POST/PATCH/PUT /v2/entities/<id>

{
    "id": "ID",            # Mandatory for 1A, ERROR for 1B
    "type": "TYPE",        # Optional for 1A, ERROR for 1B
    "attr1": <value>,      # string, number, bool, null, json object (if value present), compound (vector or object)
    "attr2": {             # Not a compound as it contains "value"
        "value": <value>,
        "type": "type",
        "metadata": {
            "meta1": <value>,    # string, number, bool, null (compounds are not allowed)
            "meta2": "STRING" || 12 || 3.14 || true || false || null,
            "meta3": {           # Not ERROR as it contains "value" or "type"
                "type": "TYPE",
                "value": <value>,  # string, number, bool, null
            }
        },
        "anything_else_for_attribute": ERROR
    }
}



2. Rendering of Entity

Response from
* GET /v2/entities
* GET /v2/entities/<EID>

2A. Compact    (?options=keyvalue)
2B. Normalized (default)
2C. Value      (?options=value)


2A. Compact
{
  "id": "ID",
  "type": "TYPE" || null,
  "attr1": 14 || 3.14 || "STRING" || true || false || null || { compound-obj } || [ compound-vec ],
  "attr2": ...
}

2B. Normalized
{
  "id": "ID",
  "type": "TYPE" || null,
  "attr1": {
    "value": 14 || 3.14 || "STRING" || true || false || null || { compound-obj } || [ compound-vec ],
    "type": "TYPE" || null,
    "metadata": {} ||  {
      "m1": {
        "value": null || 14 || 3.14 || "STRING" || true || false,
        "type": null || "TYPE"
      },
      "m2": ...
    }
  }
}

2C. Value (not to be implemented in this issue - see separate issue #1049)
{
  "id": "ID",
  "type": "TYPE" || null,
  "attributes": [ 1, "STRING", 3.14, false, ... ]
}



3. Creation/Update of Attribute
 This is pretty much implemented already, done as part of *1. Creation/Update of Entities*.


4. Rendering of Attribute
    Like (3), this part 4 was implemented during *Part 2*


5. Update of Attribute Value

PUT /v2/entities/<entityId>/attrs/<attrName>/value

1. If Content-Type == application/json, the payload is a JSON Object or a JSON Array.
2. If Content-Type == text/plain, then the value of the attribute is found examining the
     textual payload (the payload is investigated "in order", first 1, if not, then 2, etc.):
  1. If the payload starts and ends with citation-marks, the value is taken as a string.
  2. If true/false/True/False/TRUE/FALSE, the value is taken as a Boolean.
  3. If null/Null/NULL, the value is taken as NULL.
  4. If these first three tests 'fails', the text is interpreted as a Number.
  5. If not a Number, then ERROR. The attribute's value is unchanged.


6. Rendering of Attribute Value

GET /v2/entities/<entityId>/attrs/<attrName>/value
No URI param option "text".

1. "Accept: application/json" + value is JSON Object/Array (compound), all ok,
   the response is the JSON rendering of the value.
2. "Accept: application/json" + value is NOT a compound:
   Error 406 Not Acceptable: accepted MIME types: text/plain
3. "Accept: text/plain" + value is JSON compound, 
    the response is a stringification of the JSON compound (this needs to be clarified)
    Same response as (1) ?  Response is a string? All quotation marks would have to be escaped ...
4.  "Accept: text/plain" + value is NOT a compound:
     the response is the textual rep of the Number/Bool/Null, etc.
     What about strings, should they be surrounded by quotes?



7. Selector to make the broker accept a different input payload for /v2/entities

7A: POST /v2/entities?options=keyValues
7B: POST/PATCH/PUT /v2/entities/<entityId>?options=keyValues

Format of the payload:

{
  "id": "entityId",    # Mandatory for 7A, ERROR for 7B
  "type": "TYPE",      # Optional for 7A, ERROR for 7B
  "attr1": <value1>,   # Any other item is treated as an attribute
  "attr2": <value2>
}

Using *options=keyValues*, attributes have no **type** nor **metadata**, just a value.
If the <value_x> is a JSON object, it is ALWAYS treated as a compound value.

[ If <value_x> is a JSON object, containing "value", "type" or "metadata", 
  the payload may be identical to the input in the case of options != keyValues,
  and probably the user has made a mistake.
  However, this will be clearly documented and should not cause any problems. ]

Exceptions:

• options=values not to be implemented as part of this issue.
• null values for attributes: depends on issue XXX, postponed until that issue is fixed.

[fgalan]

It has been proposed by @jmcanterafonseca to allow PUT/POST payload without attribute value. In these cases, the value of the attribute is implicit null (thus, the NGSI rule that makes value mandatory for an attribute doesn't break).

[fgalan]

In the case the parser detect a unknown field in the JSON, it should return a BadRequest error (see #1286) error which description message includes the unkwown field, e.g:

400 Bad Request 
{
  "error": "BadRequest, 
  "description": "unknown field in JSON payload request: foo"
}

[fgalan]

This has been worked in several task/ -> feature/1259_XX PRs. The final PR that merges features/1259_XX into develop is this one: #1685