[BUG] when is exceeded the index key limit in mongo

[iariasleon]

the index key limit in mongo is 1024, see issue #1283

if type or id exceed index key limit ,CB responds 201 and nothing is stored in mongo.

Request:

 POST http://qa-orion-fe-01:1026/v2/entities
  Content-Type: application/json
  Fiware-Service: test_entities_type
  Fiware-ServicePath: /test
  {"type": "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04WiCUekxiLxgHM6ToydSt8XZ2yDdYcXpXBSVOx66bIi8CoImJAqLYc39apAF77hI1Y1RPksM1S8z9d5sVp0IxhnFN27Al72XXQDqwSTW7YUZyJJeXzM9OI4lQY0CnZLignlNqxiW8Hr4SfZrqWhailwIZ9Q6FyWfjhcSrWK2RF89kVkGIt9DuO7yaGQ5ONCessB2f8rab0CVvVC6WGafUtvDJs8raCQfEzMnLZ2PW6jDMFH1RXPAoxRN78Ho1QLgJB4YVqUWjgB4ZMQCWFQbaG0fSMAImLyzwqjSAoIbobooQzKjaL6aZzIYV1HlVsZSzrnlz68cHD62ZQxIjxsTjPjk2jlXAPCWi23KO4XOB3x9NrzYN2Eh0aN4VDCHfnFQWj4LkqoF2LbZLYonLLrty9yLRqUD1Sq4dpHO52HTWI6dh9LZI0oT4spnPHNmlnsifjatn3e6hmH6gFiXiyXKdndptaHUVaI8GwDr6zeQH0GYlfw4Ho46iaECauPbpgD9YONA0kAl8BFZpL83yz3pJyhRNK4vCWHS676PBEGr8GXBkwTScge5f2pACnvfNmHbrPtm1FHuZ6ZRH318hBt7p98XSF757lz3ML2ixoecLBORRTBgHto95feclNIyrbRgZ9LiOfQrifoagfhGh04eLN6dPCt3mXDmirYR6kU2xUGOmGug4QM1OKUyWUK8vPCqfG4iu4Klt9Pv2s1ttPHmloeEvZ580b7AODyxQqWhHbVJ0OmAJ02h14ONdkvGKDBBzcmh5cFiTDMZDRMFU88NnTQJkFoEBjtz6OYkRdR2U1ctpPXyYkFP0xednTNl6cC9oRz7vXiOXFDozYFPV3lZqeenIsPlQhFpqim11DxTdEbqM", "temperature": "34", "id": "room_1"}

Response

http code: 201
 content-length: 0

Log

time=2015-09-25T08:56:09.696CEST | lvl=INFO | trans=1443164167-603-00000000003 | function=processContextElement | comp=Orion | msg=MongoCommonUpdate.cpp[2328]: Database Operation Successful ({ _id.id: "room_1", _id.type: "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04Wi...", _id.servicePath: /^\/test$/ })
time=2015-09-25T08:56:09.696CEST | lvl=INFO | trans=1443164167-603-00000000003 | function=processContextElement | comp=Orion | msg=MongoCommonUpdate.cpp[2388]: Database Operation Successful ({ _id.id: "room_1", _id.type: "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04Wi...", _id.servicePath: /^\/test$/ })
time=2015-09-25T08:56:09.698CEST | lvl=ERROR | trans=1443164167-603-00000000003 | function=createEntity | comp=Orion | msg=MongoCommonUpdate.cpp[1993]: Database Error (Database Error: collection: orion.entities - insert(): { _id: { id: "room_1", type: "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04Wi...", servicePath: "/test" }, attrNames: [ "temperature" ], attrs: { temperature: { type: "", creDate: 1443164169, modDate: 1443164169, value: "34" } }, creDate: 1443164169, modDate: 1443164169 } - exception: OperationException: { index: 0, code: 17280, errmsg: "insertDocument :: caused by :: 17280 Btree::insert: key too large to index, failing orion.entities.$_id_ 1086 { : { id: "room_1", type: "TZQIpnz4YX7Ng...", op: { _id: { id: "room_1", type: "TZQIpnz4YX7NgOo2DCSJ6llpviu6mG2Xc2H7MqPaJy9fGRtUcjg1UmguN39pwSKkFJpIJJ1n4RGwjJBkk5iGDl7OY1RW5dZRKQ6hKCSSufXZqm02ECdoqQWnjR6WZ5PGJMRpSmnrVbdZMKfXkJ04Wi...", servicePath: "/test" }, attrNames: [ "temperature" ], attrs: { temperature: { type: "", creDate: 1443164169, modDate: 1443164169, value: "34" } }, creDate: 1443164169, modDate: 1443164169 } })

[kzangeli]

Interesting ...
We'll have to impose some restriction and return an error.
strlen(id) + strlen(type) + some-constant > 1024 => ERROR?

[iariasleon]

service path should also be included

[fortizc]

What if the restriction for 1024 is implemented in the mongoUpdateContext.cpp ?

[fgalan]

What if the restriction for 1024 is implemented in the mongoUpdateContext.cpp ?

Pros:

• Given that the problem is related with a DB limitation, probably is the layer where that check belongs.

Cons:

• mongoBackend operations run protected within a take/give sempahore area (except if -reqMutexPolicy is used), i.e. only one thread runs inside the mongoBackend. Thus, it is better to put checks that don't depend on DB results (as the ones that depend only of the request input, as this one) outside mongoBackend.

[fortizc]

Ok, now I check mongoCommonUpdate.cpp in line 2027, the error is captured here in createEntity function. I can use it?

See https://github.com/telefonicaid/fiware-orion/blob/develop/src/lib/mongoBackend/MongoCommonUpdate.cpp#L2027

[fgalan]

Ok, now I check mongoCommonUpdate.cpp in line 2027, the error is captured here in createEntity function. I can use it?

It seem that the code has "moved" in develop branch (is no longer pointing inside createEntity method)... so I'm not sure of the reference.

Hint: to get "inmutable" references, use a commits in the develop branch instead of the develop branch itself (which point to HEAD commit), e.g.

fiware-orion/src/lib/mongoBackend/MongoCommonUpdate.cpp

Line 2027 in 954f812

/* First CPr lookup (in the case some CER is not found): looking in E-A registrations */

[fortizc]

I pull the latest commit, compile and test with the case posted by iariasleon and surprise! it works... can anybody confirm this?

[iariasleon]

@fortizc
What http code is returned?
What are the error and the description messages expected ?

[fortizc]

@iariasleon
Sorry but the last days I'm not working in this bug, but I will return from tomorrow

[fortizc]

@kzangeli what should be the constant value?

[fgalan]

Actually, the formula should be:

strlen(id) + strlen(type) + strlen(servicePath) + some-constant > 1024 => ERROR

Not sure about "some-constant". I guess that it depends on how much additional space it used in the embedded document identified by _id (i.e. {id: ..., type: ..., servicePath: ...}) appart from the 3 subfields themselves.

@iariasleon , have your test bring some light about which value "some-constant" could take?

[fgalan]

Just it case it may help: http://stackoverflow.com/questions/33377441/mongodb-check-length-when-using-embedded-documents-for-id-index

[fgalan]

While we get the right answer, in order not be blocked, I'd suggest to assume a reasonable large value for "some-constant". Maybe 100 is a good heuristic (100 is around 10% of max lenght 1024).

[iariasleon]

I think, first is to define the behaviour, see #1283
and second the max length is 1024 bytes, no chars.

[fgalan]

Just to clarify things:

#1283 is related with the lengths of the indivitual fields id, type and servicePath. Unfortunatelly, we don't have such information (the decission is a functional requirement and the development team -i.e. us- don't have the ownership to define such requirement).

Good news are that we don't need solving #1283 to solve this (#1289) issue. Otherwise, we would be blocked! This (#1289) only needs to define a limit in the sum of the field. The limit should be:

strlen(id) + strlen(type) + strlen(servicePath) + C < 1024

We don't know the value of C (question at http://stackoverflow.com/questions/33377441/mongodb-check-length-when-using-embedded-documents-for-id-index is about trying to find it). In abcense of an answer, we have to do some guessing. I think that C = 100 is a good guessing, but if you have another suggestion, please go ahead.

If after reading this explantion is not 100% clear how this (#1289) has to be solved, please contact me at skype.

[fgalan]

According to #1286 "rules" for NGSIv2 errors, I'd say that this case should be:

400 - Bad Request {"error": "BadRequest", "description": "Too long entity id/type/servicePath combination" }

Don't forget to include a .test to cover the case! ;)

[fortizc]

Pull request sent

[fgalan]

Fixed in PR #1441. @iariasleon please have a look and close it if you find correct.

[iariasleon]

LGTM

http code: 400
{
     "error":"BadRequest",
     "description":"Too long entity id/type/servicePath combination"
}

[mauze-muawwaz-elastica]

How do I figure out programatically (if possible) that the insertion I'm going to perform will throw this error?
In my case, there is an indexed field on a collection which doesn't allow me to save a value greater than 1007. The documentation says 1024, where are those extra 17 bytes used?
Is there any formula which can help me identify this prior to insertion?

[fgalan]

The "formula" being used should be in the code. @kzangeli , @fortizc , could you help to locate it an add it as comment to this issue, please?

(Reopening issue while the answer is in progress)

[kzangeli]

In postEntities.cpp, the function legalEntityLength determines if the total is too long:

static bool legalEntityLength(Entity* eP, const std::string& servicePath)
{
  return (servicePath.size() +
          eP->id.size()      +
          eP->type.size()    +
          STRUCTURAL_OVERHEAD_BSON_ID) < 1024;
}

// servicePath.size:    the string-length of the service path
// eP->id.size() :      the string-length of the entity id
// eP->type.size();     the string-length of the entity type
//
// STRUCTURAL_OVERHEAD_BSON_ID is 10
//
// Now, if the sum of these 4 sizes is less than 1024, then all is OK

[mauze-muawwaz-elastica]

Thanks.

[fgalan]

Maybe we should put the rule in the documentation (although I'm not sure in which section...)

[fgalan]

At the end "Known limitations" seems to be a good candidate section in the documentation. Fixed in PR #1899