Issue loading vertices

[danciui]

OrientDB Version: orientdb-community-2.2.16

Java Version: Java 8 Update 91

OS: Mac Os Sierra 10.12.3

Expected behavior

Vertex has all the info.

Actual behavior

I'm using oetl.sh with JSON to load data on an OrientDB remote server. For some reason oetl.sh sees an edge that doesn't have the source/destination yet and adds the missing vertex. When it actually encounters the missing vertex in the json file it just skips it. So I am missing the attributes for that vertex.

Steps to reproduce

This happens only sometimes, not always.
Here is a snippet of the debug.

[444:merge] DEBUG Transformer input: {inherits:[0],kind:ComponentAssembly,name:DesignContainer_Full_Earm_cfg2,contains:[142],connects:[0],references:[0],edgeProperties:[0],guid:{338e3a27-7a66-492d-8fce-d356c35c28b6},type:model,id:id-0065-00000420}
[444:merge] DEBUG joinValue=id-0065-00000420, lookupResult=null
[444:merge] DEBUG unresolved link!!! 0
[444:merge] DEBUG Transformer output: {inherits:[0],kind:ComponentAssembly,name:DesignContainer_Full_Earm_cfg2,contains:[142],connects:[0],references:[0],edgeProperties:[0],guid:{338e3a27-7a66-492d-8fce-d356c35c28b6},type:model,id:id-0065-00000420}
[444:vertex] DEBUG Transformer input: {inherits:[0],kind:ComponentAssembly,name:DesignContainer_Full_Earm_cfg2,contains:[142],connects:[0],references:[0],edgeProperties:[0],guid:{338e3a27-7a66-492d-8fce-d356c35c28b6},type:model,id:id-0065-00000420}
[444:vertex] DEBUG Transformer output: v(Entity)[#18:198]
[444:edge] DEBUG Transformer input: v(Entity)[#18:198]
[444:edge] DEBUG joinCurrentValue=id-0065-000004f2, lookupResult=null
[444:edge] DEBUG created new vertex=Entity#19:198{id:id-0065-000004f2} v1
[444:edge] DEBUG created new edge=e[#23:43][#18:198-Containment->#19:198]
[444:edge] DEBUG joinCurrentValue=id-0065-000004f5, lookupResult=null
[444:edge] DEBUG created new vertex=Entity#20:198{id:id-0065-000004f5} v1
[444:edge] DEBUG created new edge=e[#24:43][#18:198-Containment->#20:198]
[444:edge] DEBUG joinCurrentValue=id-0065-000004f8, lookupResult=null
[444:edge] DEBUG created new vertex=Entity#17:199{id:id-0065-000004f8} v1
[444:edge] DEBUG created new edge=e[#21:44][#18:198-Containment->#17:199]
[444:edge] DEBUG joinCurrentValue=id-0065-000004fb, lookupResult=null
[444:edge] DEBUG created new vertex=Entity#18:199{id:id-0065-000004fb} v1
[444:edge] DEBUG created new edge=e[#22:44][#18:198-Containment->#18:199]
[444:edge] DEBUG joinCurrentValue=id-0065-000004fe, lookupResult=null
[444:edge] DEBUG created new vertex=Entity#19:199{id:id-0065-000004fe} v1
[444:edge] DEBUG created new edge=e[#23:44][#18:198-Containment->#19:199]
[444:edge] DEBUG joinCurrentValue=id-0065-00000501, lookupResult=null
[444:edge] DEBUG created new vertex=Entity#20:199{id:id-0065-00000501} v1
[444:edge] DEBUG created new edge=e[#24:44][#18:198-Containment->#20:199]

• extracted 946 entries (16 entries/sec) - 946 entries -> loaded 444 vertices (16 vertices/sec) Total time: 16036ms [0 warnings, 0 errors]

In the json file i am trying to load
{"inherits": [], "kind": "ComponentAssembly", "name": "inhibit # C8 + BAR/BAX <--> BAR/BAX:C8 __inhibit", "contains": ["id-0065-000004f3", "id-0065-000004f4", "id-0066-000001b9", "id-0066-000001ba", "id-0066-000001bb", "id-0067-000007d7", "id-0067-000007d8"], "connects": [], "references": [], "edgeProperties": {}, "guid": "{c4078e7a-a99c-431f-8a0f-b9e0b465da70}", "type": "model", "id": "id-0065-000004f2"}

But in the resulting orientdb database id-0065-000004f2 is missing kind, name etc.

[danciui]

Hi, Do you know how much longer this will take to fix? oetl.sh is unpredictable without this fix.

[danciui]

the query to get vertices that are mis-inserted: "select from v where kind is null"
Archive.zip

[robfrank]

All is woking fine here, but I switched off parallel execution AND I used plocal

odb-frank:7191 frank$ ~/local/orientdb/orientdb-community-2.2.17/bin/console.sh

OrientDB console v.2.2.17-SNAPSHOT (build UNKNOWN@re7562eaab24ebfbebf10c2a819f660c3b584ac98; 2017-02-16 09:22:14+0000) www.orientdb.com
Type 'help' to display all the supported commands.
Installing extensions for GREMLIN language v.2.6.0

orientdb> connect plocal:..../database admin admin

Connecting to database [plocal:..../database] with user 'admin'...OK
orientdb {db=database}> select from v where kind is null


0 item(s) found. Query executed in 0.347 sec(s).
orientdb {db=database}>

[danciui]

Using 2.2.17:

From the oetl.sh log searching for id-0065-0000a29f:
"[111:merge] DEBUG Transformer input: {guid:{0bd6349f-37e9-4258-b3a8-8b3f4dd4159c},edgeProperties:[0],kind:ComponentAssembly,references:[0],connects:[0],attributes:[8],contains:[36],type:model,id:id-0065-0000a29c,name:DC_Spacecraft_V2_cfg2,inherits:[0]}
[111:merge] DEBUG joinValue=id-0065-0000a29c, lookupResult=null
[111:merge] DEBUG unresolved link!!! 0
[111:merge] DEBUG Transformer output: {guid:{0bd6349f-37e9-4258-b3a8-8b3f4dd4159c},edgeProperties:[0],kind:ComponentAssembly,references:[0],connects:[0],attributes:[8],contains:[36],type:model,id:id-0065-0000a29c,name:DC_Spacecraft_V2_cfg2,inherits:[0]}
[111:vertex] DEBUG Transformer input: {guid:{0bd6349f-37e9-4258-b3a8-8b3f4dd4159c},edgeProperties:[0],kind:ComponentAssembly,references:[0],connects:[0],attributes:[8],contains:[36],type:model,id:id-0065-0000a29c,name:DC_Spacecraft_V2_cfg2,inherits:[0]}
[111:vertex] DEBUG Transformer output: v(Entity)[#19:66]
[111:edge] DEBUG Transformer input: v(Entity)[#19:66]
[111:edge] DEBUG joinCurrentValue=id-0065-0000a29d, lookupResult=null
[111:edge] DEBUG created new vertex=Entity#20:66{id:id-0065-0000a29d} v1

• extracted 613 entries (1 entries/sec) - 613 entries -> loaded 111 vertices (1 vertices/sec) Total time: 19067ms [0 warnings, 0 errors]
  [111:edge] DEBUG created new edge=e[full text search support [moved] #21:20][LocalCreateObjectSpeedTest is very very slow [moved] #19:66-Containment->error when run speed test in remote model [moved] #20:66]
  [111:edge] DEBUG joinCurrentValue=id-0065-0000a29f, lookupResult=null
  [111:edge] DEBUG created new vertex=Entity#17:67{id:id-0065-0000a29f} v1
  [111:edge] DEBUG created new edge=e[Native Query update? [moved] #22:20][LocalCreateObjectSpeedTest is very very slow [moved] #19:66-Containment->Distributed version of Key/Value Server [moved] #17:67]
  [111:edge] DEBUG joinCurrentValue=id-0065-0000a2a1, lookupResult=null
  "

[robfrank]

I was able to reproduce the issue, don't know why it happens yet.
In the meanwhile, my suggestion is to bulk load using plocal.

[danciui]

I am using pyorient atm ( also due to the fact that I cannot figure out how to configure the loader template to add different properties on different edges. e.g. edgeProperties": {"id-0067-000001bf": "Built", "id-0067-00000462": "Built"} So I am adding that dictionary as a property on every Connection edge with src. e.g. The [id-0067-00000456, id-0067-000001bf] edge has the {"id-0067-000001bf": "Built", "id-0067-00000462": "Built"} edge property, and the [id-0067-00000456, id-0067-00000462] has the same {"id-0067-000001bf": "Built", "id-0067-00000462": "Built"} edge property. Instead I would want the [id-0067-00000456, id-0067-000001bf] edge to have the "Build" edge property, and the [id-0067-00000456, id-0067-00000462] to have the "Build" edge property. I will submit that as a new request).

[robfrank]

Hi @danciui , I found a solution to let the configuration work on remote.
I simply add a unique index on entity id

  "classes": [
        {
          "name": "Entity",
          "extends": "V"
        },
        {
          "name": "Containment",
          "extends": "E"
        },
        {
          "name": "Connection",
          "extends": "E"
        },
        {
          "name": "Inheritance",
          "extends": "E"
        }
      ],
      "indexes": [
        {"class":"Entity", "fields":["id:string"], "type":"UNIQUE" }
      ]

The other good side-effect is a great speedup of the import .

Let me know if it works

[danciui]

Thank you. That workaround worked. This issue is still a bug though and will need to be fixed.

[danciui]

The workaround worked for that file. Here's an example where it doesn't work but maybe for a different reason. id-0067-0000008b does not have kind. Maybe due to #7252
Archive 3.zip