Multithreaded reads not possible with OrientGraph because of java.nio.BufferUnderflowException?

[Spinoza42]

Version 1.6. Community Edition

After two days of trial and error i think the behavior is not correct, because i cannot get any environment running without the error java.nio.BufferUnderflowException in OByteBufferUtils. Only single threaded code runs without problems and one single run of the example with 10 threads finished without the exception.

Maybe i do something wrong, but this exception gives me no hint what it is. See example and output below.

Expected: Read records from OrientGraph with multiple threads from the same database.

This should not be that hard i thought, but even the simplest query (getVertex with fix RecordID) runs into the exception.

According to the documentation i created a separate OrientGraph instance per thread. So i expect the example should be thread safe.

The example which crashes:

public static void main(String[] args) {

    Thread[] threads = new Thread[10];

    for (int i = 0; i < threads.length; i++) {
        Thread thread = new Thread(new Runnable() {
            @Override
            public void run() {
                OrientGraph graph = null;
                try {
                    graph = new OrientGraph(
                            "local:/xxx/yyy/gremlin_orientdb",
                            "admin", "admin");
                    Vertex v = graph.getVertex("#11:1"); // Simplest form of query, same exception happens with gremlin query too. (This will be the real use case.)
                    System.out.println("Thread:" + Thread.currentThread() + " " + v.toString());
                } finally {
                    graph.getRawGraph().close(); // I am not sure if this is necessary but i read it in the documentation. It makes no difference at all, the exception occurs with and without this line.
                }
            }
        });
        threads[i] = thread;
    }

    for (int i = 0; i < threads.length; i++) {
        threads[i].start();
    }

}

My output:

Thread:Thread[Thread-4,5,main] v(Products)[#11:1]
Thread:Thread[Thread-8,5,main] v(Products)[#11:1]
Thread:Thread[Thread-5,5,main] v(Products)[#11:1]
Exception in thread "Thread-9" Thread:Thread[Thread-3,5,main] v(Products)[#11:1]
Exception in thread "Thread-0" com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #11:1 (cluster: products)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:249)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:706)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:258)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:202)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:38)
at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:296)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.getVertex(OrientBaseGraph.java:429)
at com.orientdbtest.BugTest$1.run(BugTest.java:21)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:472)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:219)
at com.orientechnologies.common.util.OByteBufferUtils.mergeShortFromBuffers(OByteBufferUtils.java:63)
at com.orientechnologies.orient.core.storage.fs.OFileMMap.readShort(OFileMMap.java:225)
at com.orientechnologies.orient.core.storage.impl.local.OClusterLocal.getPhysicalPosition(OClusterLocal.java:316)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.readRecord(OStorageLocal.java:1764)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.readRecord(OStorageLocal.java:1127)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:243)
... 8 more
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #11:1 (cluster: products)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:249)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:706)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:258)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:202)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:38)
at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:296)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.getVertex(OrientBaseGraph.java:429)
at com.orientdbtest.BugTest$1.run(BugTest.java:21)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:472)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:219)
at com.orientechnologies.common.util.OByteBufferUtils.mergeShortFromBuffers(OByteBufferUtils.java:63)
at com.orientechnologies.orient.core.storage.fs.OFileMMap.readShort(OFileMMap.java:225)
at com.orientechnologies.orient.core.storage.impl.local.OClusterLocal.getPhysicalPosition(OClusterLocal.java:316)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.readRecord(OStorageLocal.java:1764)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.readRecord(OStorageLocal.java:1127)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:243)
... 8 more
Exception in thread "Thread-7" com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #11:1 (cluster: products)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:249)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordAbstract.executeReadRecord(ODatabaseRecordAbstract.java:706)
at com.orientechnologies.orient.core.tx.OTransactionOptimistic.loadRecord(OTransactionOptimistic.java:258)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:202)
at com.orientechnologies.orient.core.db.record.ODatabaseRecordTx.load(ODatabaseRecordTx.java:38)
at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:296)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.getVertex(OrientBaseGraph.java:429)
at com.orientdbtest.BugTest$1.run(BugTest.java:21)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.BufferUnderflowException
at java.nio.Buffer.nextGetIndex(Buffer.java:472)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:219)
at com.orientechnologies.common.util.OByteBufferUtils.mergeShortFromBuffers(OByteBufferUtils.java:63)
at com.orientechnologies.orient.core.storage.fs.OFileMMap.readShort(OFileMMap.java:225)
at com.orientechnologies.orient.core.storage.impl.local.OClusterLocal.getPhysicalPosition(OClusterLocal.java:316)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.readRecord(OStorageLocal.java:1764)
at com.orientechnologies.orient.core.storage.impl.local.OStorageLocal.readRecord(OStorageLocal.java:1127)
at com.orientechnologies.orient.core.db.raw.ODatabaseRaw.read(ODatabaseRaw.java:243)
... 8 more
Thread:Thread[Thread-2,5,main] v(Products)[#11:1]
Thread:Thread[Thread-6,5,main] v(Products)[#11:1]
Thread:Thread[Thread-1,5,main] v(Products)[#11:1]

[lvca]

@Laa may you look at this issue please?

[lvca]

Sorry I've just got that you're using "local". Can you try the same code but using plocal?

graph = new OrientGraph( "plocal:/xxx/yyy/gremlin_orientdb", "admin", "admin");

Remember to delete the database folder before to run it.

[Spinoza42]

Thanks Luca, it seems this was the problem. With plocal the exception does not occur anymore. I tried it with 100 threads without errors.

For me it is good enough, but local is not thread safe anyways. Is "local" deprecated?

Maybe the wiki should be changed to plocal, because this wiki page below contains "local" used as protocol with hints to the proper usage of multi threading:
https://github.com/orientechnologies/orientdb/wiki/Graph-Database-Tinkerpop#work-with-graphdb

[lvca]

The new engine is "plocal". "Local" will be kept for a while. You're right, that page had the old "local. Changed it, thanks.

[nishantkumar1292]

Hi @lvca ..
I am getting the same exception when trying to execute the query SELECT out('category_listing_or').listing_id FROM #13:2. I am running this query through OrientDB studio.

Server Log:

Exception `13F66E65` in storage `db_development`
com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #12:85340 (cluster: listing_or)
        DB name="db_development"
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2050)
        at com.orientechnologies.orient.core.tx.OTransactionNoTx.loadRecord(OTransactionNoTx.java:106)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:1729)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.load(ODatabaseDocumentTx.java:102)
        at com.orientechnologies.orient.core.id.ORecordId.getRecord(ORecordId.java:329)
        at com.orientechnologies.orient.core.record.impl.ODocumentHelper.getIdentifiableValue(ODocumentHelper.java:808)
        at com.orientechnologies.orient.core.record.impl.ODocumentHelper.getFieldValue(ODocumentHelper.java:540)
        at com.orientechnologies.orient.core.record.impl.ODocumentHelper.getFieldValue(ODocumentHelper.java:222)
        at com.orientechnologies.orient.core.sql.method.misc.OSQLMethodField.execute(OSQLMethodField.java:68)
        at com.orientechnologies.orient.core.sql.method.OSQLMethodRuntime.execute(OSQLMethodRuntime.java:139)
        at com.orientechnologies.orient.core.sql.filter.OSQLFilterItemAbstract.transformValue(OSQLFilterItemAbstract.java:140)
        at com.orientechnologies.orient.core.sql.functions.OSQLFunctionRuntime.execute(OSQLFunctionRuntime.java:134)
        at com.orientechnologies.orient.core.sql.ORuntimeResult.applyRecord(ORuntimeResult.java:141)
        at com.orientechnologies.orient.core.sql.ORuntimeResult.getProjectionResult(ORuntimeResult.java:255)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.addResult(OCommandExecutorSQLSelect.java:703)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.handleResult(OCommandExecutorSQLSelect.java:669)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearchRecord(OCommandExecutorSQLSelect.java:626)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1615)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1562)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:521)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:484)
        at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:74)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3200)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3138)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.command(ODistributedStorage.java:280)
        at com.orientechnologies.orient.core.command.OCommandRequestTextAbstract.execute(OCommandRequestTextAbstract.java:69)
        at com.orientechnologies.orient.server.network.protocol.http.command.post.OServerCommandPostCommand.execute(OServerCommandPostCommand.java:106)
        at com.orientechnologies.orient.graph.server.command.OServerCommandPostCommandGraph.execute(OServerCommandPostCommandGraph.java:37)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.service(ONetworkProtocolHttpAbstract.java:169)
        at com.orientechnologies.orient.server.network.protocol.http.ONetworkProtocolHttpAbstract.execute(ONetworkProtocolHttpAbstract.java:621)
        at com.orientechnologies.common.thread.OSoftThread.run(OSoftThread.java:77)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot route read record operation for #12:85340 to the distributed node
        DB name="db_development"
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.handleDistributedException(ODistributedStorage.java:1977)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.readRecord(ODistributedStorage.java:791)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx$SimpleRecordReader.readRecord(ODatabaseDocumentTx.java:3361)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeReadRecord(ODatabaseDocumentTx.java:2012)
        ... 30 more
Caused by: java.nio.BufferUnderflowException
        at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:271)
        at java.nio.ByteBuffer.get(ByteBuffer.java:715)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurablePage.getBinaryValue(ODurablePage.java:138)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.OClusterPage.getRecordBinaryValue(OClusterPage.java:377)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readFullEntry(OPaginatedCluster.java:1834)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecordBuffer(OPaginatedCluster.java:704)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:667)
        at com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster.readRecord(OPaginatedCluster.java:646)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doReadRecord(OAbstractPaginatedStorage.java:3980)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:3595)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.readRecord(OAbstractPaginatedStorage.java:1250)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage$6.call(ODistributedStorage.java:762)
        at com.orientechnologies.orient.core.db.OScenarioThreadLocal.executeAsDistributed(OScenarioThreadLocal.java:70)
        at com.orientechnologies.orient.server.distributed.impl.ODistributedStorage.readRecord(ODistributedStorage.java:759)
        ... 32 more

[andrii0lomakin]

Hi @nishantkumar1292 what version do you use?
Under what version database was created? Did you experience server crash or something like that?

[nishantkumar1292]

Hi @Laa
Version: 2.2.21
Database created under 2.2.17. Sometimes the server was force closed.
The database was working fine only since yesterday we started getting this error.

[andrii0lomakin]

@nishantkumar1292 I suppose page corruption was caused by force close. Do you use WAL when you work with a server? About solution for your problem, you should make export/import database from to JSON. What was the reason of force close? Could you explain?

[nishantkumar1292]

Will export/import rectify the Error: java.nio.BufferUnderflowException
No I don't use WAL. By the way, what is WAL. Not familiar with it.
Sometimes the queries used to hang or the distributed nodes had a problem syncing. Closing and reopening the database would sometimes fix this.

Also tried to run REPAIR DATABASE --fix-graph from console.
Server log:

orientdb {db=db_development}> REPAIR DATABASE --fix-graph
Repair of graph 'remote:/home/ubuntu/orientdb-community-2.2.21/databases/db_development' is started ...
Scanning 6190806 edges (skipEdges=0)...

Error: com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #12:15017 (cluster: listing_or)
        DB name="db_development"

Error: com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #12:15017 (cluster: listing_or)
        DB name="db_development"
        DB name="db_development"

Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot route read record operation for #12:15017 to the distributed node
        DB name="db_development"

Error: java.nio.BufferUnderflowException

orientdb {db=db_development}> select from #12:15017;

Error: com.orientechnologies.orient.core.exception.ODatabaseException: Error on retrieving record #12:15017 (cluster: listing_or)
        DB name="db_development"
        DB name="db_development"

Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot route read record operation for #12:15017 to the distributed node
        DB name="db_development"

Error: java.nio.BufferUnderflowException

@Laa Please help....this is now becoming a bottle neck for our project. We are not able to proceed further due to this error.

[andrii0lomakin]

export/import will restore your physical database struсture, so issues will be gone. I suppose you did not change any settings, I mean you did not switch off files explicitly. WAL is database operation journal. Do you see database restore procedure after database close?

[nishantkumar1292]

Should I remove the database after export and then import? Or directly perform the export/import one after another?
@Laa

[andrii0lomakin]

You export from old database and import to completely new database

[nishantkumar1292]

@Laa ..One more thing.
I have distributed server with 2 nodes.
Do I need to do this on both the nodes separately or just need to export/import on the master node?

[andrii0lomakin]

@nishantkumar1292 I suppose, I am not expert in distributed, you have to do this on a single node and then wait till it will be replicated on the second node.

[nishantkumar1292]

Hi @Laa Just tried importing the database but got an error.
Please help!

LOGS

orientdb {db=db_development}> IMPORT DATABASE /home/ubuntu/export/db_development.export.gz

Importing database DATABASE /home/ubuntu/export/db_development.export.gz...
Started import of database 'plocal:/home/ubuntu/data/databases/db_development' from /home/ubuntu/export/db_development.export.gz...
Non merge mode (-merge=false): removing all default non security classes
- Class E was removed.
- Class V was removed.
- Class ORestricted was removed.
- Class OTriggered was removed.
- Class OSchedule was removed.
- Class OSequence was removed.
- Class OFunction was removed.
Removed 7 classes.
Importing database info...OK
Importing clusters...
- Creating cluster 'internal'...OK, assigned id=0
- Creating cluster 'default'...OK, assigned id=3
- Creating cluster 'orole'...OK, assigned id=4
- Creating cluster 'ouser'...OK, assigned id=5
- Creating cluster 'ofunction'...OK, assigned id=6
- Creating cluster 'osequence'...OK, assigned id=7
- Creating cluster 'oschedule'...OK, assigned id=8
- Creating cluster 'v'...OK, assigned id=9
- Creating cluster 'e'...OK, assigned id=10
- Creating cluster 'brand_or'...OK, assigned id=11
- Creating cluster 'listing_or'...OK, assigned id=12
- Creating cluster 'category_or'...OK, assigned id=13
- Creating cluster 'super_category_or'...OK, assigned id=14
- Creating cluster 'attribute_class_or'...OK, assigned id=15
- Creating cluster 'attribute_key_or'...OK, assigned id=16
- Creating cluster 'product_attribute_value_or'...OK, assigned id=17
- Creating cluster 'brand_listing_or'...OK, assigned id=18
- Creating cluster 'super_category_brand_or'...OK, assigned id=19
- Creating cluster 'category_listing_or'...OK, assigned id=20
- Creating cluster 'category_category_or'...OK, assigned id=21
- Creating cluster 'category_attribute_class_or'...OK, assigned id=22
- Creating cluster 'attribute_class_key_or'...OK, assigned id=23
- Creating cluster 'attribute_key_value_or'...OK, assigned id=24
- Creating cluster 'listing_product_attribute_value_or'...OK, assigned id=25
- Creating cluster 'test'...OK, assigned id=26
- Creating cluster '_studio'...OK, assigned id=27
- Creating cluster 'osequence_0'...OK, assigned id=28
- Creating cluster 'super_category_or_0'...OK, assigned id=29
- Creating cluster 'category_or_0'...OK, assigned id=30
- Creating cluster 'oschedule_0'...OK, assigned id=31
- Creating cluster 'listing_or_0'...OK, assigned id=32
- Creating cluster 'ofunction_0'...OK, assigned id=33
- Creating cluster 'v_0'...OK, assigned id=34
- Creating cluster 'test_0'...OK, assigned id=35
- Creating cluster 'brand_listing_or_0'...OK, assigned id=36
- Creating cluster 'product_attribute_value_or_0'...OK, assigned id=37
- Creating cluster 'category_category_or_0'...OK, assigned id=38
- Creating cluster 'attribute_class_or_0'...OK, assigned id=39
- Creating cluster 'category_listing_or_0'...OK, assigned id=40
- Creating cluster 'category_attribute_class_or_0'...OK, assigned id=41
- Creating cluster 'super_category_brand_or_0'...OK, assigned id=42
- Creating cluster 'ouser_0'...OK, assigned id=43
- Creating cluster '_studio_0'...OK, assigned id=44
- Creating cluster 'attribute_key_value_or_0'...OK, assigned id=45
- Creating cluster 'e_0'...OK, assigned id=46
- Creating cluster 'orole_0'...OK, assigned id=47
- Creating cluster 'listing_product_attribute_value_or_0'...OK, assigned id=48
- Creating cluster 'attribute_class_key_or_0'...OK, assigned id=49
- Creating cluster 'brand_or_0'...OK, assigned id=50
- Creating cluster 'attribute_key_or_0'...OK, assigned id=51
Rebuilding indexes of truncated clusters ...
- Cluster content was updated: rebuilding index 'ORole.name'... Index ORole.name was successfully rebuilt.
- Cluster content was updated: rebuilding index 'OUser.name'... Index OUser.name was successfully rebuilt.
Done 2 indexes were rebuilt.
Done. Imported 50 clusters
Importing database schema...OK (27 classes)

Importing records...
- Imported 23,473 records into clusters: [brand_or, brand_or_0, internal, listing_or, listing_or_0, orole, orole_0, ouser, ouser_0]. Total records imported so far: 23,4
73 (4,694.60/sec)
- Imported 38,129 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 61,602 (7,625.80/sec)
- Imported 33,768 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 95,370 (6,753.60/sec)
- Imported 33,475 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 128,845 (6,695.00/sec)
- Imported 33,826 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 162,671 (6,765.20/sec)
- Imported 31,107 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 193,778 (6,221.40/sec)
- Imported 31,673 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 225,451 (6,334.60/sec)
- Imported 32,060 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 257,511 (6,412.00/sec)
- Imported 34,711 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 292,222 (6,942.20/sec)
- Imported 33,474 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 325,696 (6,694.80/sec)
- Imported 39,819 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 365,515 (7,963.80/sec)
- Imported 38,360 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 403,875 (7,672.00/sec)
- Imported 37,623 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 441,498 (7,524.60/sec)
- Imported 36,713 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 478,211 (7,342.60/sec)
- Imported 36,214 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 514,425 (7,242.80/sec)
- Imported 37,428 records into clusters: [listing_or, listing_or_0]. Total records imported so far: 551,853 (7,485.60/sec)Error on database import happened just before 
line 0, column 398493000
com.orientechnologies.orient.core.exception.ODatabaseException: Error during saving of record with rid #15:-1
        DB name="db_development"
        at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:203)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:2753)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.save(ODatabaseDocumentTx.java:102)
        at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1805)
        at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:1796)
        at com.orientechnologies.orient.core.record.impl.ODocument.save(ODocument.java:63)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecord(ODatabaseImport.java:1405)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importRecords(ODatabaseImport.java:1274)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:418)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:2175)
        at com.orientechnologies.orient.graph.console.OGremlinConsole.importDatabase(OGremlinConsole.java:210)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405)
        at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260)
        at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131)
        at com.orientechnologies.orient.graph.console.OGremlinConsole.main(OGremlinConsole.java:72)
Caused by: com.orientechnologies.orient.core.exception.ODatabaseException: Impossible to serialize invalid link #-1:-1
        DB name="db_development"
        at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.writeOptimizedLink(ORecordSerializerBinaryV0.java:867)
        at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.writeLinkCollection(ORecordSerializerBinaryV0.java:890)
        at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.serializeValue(ORecordSerializerBinaryV0.java:748)
        at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinaryV0.serialize(ORecordSerializerBinaryV0.java:385)
        at com.orientechnologies.orient.core.serialization.serializer.record.binary.ORecordSerializerBinary.toStream(ORecordSerializerBinary.java:99)
        at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:2381)
        at com.orientechnologies.orient.core.record.impl.ODocument.toStream(ODocument.java:664)
        at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.executeSaveRecord(ODatabaseDocumentTx.java:2178)
        at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveNew(OTransactionNoTx.java:241)
        at com.orientechnologies.orient.core.tx.OTransactionNoTx.saveRecord(OTransactionNoTx.java:171)
        ... 18 more

Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'db_development' from file: /home/ubuntu/export/
db_development.export.gz

Error: com.orientechnologies.orient.core.exception.ODatabaseException: Error during saving of record with rid #15:-1
        DB name="db_development"

Error: com.orientechnologies.orient.core.exception.ODatabaseException: Impossible to serialize invalid link #-1:-1
        DB name="db_development"

[andrii0lomakin]

@nishantkumar1292 Could you send me your JSON file?

[nishantkumar1292]

Hi .. @Laa
Could not share the database as the information in internal company data.
Is there a way to debug this error?

Had to remove the link because of privacy reasons.

[andrii0lomakin]

@nishantkumar1292 here is fixed db the only issue left is that index category_listing_or_idx indicates that there are some duplicated records. there are few of them but they have to be removed manually.

[nishantkumar1292]

@Laa ... I have the corrected file now. Please remove that file from your comment as it contains internal company data. We can share it privately if needed.
I'll look at the fixed DB.