Distributed mode not working with concurrent calls.

[gtadudeps]

OrientDB Version: v3.0.4

Java Version: 8

OS: macOS

Expected behavior

Blueprints implementation of orient graph should be able to connect to cluster and create vertices and edges on the graph concurrently (In our case there are 16 threads each with its own graph instance modifying the graph) . Client app is handling ONeedRetryException and all the server/client side configuration is unmodified. Moreover, it works fine with OrientDB v2.2.35.

Actual behavior

Three server nodes are able to join cluster and talk to each other (sync databases) but client throws a exception while creating graph.

Throws a exception

Caused by: com.orientechnologies.orient.server.distributed.task.ODistributedRecordLockedException: Timeout (0ms) on acquiring lock on record #-1:-1 on server 'DeadLock'. It is locked by request 2.1301
	DB name="apmnew1"
	DB name="apmnew1"
	at sun.reflect.GeneratedConstructorAccessor67.newInstance(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_152]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_152]
	at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.throwSerializedException(OChannelBinaryAsynchClient.java:318) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.handleStatus(OChannelBinaryAsynchClient.java:275) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:191) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.beginResponse(OChannelBinaryAsynchClient.java:153) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.remote.OStorageRemote.beginResponse(OStorageRemote.java:1812) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.remote.OStorageRemote.lambda$networkOperationRetryTimeout$2(OStorageRemote.java:226) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.remote.OStorageRemote.baseNetworkOperation(OStorageRemote.java:285) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationRetryTimeout(OStorageRemote.java:214) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.remote.OStorageRemote.networkOperationNoRetry(OStorageRemote.java:239) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.client.remote.OStorageRemote.commit(OStorageRemote.java:997) ~[orientdb-client-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.internalCommit(ODatabaseDocumentAbstract.java:2744) ~[orientdb-core-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.doCommit(OTransactionOptimistic.java:534) ~[orientdb-core-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.core.tx.OTransactionOptimistic.commit(OTransactionOptimistic.java:100) ~[orientdb-core-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2232) ~[orientdb-core-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentAbstract.commit(ODatabaseDocumentAbstract.java:2202) ~[orientdb-core-3.0.4.jar:3.0.4]
	at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.commit(ODatabaseDocumentTx.java:734) ~[orientdb-core-3.0.4.jar:3.0.4]
	at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.commit(OrientTransactionalGraph.java:184) ~[orientdb-graphdb-3.0.4.jar:3.0.4]

Steps to reproduce

1. Create a cluster of three server nodes with default hazelcast and default-distributed-db-config
2. Run 3 server nodes as master
3. Connect to cluster through blueprints graph API.
4. Make concurrent creation/update calls to build graph.

[cloudloveseven]

hi, @gtadudeps ,I have no idea for your question,but another question for you ,I do the same thing ,I create a distribute cluster of three server nodes with default hazelcast and default-distributed-db-config，but several nodes shut down and never recover ,do you know why ?

[gtadudeps]

@cloudloveseven what error do see in the logs of servers which go down?