[2.6][T4] - Sync from 0 does not work

[galperins4]

Expected Behavior

Sync from 0 on a new install with no issues

Current Behavior

Errors noted during sync starting around round 2483 and the blockchain stops and says 100% in sync at round 3200 and no more blocks are downloaded.

Possible Solution

Unknown

Steps to Reproduce (for bugs)

Sync ark database from 0. Errors that pop up are as follows:

4|ark-relay  | [2019-10-16 02:27:28.200] WARN : Downloaded block not accepted: {"id":"15051108891414705441","version":0,"timestamp":46113370,"height":133602,"reward":"200000000","previousBlock":"7760605456110665353","numberOfTransactions":0,"totalAmount":"0","totalFee":"0","payloadLength":0,"payloadHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","generatorPublicKey":"030c0d642b9b5337d8f9f257e9e460bdd9d15e0581391896336dd8acb24b5d47b4","blockSignature":"30450221008597b0b382833f354e3745af0a8fd9700fa1ee25d34912568f5e0f6134481fa10220411a9647dd12839ce7ca527832955cd79894bf705ce99c9498f1718afac85495"}
4|ark-relay  | [2019-10-16 02:27:28.202] WARN : Last downloaded block: {"id":"11486934052286949346","version":0,"timestamp":46106570,"height":132801,"reward":"200000000","previousBlock":"14484635651814001471","numberOfTransactions":0,"totalAmount":"0","totalFee":"0","payloadLength":0,"payloadHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","generatorPublicKey":"03d7a20b3d39b7526a5057a9b486f0200bc57543e69e5fa61d9ce0bdd7784162c3","blockSignature":"30450221008c9584ed1f97fac9acafc81f16040cc3480d47913c23a62a77f2efa0d70233490220525e637e363235d2f84a5c300061dec0853c9ad6c9d70fff9f22941590cb8171"}

4|ark-relay  | [2019-10-16 02:27:57.906] WARN : Downloaded block not accepted: {"id":"8244185148301536109","version":0,"timestamp":46381690,"height":166939,"reward":"200000000","previousBlock":"412889844469480445","numberOfTransactions":0,"totalAmount":"0","totalFee":"0","payloadLength":0,"payloadHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","generatorPublicKey":"022eedf9f1cdae0cfaae635fe415b6a8f1912bc89bc3880ec41135d62cbbebd3d3","blockSignature":"3045022100ed2eab889ed4d3830fb8a95e8e5b829a9db00b5d6d9880a6ae4afa4102925c5c02201cb87039a5912ab15d852e530c7e3e3a69a257d8d5810ae6f8f91109044f2ffa"}
4|ark-relay  | [2019-10-16 02:27:57.907] WARN : Last downloaded block: {"id":"17926770977447731644","version":0,"timestamp":46375154,"height":166138,"reward":"200000000","previousBlock":"4728090936092083041","numberOfTransactions":0,"totalAmount":"0","totalFee":"0","payloadLength":0,"payloadHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","generatorPublicKey":"03120f521f7025f76341a09112f88a6c072411c549e4bfa8c92946fcf1c57cdf1e","blockSignature":"3045022100b791b25c942107269ef8c61bea8bebee4cd0fbf75ed1707badabc9303cb581f002200914950edacb76bffef45baa204b06608accf8dcc34abba15e6b08f0ef528b20"}

Context

Sync from 0 should work with no errors otherwise new installs have to rely on snapshots which is not preferred compared to complete sync from 0.

Your Environment

• Version used: 2.6.0-next.1
• Operating System and version: Ubuntu 18.04

[ghost]

Thanks for opening this issue! A maintainer will review this in the next few days and explicitly select labels so you know what's going on.

If no reviewer appears after a week, a reminder will be sent out.

[boldninja]

@galperins4 I have tried to reproduce this on 2 servers:

1. Server was synced, dropped db and started to sync from 0

2. Created a new server, installed Core with installation script and started to sync from 0

In both (so far) there was no errors or stops in syncin and both are well past Round 3200. I'll let them both sync and let you know if they stop somewhere along the way.

In the meantime can you on that server try dropping DB and start again and see if it goes past that?

pm2 stop all
dropdb ark_devnet && createdb ark_devnet < replace ark_devnet with you actual DB name
And after start pm2 process again pm2 start all or one of the ark <process>:start commands.

[zillionn]

I was also able to sync from 0 on a brand new server.

[boldninja]

@galperins4 I didn't experience any issues syncing from 0 on my two nodes, both synced without errors or stops up to current height. If you experience this again or some new errors appear don't hesitate to re-open this issue.

[ghost]

This issue has been closed. If you wish to re-open it please provide additional information.

[vasild]

This issue is real, it just does not manifest every time and depends on network conditions.

[ghost]

This issue has now been re-opened. If applicable please provide additional information as requested by one of the reviewers.

[vasild]

Here is a possible scenario that could cause the described problem (there may be more bugs exhibiting the same symptom):

The anonymous function that downloads a chunk of blocks, which calls BeerCommunicator.downloadBlocks():
https://github.com/ArkEcosystem/core/blob/develop/packages/core-p2p/src/network-monitor.ts#L334
https://github.com/ArkEcosystem/core/blob/develop/packages/core-p2p/src/peer-communicator.ts#L20
that function will never throw an exception even if downloading the chunk fails. It will instead return an empty array.

As a result the produced array, where all chunks are concatenated, could have neighboring elements (blocks) with non-sequential heights. This would confuse the caller and cause exactly the symptoms described in this bug report - in the log we have last downloaded block with height 132801 and trying to apply block 133602. The gap is 801 - exactly 2 chunks of 400 blocks each failed downloading (the printout is off-by-one).

This will be fixed by #3092

[ghost]

This issue has been closed. If you wish to re-open it please provide additional information.