New witness node: Failed to push new block

[risyasin]

Instructions
Just start a witness node with empty database v.3.2.0 on Ubuntu 18.4 . Just wait a few hours.
Bug Description
I've started a single witness node on an empty database with default settings. Update seed nodes from http://status.bitsharesnodes.com/ After a few hours Daemon stops p2p traffic. Does nothing no I/O or no log. Last log given by daemon (on top)

248400ms th_a       db_management.cpp:71          reindex              ] Replaying blocks, starting at 16334374...
1248401ms th_a       db_block.cpp:209              _push_block          ] Failed to push new block:
10 assert_exception: Assert Exception
trx_idx.indices().get<by_trx_id>().find(trx.id()) == trx_idx.indices().get<by_trx_id>().end(): 
    {}
    th_a  db_block.cpp:639 _apply_transaction

    {"trx":{"ref_block_num":15908,"ref_block_prefix":795790010,"expiration":"2017-05-08T17:11:49","operations":[[2,{"fee":{"amount":725,"asset_id":"1.3.0"},"fee_paying_account":"1.2.155562","order":"1.7.4671801"
,"extensions":[]}],[1,{"fee":{"amount":7256,"asset_id":"1.3.0"},"seller":"1.2.155562","amount_to_sell":{"amount":7872036,"asset_id":"1.3.0"},"min_to_receive":{"amount":193214,"asset_id":"1.3.861"},"expiration":"
2100-01-01T00:00:00","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f21e78db391618509d782eb94d883917c195d779a08ebc337060b135bdbac931211f16a194271b23d40ffe364d4b3652d4b2fa08bacc35511afa68
c2bf119eef9"]}}
    th_a  db_block.cpp:702 _apply_transaction

    {"next_block.block_num()":16334374}
    th_a  db_block.cpp:616 _apply_block
1248402ms th_a       db_management.cpp:225         open                 ] 10 assert_exception: Assert Exception
trx_idx.indices().get<by_trx_id>().find(trx.id()) == trx_idx.indices().get<by_trx_id>().end(): 
    {}
    th_a  db_block.cpp:639 _apply_transaction

    {"trx":{"ref_block_num":15908,"ref_block_prefix":795790010,"expiration":"2017-05-08T17:11:49","operations":[[2,{"fee":{"amount":725,"asset_id":"1.3.0"},"fee_paying_account":"1.2.155562","order":"1.7.4671801"
,"extensions":[]}],[1,{"fee":{"amount":7256,"asset_id":"1.3.0"},"seller":"1.2.155562","amount_to_sell":{"amount":7872036,"asset_id":"1.3.0"},"min_to_receive":{"amount":193214,"asset_id":"1.3.861"},"expiration":"
2100-01-01T00:00:00","fill_or_kill":false,"extensions":[]}]],"extensions":[],"signatures":["1f21e78db391618509d782eb94d883917c195d779a08ebc337060b135bdbac931211f16a194271b23d40ffe364d4b3652d4b2fa08bacc35511afa68
c2bf119eef9"]}}
    th_a  db_block.cpp:702 _apply_transaction

Details here: https://i.imgur.com/SiuHJA1.png

Impacts
Sync stops. No P2P traffic.

Steps To Reproduce

• Clone repository
• Compile via commands given in documentation
  /usr/local/bin/witness_node --data-dir=/home/ubuntu/data/btsdata/trusted_node --rpc-endpoint=127.0.0.1:8090
  This bug can also be created, updating seed list then trying with --resync-blockchain

Expected Behavior
Sync normally

Host Environment
Amazon LightSail 16 GB RAM, 4 vCPUs, 320 GB SSD + additional 128GB disk
Linux node-bts-bitshares 4.15.0-1043-aws #45-Ubuntu SMP Mon Jun 24 14:07:03 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
libboost-all-dev 1.65.1.0ubuntu1 amd64
openssl 1.1.1-1ubuntu2.1~18.04.3 amd64
Bitshares-core checkout at 3.2.0 tag

Impacts
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

[abitmore]

The log indicates that the node failed to (partially) replay from a local cache of object database, probably related to #946, which means the local cache is corrupted or unclean. Restarting witness_node with --replay-blockchain option would fix the local cache.

It seems that the node tried to push the same block twice. I'm not sure if it's related to #1679 or #1703.

[risyasin]

I can confirm that this bug no longer exist after 3.2.1 patch. I could able to replay-blockhain as well. Now my witness node works as expected. Thank you. If you need further investigation, let me know I can re-open this issue.