Nodeos takes very long time to start

[hongshu7]

I upgraded the nodeos to 1.4.1 and ran it. But it took about 30 minutes for startup. Why so long? My system is CentOS 7.4, CPUx2, RAM 8G.

These are logs:

info 2018-10-18T17:33:30.807 thread-0 chain_plugin.cpp:333 plugin_initialize ] initializing chain plugin
info 2018-10-18T17:33:30.812 thread-0 block_log.cpp:134 open ] Log is nonempty
info 2018-10-18T17:33:30.812 thread-0 block_log.cpp:161 open ] Index is nonempty
info 2018-10-18T17:33:30.836 thread-0 net_plugin.cpp:2901 plugin_initialize ] Initialize net plugin
info 2018-10-18T17:33:30.836 thread-0 net_plugin.cpp:2927 plugin_initialize ] host: 0.0.0.0 port: 6688
info 2018-10-18T17:33:30.837 thread-0 net_plugin.cpp:2998 plugin_initialize ] my node_id is d91cae0be414716bb94bb858caf7a056bd822f9bb7ffe869c0af92f2daa9f65a
info 2018-10-18T17:33:30.837 thread-0 http_plugin.cpp:422 plugin_initialize ] configured http to listen on 0.0.0.0:80
info 2018-10-18T17:33:30.837 thread-0 http_plugin.cpp:460 plugin_initialize ] configured https to listen on 0.0.0.0:443 (TLS configuration will be validated momentarily)
warn 2018-10-18T17:33:30.837 thread-0 net_api_plugin.cpp:96 plugin_initialize ]

info 2018-10-18T17:33:30.837 thread-0 http_plugin.cpp:377 operator() ] configured http with Access-Control-Allow-Origin: *
info 2018-10-18T17:33:30.837 thread-0 main.cpp:110 main ] nodeos version v1.4.1
info 2018-10-18T17:33:30.837 thread-0 main.cpp:111 main ] eosio root is /root/.local/share

---

[ Nodeos was stuck here about 30 minutes ]

---

info 2018-10-18T17:33:30.837 thread-0 controller.cpp:389 init ] database initialized with hash: b0a8a50183a121074a9506423313ba5ac243baeb93b1e5dcb982c2c08ba76b6d
info 2018-10-18T18:02:40.436 thread-0 chain_plugin.cpp:707 plugin_startup ] starting chain in read/write mode
info 2018-10-18T18:02:40.519 thread-0 chain_plugin.cpp:711 plugin_startup ] Blockchain started; head block is #22264929, genesis timestamp is 2018-06-08T08:08:08.888
info 2018-10-18T18:02:40.519 thread-0 net_plugin.cpp:3017 plugin_startup ] starting listener, max clients is 25

[brianjohnson5972]

The 30 minutes is visible in the log between the following 2 lines:
info 2018-10-18T17:33:30.837 thread-0 controller.cpp:389 init ] database initialized with hash: b0a8a50183a121074a9506423313ba5ac243baeb93b1e5dcb982c2c08ba76b6d
info 2018-10-18T18:02:40.436 thread-0 chain_plugin.cpp:707 plugin_startup ] starting chain in read/write mode
but in the code there is either nothing it processes between (if you didn't use a snapshot) or just the closing of files and deleting the file stream and snapshot (if a snapshot was used), so I am not sure what could have caused that massive delay.
Were you using a snapshot? Also, were you running this in docker, a virtual machine, or directly on your machine? Did you look at what percentage of processing cpus were being used for nodeos during that 30 minutes? Have you seen this just the one time, or have you seen it more than once.

[hongshu7]

I used the data directory that copied from other vps, and ran it in docker, cpu used 3%. it happened every time, also happened in my different vps.

[heifner]

Resolved by #6164