Not producing block because node didn't wake up within 500ms of the slot time.

[abitmore]

Log:

2018-07-18T18:58:27 th_a:invoke handle_block handle_block ] Got 34 trx on block 24291322 by someguy123 -- 400 ms application.cpp:514
2018-07-18T18:58:34 th_a:invoke handle_block handle_block ] Got 29 trx on block 24291323 by aggroed -- 4468 ms application.cpp:514
2018-07-18T18:58:34 th_a:Witness Block Production block_production_loo ] Not producing block because node didn't wake up within 500ms of the slot time. witness_plugin.cpp:624
2018-07-18T18:58:35 th_a:Witness Block Production block_production_loo ] Not producing block because node didn't wake up within 500ms of the slot time. witness_plugin.cpp:624
2018-07-18T18:58:36 th_a:invoke handle_block handle_block ] Got 56 trx on block 24291324 by ausbitbank -- 855 ms application.cpp:514

Possible scenario:

• a block arrived with high latency
• before it arrived, my node didn't try to produce a block (perhaps because it's not its turn?)
• after it arrived (witness list reshuffled or somehow?), my node suddenly found it's its turn to produce next block, however, it's already in the 2nd second, so it tried twice to produce a block
• the result (fail) was guaranteed by this code (time diff was always more than 500ms on the 2nd and 3rd try):

steem/libraries/plugins/witness/witness_plugin.cpp

Lines 702 to 706 in 1b53cf6

	if( llabs((scheduled_time - now).count()) > fc::milliseconds( 500 ).count() )
	{
	capture("scheduled_time", scheduled_time)("now", now);
	return block_production_condition::lag;
	}

[mvandeberg]

What version of steemd did this happen on?

[abitmore]

Was on 0.19.5. Although in OP the quoted code is master branch.