Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Call two times maybe_switch_forks consecutive, and apply_block exception #114

Closed
Frank-AFN opened this issue Jul 13, 2019 · 1 comment
Closed

Call two times maybe_switch_forks consecutive, and apply_block exception #114

Frank-AFN opened this issue Jul 13, 2019 · 1 comment

Comments

@Frank-AFN
Copy link

version: boscore: v3.0.0
env: k8s cluster 1

errorlog:

info 2019-07-09T07:02:23.688 thread-0 controller.cpp:1701 maybe_switch_forks ] switching forks from 000007d98e7f513387ea449914c202cfd2ccdfac47780844170362b1c1721246 (block number 2009) to 000007e8b16d2a694055f26e5635f6c04906fe39e03908727130c8f0e4d2e54d (block number 2024)
info 2019-07-09T07:02:23.689 thread-0 controller.cpp:1701 maybe_switch_forks ] switching forks from 000007bf9304f885f7cc60a868abe6f471c36a860fa238ef68e0db06fff3233e (block number 1983) to 000007e8b16d2a694055f26e5635f6c04906fe39e03908727130c8f0e4d2e54d (block number 2024)
info 2019-07-09T07:02:23.709 thread-0 controller.cpp:1746 maybe_switch_forks ] successfully switched fork to new head 000007e8b16d2a694055f26e5635f6c04906fe39e03908727130c8f0e4d2e54d
error 2019-07-09T07:02:23.709 thread-0 controller.cpp:1546 apply_block ] e.to_detail_string(): 3030000 block_validate_exception: Block exception
next block must be in the future
{}
thread-0 block_header_state.cpp:40 generate_next
error 2019-07-09T07:02:23.709 thread-0 controller.cpp:1721 maybe_switch_forks ] exception thrown while switching forks 3030000 block_validate_exception: Block exception
next block must be in the future
{}
thread-0 block_header_state.cpp:40 generate_next

{}
thread-0  controller.cpp:1550 apply_block

error 2019-07-09T07:02:23.747 thread-0 pbft.cpp:153 on_new_view ] apply new view failed, waiting for next round.. {"common":{"type":"new_view","timestamp":"2019-07-09T07:00:42.897"},"new_view":2,"prepared_cert":{"block_info":{"block_id":"0000000000000000000000000000000000000000000000000000000000000000"},"pre_prepares":[],"prepares":[]},"committed_certs":[{"block_info":{"block_id":"00000764be6c26b6c9cfb95d49987a4c4bedfed4d75c5d9572e1d8bef469f9f1"},"commits":[{"common":{"type":"commit","timestamp":"2019-07-09T07:00:05.694"},"view":1,"block_info":{"block_id":"00000764be6c26b6c9cfb95d49987a4c4bedfed4d75c5d9572e1d8bef469f9f1"},"sender_signature":"SIG_K1_K2UcwhvNn4H8guk94b2K3EhaH54yfbAYPwKVpSEUC7tPwoak5Tjdx2gSD11GAxUm97SWACAwGWFQcuG7WzQdzgqb6k877R"},{"common":{"type":"commit","timestamp":"2019-07-09T07:00:05.694"},"view":1,"block_info":{"block_id":"00000764be6c26b6c9cfb95d49987a4c4bedfed4d75c5d9572e1d8bef469f9f1"},"sender_signature":"SIG_K1_JvM7V3prjK3CKtd5ariRobGKkrLd2LQR2aja2dwnQ9uETW5snxo3UKZwZM8bnfqYsmWeXCG5pPEAUWL2dmqea3vwBHa4qh"},{"common":{"type":"commit","timestamp":"2019-07-09T07:00:05.764"},"view":1,"block_info":{"block_id":"00000764be6c26b6c9cfb95d49987a4c4bedfed4d75c5d9572e1d8bef469f9f1"},"sender_signature":"SIG_K1_KXo8gNsnxz65UTbqiu42F53JgWR1U3nkVBSWxwfWxgaCS9aRUfsCEp6zBZnDhoKMnnr3rMNj6HipB4Z9YY9rdhCECoGQun"},{"common":{"type":"commit","timestamp":"2019-07-09T07:00:05.908"},"view":1,"block_info":{"block_id":"00000764be6c26b6c9cfb95d49987a4c4bedfed4d75c5d9572e1d8bef469f9f1"},"sender_signature":"SIG_K1_Jv1un11VYyQUHjFYv4CE6tVsiLNTXtdfcrR1SZCStp48anuMVWpa2GoJghoD5eAE9RQeQKrjcfoe3hpddF5iJj1hMDrEm8"},{"common":{"type":"commit","timestamp":"2019-07-09T07:00:05.743"},"view":1,"block_info":{"block_id":"00000764be6c26b6c9cfb95d49987a4c4bedfed4d75c5d9572e1d8bef469f9f1"},"sender_signature":"SIG_K1_K68LECAZRybZYLXxns6Bs9mTwbk3McPGqxAKbKaYHSa5MLjqvn8LydSdWntv1crFgPajZ6YfP8JqLRUifbdPXWABNn3yau"}....

committed block did not become the new head in fork database
{}
thread-0 controller.cpp:927 commit_block
warn 2019-07-09T07:03:11.504 thread-0 producer_plugin.cpp:1147 start_block ] 3060000 database_exception: Database exception
db revision is not on par with head block
{"db.revision()":2024,"controller_head_block":2009,"fork_db_head_block":2009}
thread-0 controller.cpp:1358 start_block
error 2019-07-09T07:03:11.504 thread-0 producer_plugin.cpp:1446 schedule_production_ ] Failed to start a pending block, will try again later
warn 2019-07-09T07:03:11.554 thread-0 producer_plugin.cpp:1147 start_block ] 3060000 database_exception: Database exception
db revision is not on par with head block
{"db.revision()":2024,"controller_head_block":2009,"fork_db_head_block":2009}
thread-0 controller.cpp:1358 start_block
error 2019-07-09T07:03:11.554 thread-0 producer_plugin.cpp:1446 schedule_production_ ] Failed to start a pending block, will try again later
warn 2019-07-09T07:03:11.605 thread-0 producer_plugin.cpp:1147 start_block ] 3060000 database_exception: Database exception
db revision is not on par with head block
{"db.revision()":2024,"controller_head_block":2009,"fork_db_head_block":2009}
thread-0 controller.cpp:1358 start_block
error 2019-07-09T07:03:11.605 thread-0 producer_plugin.cpp:1446 schedule_production_ ] Failed to start a pending block, will try again later
warn 2019-07-09T07:03:11.655 thread-0 producer_plugin.cpp:1147 start_block ] 3060000 database_exception: Database exception
db revision is not on par with head block
{"db.revision()":2024,"controller_head_block":2009,"fork_db_head_block":2009}
@Frank-AFN
Copy link
Author

It will call on new view when handling a new view message. The new view with committed certificate will commit local and set pending_pbft_lib value to be true. This will trigger the second time maybe_switch_forks function call. First of all, maybe_switch_forks is called in the cleanup_on_new_view. At this time, the controller head 2009 try to switch forks to fork_db head 2024. After it pops from block 2009 to 1983. Then it tries to apply the block 1983 to 2024, but when apply_block it will trigger set_pbft_lib in start_block function. Then it will call maybe_switch_forks second time because of pending_pbft_lib is true which was set previously. The controller will switch fork from current head 1983 to 2024 in the second maybe_switch_forks function call. This time it success to switch forks to head 2024. After finish the second time switch fork, it will go back to the first time switch fork. It will throw an exception block_validate_exception, "next block must be in the future" when generating next block.

This issue has a full test case called new_view_with_committed_cert_call_two_times_maybe_switch_forks. And fixed in commit 20e08db

vlbos pushed a commit to vlbos/bos that referenced this issue Sep 18, 2019
# This is a combination of 186 commits.
6c09577 
# This is the 1st commit message:

various improvements

# This is the commit message #2:

new hash

# This is the commit message #3:

fix for script path

# This is the commit message #4:

fixes

# This is the commit message #5:

fixes

# This is the commit message #6:

fixes

# This is the commit message #7:

fixes

# This is the commit message #8:

fixes

# This is the commit message #9:

fixes

# This is the commit message #10:

fixes

# This is the commit message boscore#11:

fixes

# This is the commit message boscore#12:

fixes

# This is the commit message boscore#13:

fixes

# This is the commit message boscore#14:

fixes

# This is the commit message boscore#15:

fixes

# This is the commit message boscore#16:

fixes

# This is the commit message boscore#17:

fixes

# This is the commit message boscore#18:

fixes

# This is the commit message boscore#19:

fixes

# This is the commit message boscore#20:

fixes

# This is the commit message boscore#21:

fixes

# This is the commit message boscore#22:

fixes

# This is the commit message boscore#23:

fixes

# This is the commit message boscore#24:

fixes

# This is the commit message boscore#25:

fixes

# This is the commit message boscore#26:

testing

# This is the commit message boscore#27:

testing

# This is the commit message boscore#28:

testing

# This is the commit message boscore#29:

testing

# This is the commit message boscore#30:

testing

# This is the commit message boscore#31:

testing

# This is the commit message boscore#32:

testing

# This is the commit message boscore#33:

testing

# This is the commit message boscore#34:

testing

# This is the commit message boscore#35:

testing

# This is the commit message boscore#36:

testing

# This is the commit message boscore#37:

testing

# This is the commit message boscore#38:

testing

# This is the commit message boscore#39:

testing

# This is the commit message boscore#40:

testing

# This is the commit message boscore#41:

testing

# This is the commit message boscore#42:

testing

# This is the commit message boscore#43:

testing

# This is the commit message boscore#44:

fixes

# This is the commit message boscore#45:

fixes

# This is the commit message boscore#46:

fixes

# This is the commit message boscore#47:

fixes

# This is the commit message boscore#48:

fixes

# This is the commit message boscore#49:

fixes

# This is the commit message boscore#50:

fixes

# This is the commit message boscore#51:

fixes

# This is the commit message boscore#52:

fixes

# This is the commit message boscore#53:

fixes

# This is the commit message boscore#54:

fixes

# This is the commit message boscore#55:

fixes

# This is the commit message boscore#56:

fixes

# This is the commit message boscore#57:

fixes

# This is the commit message boscore#58:

fixes

# This is the commit message boscore#59:

fixes

# This is the commit message boscore#60:

fixes

# This is the commit message boscore#61:

fixes

# This is the commit message boscore#62:

fixes

# This is the commit message boscore#63:

fixes

# This is the commit message boscore#64:

fixes

# This is the commit message boscore#65:

fixes

# This is the commit message boscore#66:

fixes

# This is the commit message boscore#67:

fixes

# This is the commit message boscore#68:

fixes

# This is the commit message boscore#69:

fixes

# This is the commit message boscore#70:

fixes

# This is the commit message boscore#71:

fixes

# This is the commit message boscore#72:

fixes

# This is the commit message boscore#73:

fixes

# This is the commit message boscore#74:

fixes

# This is the commit message boscore#75:

fixes

# This is the commit message boscore#76:

fixes

# This is the commit message boscore#77:

fixes

# This is the commit message boscore#78:

fixes

# This is the commit message boscore#79:

more testing

# This is the commit message boscore#80:

testing

# This is the commit message boscore#81:

fixes

# This is the commit message boscore#82:

fixes

# This is the commit message boscore#83:

fixes

# This is the commit message boscore#84:

fixes

# This is the commit message boscore#85:

fixes

# This is the commit message boscore#86:

fixes

# This is the commit message boscore#87:

fixes

# This is the commit message boscore#88:

fixes

# This is the commit message boscore#89:

fixes

# This is the commit message boscore#90:

fixes

# This is the commit message boscore#91:

fixes

# This is the commit message boscore#92:

fixes

# This is the commit message boscore#93:

propagate-environment for buildkite-agent

# This is the commit message boscore#94:

propagate-environment for buildkite-agent

# This is the commit message boscore#95:

propagate-environment for buildkite-agent

# This is the commit message boscore#96:

propagate-environment for buildkite-agent

# This is the commit message boscore#97:

fixes

# This is the commit message boscore#98:

fixes

# This is the commit message boscore#99:

fixes

# This is the commit message boscore#100:

fixes

# This is the commit message boscore#101:

fixes

# This is the commit message boscore#102:

fixes

# This is the commit message boscore#103:

fixes

# This is the commit message boscore#104:

fixes

# This is the commit message boscore#105:

fixes

# This is the commit message boscore#106:

fixes

# This is the commit message boscore#107:

fixes

# This is the commit message boscore#108:

fixes

# This is the commit message boscore#109:

fixes

# This is the commit message boscore#110:

fixes

# This is the commit message boscore#111:

fixes

# This is the commit message boscore#112:

fixes

# This is the commit message boscore#113:

fixes

# This is the commit message boscore#114:

fixes

# This is the commit message boscore#115:

fixes

# This is the commit message boscore#116:

fixes

# This is the commit message boscore#117:

fixes

# This is the commit message boscore#118:

fixes

# This is the commit message boscore#119:

fixes

# This is the commit message boscore#120:

fixes

# This is the commit message boscore#121:

fixes

# This is the commit message boscore#122:

fixes

# This is the commit message boscore#123:

fixes

# This is the commit message boscore#124:

fixes

# This is the commit message boscore#125:

fixes

# This is the commit message boscore#126:

fixes

# This is the commit message boscore#127:

fixes

# This is the commit message boscore#128:

fixes

# This is the commit message boscore#129:

fixes

# This is the commit message boscore#130:

fixes

# This is the commit message boscore#131:

fixes

# This is the commit message boscore#132:

fixes

# This is the commit message boscore#133:

fixes

# This is the commit message boscore#134:

fixes

# This is the commit message boscore#135:

fixes

# This is the commit message boscore#136:

fixes

# This is the commit message boscore#137:

fixes

# This is the commit message boscore#138:

fixes

# This is the commit message boscore#139:

fixes

# This is the commit message boscore#140:

fixes

# This is the commit message boscore#141:

fixes

# This is the commit message boscore#142:

fixes

# This is the commit message boscore#143:

fixes

# This is the commit message boscore#144:

fixes

# This is the commit message boscore#145:

fixes

# This is the commit message boscore#146:

fixes

# This is the commit message boscore#147:

fixes

# This is the commit message boscore#148:

fixes

# This is the commit message boscore#149:

fixes

# This is the commit message boscore#150:

fixes

# This is the commit message boscore#151:

fixes

# This is the commit message boscore#152:

fixes

# This is the commit message boscore#153:

testing

# This is the commit message boscore#154:

fixes

# This is the commit message boscore#155:

fixes

# This is the commit message boscore#156:

fixes

# This is the commit message boscore#157:

fixes

# This is the commit message boscore#158:

fixes

# This is the commit message boscore#159:

fixes

# This is the commit message boscore#160:

fixes

# This is the commit message boscore#161:

fixes

# This is the commit message boscore#162:

fixes

# This is the commit message boscore#163:

fixes

# This is the commit message boscore#164:

fixes

# This is the commit message boscore#165:

fixes

# This is the commit message boscore#166:

fixes

# This is the commit message boscore#167:

fixes

# This is the commit message boscore#168:

fixes

# This is the commit message boscore#169:

fixes

# This is the commit message boscore#170:

fixes

# This is the commit message boscore#171:

fixes

# This is the commit message boscore#172:

fixes

# This is the commit message boscore#173:

fixes

# This is the commit message boscore#174:

fixes

# This is the commit message boscore#175:

fixes

# This is the commit message boscore#176:

fixes

# This is the commit message boscore#177:

fixes

# This is the commit message boscore#178:

fixes

# This is the commit message boscore#179:

fixes

# This is the commit message EOSIO#180:

fixes

# This is the commit message EOSIO#181:

fixes

# This is the commit message EOSIO#182:

fixes

# This is the commit message EOSIO#183:

fixes

# This is the commit message EOSIO#184:

fixes

# This is the commit message EOSIO#185:

fixes

# This is the commit message EOSIO#186:

fixes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Frank-AFN