Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unit test bootstrap_processor.push_one failed in the CI #3533

Closed
thsfs opened this issue Oct 28, 2021 · 3 comments · Fixed by #3721
Closed

Unit test bootstrap_processor.push_one failed in the CI #3533

thsfs opened this issue Oct 28, 2021 · 3 comments · Fixed by #3721
Assignees
@thsfs
Labels
unit test Related to a new, changed or fixed unit test

Comments

@thsfs
Copy link
Contributor

thsfs commented Oct 28, 2021

2021-10-28T15:41:17.1958111Z [ RUN ] bootstrap_processor.push_one
2021-10-28T15:41:38.3001362Z /workspace/nano/core_test/bootstrap.cpp:474: Failure
2021-10-28T15:41:38.3011285Z Value of: system.poll ()
2021-10-28T15:41:38.3011862Z Actual: Deadline expired
2021-10-28T15:41:38.3012245Z Expected:
2021-10-28T15:41:38.3012725Z [ FAILED ] bootstrap_processor.push_one (20984 ms)

https://github.com/nanocurrency/nano-node/runs/4036418895?check_suite_focus=true
@thsfs thsfs self-assigned this Oct 28, 2021
@thsfs thsfs mentioned this issue Oct 28, 2021
Merged
@thsfs
Copy link
Contributor Author

thsfs commented Oct 28, 2021

Disabled by PR: #3532

@zhyatt zhyatt added the unit test Related to a new, changed or fixed unit test label Oct 28, 2021
@thsfs
Copy link
Contributor Author

thsfs commented Nov 5, 2021

unable to reproduce this using stress/nice commands.

@thsfs thsfs mentioned this issue Feb 7, 2022
Closed
@dsiganos
Copy link
Contributor

dsiganos commented Feb 8, 2022

I can recreate this problem every time by adding this delay:
if (!new_client) std::this_thread::sleep_for (100ms);
to the top of the function:
void nano::bootstrap_connections::pool_connection (std::shared_ptr<nano::bootstrap_client> const & client_a, bool new_client, bool push_front)

@dsiganos dsiganos mentioned this issue Feb 8, 2022
Closed
dsiganos added a commit to dsiganos/nano-node that referenced this issue Feb 8, 2022
@dsiganos
fix race condition between bootstrap pull and push
fccf5db 
The unit test bootstrap_processor.push_one fails intermittently and it
exposes a race condition in the bootstrap code of the node. nanocurrency#3533

The race condition is that after finishing with bootstrap frontiers pull,
the code is supposed to put the socket back into the idle bootstrap
connections queue. However it does that in a racy way. The problem is in
the function nano::frontier_req_client::received_frontier.

Near the bottom of that function, we should call:
connection->connections.pool_connection (connection);
before calling
promise.set_value (false);

Once the promise is given a value then
nano::bootstrap_attempt_legacy::run()
continues to call push_request in parallel with pool_connection and it is
a race condition.

Although this is a bug, this is likely not a major nor catastrophic bug
because the bootstrap push functionality is only an efficiency.

resolves nanocurrency#3533
resolves nanocurrency#3720
@dsiganos dsiganos mentioned this issue Feb 8, 2022
Merged
dsiganos added a commit that referenced this issue Feb 8, 2022
@dsiganos
fix race condition between bootstrap pull and push (#3721)
ddf07f1 
The unit test bootstrap_processor.push_one fails intermittently and it
exposes a race condition in the bootstrap code of the node. #3533

The race condition is that after finishing with bootstrap frontiers pull,
the code is supposed to put the socket back into the idle bootstrap
connections queue. However it does that in a racy way. The problem is in
the function nano::frontier_req_client::received_frontier.

Near the bottom of that function, we should call:
connection->connections.pool_connection (connection);
before calling
promise.set_value (false);

Once the promise is given a value then
nano::bootstrap_attempt_legacy::run()
continues to call push_request in parallel with pool_connection and it is
a race condition.

Although this is a bug, this is likely not a major nor catastrophic bug
because the bootstrap push functionality is only an efficiency.

resolves #3533
resolves #3720
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thsfs thsfs

Labels
unit test Related to a new, changed or fixed unit test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix race condition between bootstrap pull and push dsiganos/nano-node
3 participants
@zhyatt @dsiganos @thsfs