fix(engine api): [3/3] Engine API retryWithTimeout

[shotes]

This PR introduces retries to engine_newPayload and engine_forkchoiceUpdated calls.

Additionally, remove one of the return values from NotifyForkchoiceUpdate that was never used.

This PR introduces a third party go port of google-http-client ExponentialBackOff.java. I have read through all of the code that the third party package implements. Please verify that package as well.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 151 lines in your changes missing coverage. Please review.

Project coverage is 32.65%. Comparing base (f09c436) to head (62b5838).

Files with missing lines	Patch %	Lines
execution/engine/engine.go	0.00%	147 Missing ⚠️
payload/builder/payload.go	0.00%	2 Missing ⚠️
beacon/blockchain/execution_engine.go	0.00%	1 Missing ⚠️
beacon/blockchain/payload.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2490      +/-   ##
==========================================
- Coverage   32.77%   32.65%   -0.12%     
==========================================
  Files         351      351              
  Lines       15857    15915      +58     
  Branches       20       20              
==========================================
  Hits         5197     5197              
- Misses      10276    10334      +58     
  Partials      384      384              

Files with missing lines	Coverage Δ
beacon/blockchain/execution_engine.go	0.00% <0.00%> (ø)
beacon/blockchain/payload.go	0.00% <0.00%> (ø)
payload/builder/payload.go	20.80% <0.00%> (ø)
execution/engine/engine.go	0.00% <0.00%> (ø)

[fridrik01]

@shotes I checked out this branch (to check whether it resolved some syncing issue I had locally) and I observed that now beacond hangs when it receives SIGTERM if its currently inside a retry. It requires to either wait up to 5 minutes or send it a SIGKILL to exit. This behavior was not in deposit-sync-force-startup-fcu branch.

EDIT: I think the main reason is that the context passed to NotifyForkchoiceUpdate is not properly cancelled. Let me know if you want me to take a look at that, I can push a separate PR on top of this one, should have time this weekend.

[shotes]

@shotes I checked out this branch (to check whether it resolved some syncing issue I had locally) and I observed that now beacond hangs when it receives SIGTERM if its currently inside a retry. It requires to either wait up to 5 minutes or send it a SIGKILL to exit. This behavior was not in deposit-sync-force-startup-fcu branch.

EDIT: I think the main reason is that the context passed to NotifyForkchoiceUpdate is not properly cancelled. Let me know if you want me to take a look at that, I can push a separate PR on top of this one, should have time this weekend.

@fridrik01 I am seeing this issue as well, having to specifically kill the process each time I restart. Just pushed up changes that uses a different lib, still seeing the same issue. Investigating now - feel free to test and push any fixes.

[fridrik01]

@shotes I checked out this branch (to check whether it resolved some syncing issue I had locally) and I observed that now beacond hangs when it receives SIGTERM if its currently inside a retry. It requires to either wait up to 5 minutes or send it a SIGKILL to exit. This behavior was not in deposit-sync-force-startup-fcu branch.
EDIT: I think the main reason is that the context passed to NotifyForkchoiceUpdate is not properly cancelled. Let me know if you want me to take a look at that, I can push a separate PR on top of this one, should have time this weekend.

@fridrik01 I am seeing this issue as well, having to specifically kill the process each time I restart. Just pushed up changes that uses a different lib, still seeing the same issue. Investigating now - feel free to test and push any fixes.

@shotes I pushed #2496 which addresses hanging on SIGTERM, however it unwraps an issue it seems in cometbft where it panics on a cancelled context. Need to figure out what is going on there, maybe cometbft does not support ctx cancellation