Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

go1.23 timers cause issues with libp2p connmgr and more #10501

Closed
Tracked by #10436
lidel opened this issue Aug 31, 2024 · 3 comments · Fixed by #10540
Closed
Tracked by #10436

go1.23 timers cause issues with libp2p connmgr and more #10501

lidel opened this issue Aug 31, 2024 · 3 comments · Fixed by #10540
Labels
kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization

Comments

@lidel
Copy link
Member

lidel commented Aug 31, 2024 •
edited
Loading

Version

0.30.0-rc2

Config

default `ipfs init` + custom ports to run A/B test

Description

Bug

0.30.0-rc2 switched to go1.23, and we experience an increased number of connections when running go-libp2p + go-libp2p-kad-dht server, confirmed both by me, and anecdotal evidence from discussion forums.

The problem seem to go away if running with forced go1.22 timer behavior GODEBUG=asynctimerchan=1 (Ref. https://go.dev/wiki/Go123Timer)

How to reproduce

  1. Run two instances of 0.30.0-rc2 with a public IP or UPnP, so they act as DHT servers and get a lot of connections from random peers. Run them with different env variables:
  • GODEBUG=asynctimerchan=0 to force Go 1.23 timers
  • GODEBUG=asynctimerchan=1 to force Go 1.22 timers
  1. After a few hours, the one that runs with GODEBUG=asynctimerchan=0 will have significantly more connections (e.g. 6k vs 200)
  2. Stopping with ctl-C does not work reliably with GODEBUG=asynctimerchan=0, it sometimes takes a second time to correctly stop the daemon.

Next steps

Short term, I will revert #10486 and switch back to 1.22 for 0.30.0 release and keep it until go-libp2p and go-libp2p-kad-dht are compatible with 1.23.
@lidel lidel added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Aug 31, 2024
@lidel lidel changed the title go1.23 go1.23 timers cause issues with libp2p connmgr and more Aug 31, 2024
@lidel lidel mentioned this issue Aug 31, 2024
Closed
32 tasks
@Jorropo
Copy link
Contributor

Jorropo commented Aug 31, 2024 •
edited
Loading

golang/go#69186 ?

lidel added a commit that referenced this issue Sep 2, 2024
@lidel
fix: switch back to go1.22
8af767d 
Switching back until #10501 is resolved.
@lidel lidel mentioned this issue Sep 2, 2024
Merged
lidel added a commit to ipfs/distributions that referenced this issue Sep 2, 2024
@lidel
fix: switch back to go1.22
aaa5622 
Unsafe to use 1.23 until ipfs/kubo#10501 is resolved
@lidel lidel mentioned this issue Sep 2, 2024
Merged
lidel added a commit to ipfs/distributions that referenced this issue Sep 2, 2024
@lidel
fix: switch back to go1.22 (#1096)
23a84fa 
Unsafe to use 1.23 until ipfs/kubo#10501 is resolved
lidel added a commit that referenced this issue Sep 2, 2024
@lidel
fix: switch back to go 1.22 (#10502)
23ca1dd 
Switching back until #10501 is resolved.
lidel added a commit that referenced this issue Sep 2, 2024
@lidel
fix: switch back to go 1.22 (#10502)
b50e029 
Switching back until #10501 is resolved.

(cherry picked from commit 23ca1dd)
@Stebalien
Copy link
Member

I don't think we have any cases where we copy timers.

@lidel
Copy link
Member Author

lidel commented Sep 3, 2024 •
edited
Loading

One of potential causes: quic-go/quic-go#4659
Upstream bug: golang/go#69312 / golang/go#69333

We will wait for upstream go to fix it in new patch release.

lidel added a commit that referenced this issue Oct 7, 2024
@lidel
chore: go1.23(.2)
cf364b8 
go1.23.2 includes potential fix for issue described in
#10501
@lidel lidel mentioned this issue Oct 7, 2024
Merged
5 tasks
lidel added a commit that referenced this issue Oct 8, 2024
@lidel
fix: go 1.23(.2) (#10540)
091bc08 
go1.23.2 includes potential fix for issue described in
#10501
lidel added a commit that referenced this issue Oct 8, 2024
@lidel
fix: go 1.23(.2) (#10540)
5985451 
go1.23.2 includes potential fix for issue described in
#10501

(cherry picked from commit 091bc08)
lidel added a commit to ipfs/distributions that referenced this issue Oct 8, 2024
@lidel
chore: golang 1.23.2
c5ad040 
Should be fine since ipfs/kubo#10501 was solved in 1.23.2
lidel added a commit to ipfs/distributions that referenced this issue Oct 8, 2024
@lidel
Publish Kubo: v0.31.0-rc2 (#1103)
0440348 
* chore: add Kubo release
* chore: golang 1.23.2
Should be fine since ipfs/kubo#10501 was solved in 1.23.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

chore: go 1.23(.2) ipfs/kubo
3 participants
@lidel @Stebalien @Jorropo