Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: rcmgr limit conflicts with connmgr limit #2628

Closed
richard-ramos opened this issue Oct 31, 2023 · 12 comments · Fixed by #2648
Closed

Question: rcmgr limit conflicts with connmgr limit #2628

richard-ramos opened this issue Oct 31, 2023 · 12 comments · Fixed by #2648
Assignees

Comments

@richard-ramos
Copy link
Contributor

While running this example code with GOLOG_LOG_LEVEL=WARN go run main.go , I always see this warning being printed:

2023-10-31T12:36:25.909-0400    WARN    p2p-config      config/config.go:304    rcmgr limit conflicts with connmgr limit: conn manager high watermark limit: 192, exceeds the system connection limit of: 1

What does this warning mean? I tried to follow go-libp2p code for the resource manager and connection manager to find out but had some trouble understanding it. Is there something I should change to avoid having this warning?

package main

import (
	"fmt"
	"os"
	"os/signal"
	"syscall"

	"github.com/libp2p/go-libp2p"
)

func main() {
	host, err := libp2p.New()
	if err != nil {
		panic(err.Error())
	}

	fmt.Println(host.ID())

	// Wait for a SIGINT or SIGTERM signal
	ch := make(chan os.Signal, 1)
	signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
	<-ch
	fmt.Println("\n\n\nReceived signal, shutting down...")
}

Thank you in advance.

@burdiyan
Copy link
Contributor

Facing the same issue.

@sukunrt
Copy link
Member

sukunrt commented Nov 14, 2023

This means that you have configured the connection manager in such a way that it will never be used.

Libp2p uses two modules for connection managements. Resource manager limit(rcmgr) is a hard limit. If the rcmgr limit is set to 1 you will only have 1 open connection.

The ConnMgr limit is a soft limit of sort. It allows for a burst of connections between ConnMgr.LowWaterMark and ConnMgr.HighWaterMark. This allows for a burst of connections between Low and high watermark. But when you exceed the high watermark, the system trims connections down to low watermark.

In your case HighWatermark trimming never happens because it's higher than the rcmgr limit. Ideally your high watermark limit should be a bit less than the rcmgr connections limit.

@juligasa
Copy link

I have the same issue, even if I set unlimited resources, it still complains about the system limit being 1 WARN p2p-config config/config.go:304 rcmgr limit conflicts with connmgr limit: conn manager high watermark limit: 768, exceeds the system connection limit of: 1
My configuration so far:

cm := mustConnMgr(connmgr.NewConnManager(
	512,
	768,
	connmgr.WithGracePeriod(r.cfg.ConnMgr.ConnMgrGrace),
))

opts = append(opts,
	libp2p.ConnectionManager(cm),
)

scalingLimits := rcmgr.DefaultLimits

// Add limits around included libp2p protocols
libp2p.SetDefaultServiceLimits(&scalingLimits)
// Turn the scaling limits into a concrete set of limits using `.AutoScale`. This
// scales the limits proportional to your system memory.
scaledDefaultLimits := scalingLimits.AutoScale()

// Tweak certain settings
cmcfg := rcmgr.PartialLimitConfig{
	System: rcmgr.ResourceLimits{
		Streams:  rcmgr.Unlimited,
		Conns:    rcmgr.Unlimited,
		FD:      rcmgr.Unlimited,
		Memory:  rcmgr.Unlimited64,
	},
	// Everything else is default. The exact values will come from `scaledDefaultLimits` above.
}

// Create our limits by using our cfg and replacing the default values with values from `scaledDefaultLimits`
limits := cmcfg.Build(scaledDefaultLimits)

// The resource manager expects a limiter, so we create one from our limits.
limiter := rcmgr.NewFixedLimiter(limits)

// Metrics are enabled by default. If you want to disable metrics, use the
// WithMetricsDisabled option
// Initialize the resource manager
rm, err := rcmgr.NewResourceManager(limiter)
if err != nil {
	panic(err)
}
	
opts = append(opts,
	libp2p.ResourceManager(rm),
)

r.host, err = libp2p.New(opts...)
if err != nil {
	return err
}

@burdiyan
Copy link
Contributor

@juligasa I fixed that by using rcmgr.NewFixedLimiter(rcmgr.InfiniteLimits) instead of trying to manually expand the default limits to be unlimited. That fixed the issue with the connmgr conflict.

@juligasa
Copy link

Oh, I just followed the package recommendations at
https://github.com/libp2p/go-libp2p/tree/master/p2p/host/resource-manager#usage
I wonder why it did not work as expected.

@richard-ramos
Copy link
Contributor Author

This means that you have configured the connection manager in such a way that it will never be used.

I'm confused on why I'm getting that warning even tho I have not configured any custom connection manager, but using the defaults provided by go-libp2p as seen on the example code i wrote here: #2628 (comment) . Shouldnt go-libp2p by default configure the connection manager with settings that avoid this warning being emitted?

@sukunrt
Copy link
Member

sukunrt commented Nov 14, 2023

@juligasa you need to set the Conn scope limit too.

	// Tweak certain settings
	cmcfg := rcmgr.PartialLimitConfig{
		System: rcmgr.ResourceLimits{
			Streams: rcmgr.Unlimited,
			Conns:   rcmgr.Unlimited,
			FD:      rcmgr.Unlimited,
			Memory:  rcmgr.Unlimited64,
		},
		// Everything else is default. The exact values will come from `scaledDefaultLimits` above.
		Conn: rcmgr.ResourceLimits{
			Conns: rcmgr.Unlimited,
		},
	}

Shouldnt go-libp2p by default configure the connection manager with settings that avoid this warning being emitted?

I agree. I'll check how to improve the situation here.

@sukunrt sukunrt self-assigned this Nov 14, 2023
@juligasa
Copy link

@juligasa you need to set the Conn scope limit too.

I see, I thought the System level was the top abstraction scope, though. And by setting it to unlimited would cause the rest of scopes to be also unlimited (if no further restriction was in place). According to the documentation:

The system scope is the top level scope that accounts for global resource usage at all levels of the system. This scope nests and constrains all other scopes and institutes global hard limits

I guess the libp2p.SetDefaultServiceLimits overrides the connection limit to 1 and hence, the warning.

@sukunrt
Copy link
Member

sukunrt commented Nov 16, 2023

This constraint is the other way round, no?

The system scope is the top level scope that accounts for global resource usage at all levels of the system. This scope nests and constrains all other scopes and institutes global hard limits

If the system scope is 1 connection, peer scope cannot be 10 connections

On the other hand

If the system scope is 100 connections, peer scope can be 1 connection
If the system scope is inf connections, peer scope can be 1 connection

@juligasa
Copy link

Sure you're right. The problem is with default peer scopes though. I did not expect them to be so tight by default (actually, allowing only 1 connection is pretty tight). Will set limits accordingly, thanks!

@sukunrt
Copy link
Member

sukunrt commented Nov 18, 2023

You're right, @juligasa . This warning doesn't make sense. The system connection limit is what we should check. I've raised a PR to fix this.

Just to be clear:

You DO NOT NEED to increase Conn Scope limit as I suggested here. I'm sorry about that. #2628 (comment)
The Connection scope only ever needs 1 connection because it is well tied to the connection 😓

@juligasa
Copy link

Ok, Makes sense

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants