Memory usage on CentOS Stream 9

[hrw]

We use RabbitMQ in OpenStack Kolla based deployments.

When RMQ 3.9.21 is started in Debian/stable based container it uses ~140MB ram for "/usr/lib/erlang/erts-12.2.1/bin/beam.smp" process.

When same version is started in CentOS Stream 9 container (using "el/8" packages from PackageCloud) then "beam.smp" takes ~1.6GB ram.

Any idea what causes such difference? It makes our CI nodes OOM (on 8GB ram) so we are unable to check how deployment works on CS9.

[michaelklishin]

It's not clear to me how this is relevant to this repo since you use "a Debian container" and this repo is an Erlang RPM.

We do not guess in this community. rabbitmq-diagnostics memory_breakdown should "know".

[hrw]

Packages and OS information first:

(rabbitmq)[root@kolla-cs9 /]# rpm -qa|egrep "(rabbitmq|erlang)"
erlang-24.3.4.2-1.el8.x86_64
rabbitmq-server-3.9.21-1.el8.noarch

(rabbitmq)[root@kolla-cs9 /]# uname -a
Linux kolla-cs9 5.14.0-124.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jul 4 11:45:20 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

(rabbitmq)[root@kolla-cs9 /]# cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

Then memory information (thanks for information how to gather them):

(rabbitmq)[root@kolla-cs9 /]# rabbitmq-diagnostics memory_breakdown
Reporting memory breakdown on node rabbit@kolla-cs9...
other_system: 1.6233 gb (68.59%)
allocated_unused: 0.5164 gb (21.82%)
binary: 0.1551 gb (6.55%)
code: 0.0356 gb (1.51%)
other_proc: 0.0184 gb (0.78%)
connection_other: 0.0037 gb (0.16%)
other_ets: 0.0036 gb (0.15%)
queue_procs: 0.0021 gb (0.09%)
plugins: 0.0021 gb (0.09%)
connection_readers: 0.0021 gb (0.09%)
atom: 0.0014 gb (0.06%)
mgmt_db: 0.001 gb (0.04%)
metrics: 8.0e-4 gb (0.03%)
connection_writers: 4.0e-4 gb (0.02%)
connection_channels: 4.0e-4 gb (0.02%)
mnesia: 3.0e-4 gb (0.01%)
msg_index: 0.0 gb (0.0%)
quorum_ets: 0.0 gb (0.0%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)

That 1.6GB for 'other_system' bothers me. On Debian/stable it looks different:

(rabbitmq)[root@debian /]# rabbitmq-diagnostics memory_breakdown
Reporting memory breakdown on node rabbit@debian...
binary: 0.2787 gb (70.2%)
code: 0.0355 gb (8.93%)
other_system: 0.0255 gb (6.44%)
other_proc: 0.0209 gb (5.26%)
connection_other: 0.0108 gb (2.72%)
mgmt_db: 0.0073 gb (1.84%)
plugins: 0.0048 gb (1.2%)
other_ets: 0.0037 gb (0.93%)
connection_readers: 0.003 gb (0.75%)
queue_procs: 0.0028 gb (0.7%)
atom: 0.0015 gb (0.37%)
metrics: 0.0011 gb (0.28%)
connection_channels: 8.0e-4 gb (0.21%)
mnesia: 4.0e-4 gb (0.1%)
connection_writers: 2.0e-4 gb (0.05%)
msg_index: 0.0 gb (0.01%)
quorum_ets: 0.0 gb (0.01%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)
allocated_unused: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)

[hrw]

Debian packages info:

(rabbitmq)[root@debian /]# dpkg -l|egrep "(rabbit|erlang)"
ii  erlang-asn1                    1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP modules for ASN.1 support
ii  erlang-base                    1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP virtual machine and base applications
ii  erlang-crypto                  1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP cryptographic modules
ii  erlang-eldap                   1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP LDAP library
ii  erlang-ftp                     1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP FTP client
ii  erlang-inets                   1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP Internet clients and servers
ii  erlang-mnesia                  1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP distributed relational/object hybrid database
ii  erlang-os-mon                  1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP operating system monitor
ii  erlang-parsetools              1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP parsing tools
ii  erlang-public-key              1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP public key infrastructure
ii  erlang-runtime-tools           1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP runtime tracing/debugging tools
ii  erlang-snmp                    1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP SNMP applications
ii  erlang-ssl                     1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP implementation of SSL
ii  erlang-syntax-tools            1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP modules for handling abstract Erlang syntax trees
ii  erlang-tftp                    1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP TFTP client and server
ii  erlang-tools                   1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP various tools
ii  erlang-xmerl                   1:24.2.1+dfsg-1~bpo11+1        amd64        Erlang/OTP XML tools
ii  rabbitmq-server                3.9.21-1                       all          Multi-protocol messaging broker

Erlang came from distribution, rabbitmq-server from RabbitMQ cloudsmith repository.

[michaelklishin]

Prometheus and Grafana have a way of retrieving memory allocator metrics. As would rabbitmq-diagnostics observer IIRC.

other_system and allocated_unused are influenced by the runtime allocators used. They can be changed, obviously I cannot tell you what exact combination is going to work differently but maybe +MMscs needs to be explicitly set to 512
or a lower value. What exactly may influence default allocator settings on CentOS 9 or a specific
kernel version, I'm not sure.

This is a great question for the erlang-users mailing list.

There is nothing particularly creative about this RPM package, e.g. we do not change any defaults.

[hrw]

It feels wider problem :(

As long as CentOS Stream 9 is host Erlang/rmq will eat ~1.6GB ram. Nevermind do I run:

• el/8 packages in CS9 container
• CS9 packages in CS9 container
• Debian packages in Debian container

:(

[lukebakken]

@hrw I can't reproduce what you report using the docker image of CentOS 9 Stream. Please see my repository for how I tried to reproduce what you report:

https://github.com/lukebakken/rabbitmq-erlang-rpm-104

NOTE: I am NOT using the el8 packages from Package Cloud - I am using the generic-unix package available here. Perhaps that is the difference. If you could provide a complete Dockerfile I can use to build an image in exactly the same manner as you, I might be able to reproduce your issue.

Without a way to reproduce we can't assist further.

[hrw]

It feels that the problem is not inside of container. If I run the same CS9 container on Debian host then 0.2GB ram is used.

I get ~1.6GB ram in use when I run:

• CS9 container with "el8" from packagecloud on CS9 host
• Debian container on CS9 host
• CS9 container with CS9 rabbitmq on CS9 host

When I run same containers on Debian host then 0.2GB ram is used.

[lukebakken]

I don't have access to a CS9 physical host. Can you reproduce this issue with CS9 running in a VM like virtualbox?

[hrw]

Yes, VM is enough. That's how we got to the problem too.

[michaelklishin]

This can be a kernel/runtime interaction or a different default of some kind in Stream 9 that makes the runtime allocate a much larger block upfront. This is a great question for erlang-users.

[michaelklishin]

I cannot reproduce. Using the latest Stream 9 with the 5.14 64-bit kernel:

Linux localhost.localdomain 5.14.0-134.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Jul 21 12:57:06 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

and RabbitMQ 3.10.7 installed with Erlang 25.0.2 from our Cloudsmith repositories using

sudo dnf install rabbitmq-server -y

I get
the following output from a node with just management plugin enabled and no data except for the seed:

sudo rabbitmq-diagnostics memory_breakdown --unit=mb
Reporting memory breakdown on node rabbit@localhost...
reserved_unallocated: 85.2267 mb (60.98%)
code: 34.763 mb (24.87%)
other_proc: 17.8169 mb (12.75%)
other_system: 14.2133 mb (10.17%)
other_ets: 3.0009 mb (2.15%)
atom: 1.4257 mb (1.02%)
plugins: 1.0389 mb (0.74%)
metrics: 0.6474 mb (0.46%)
mgmt_db: 0.3143 mb (0.22%)
binary: 0.2977 mb (0.21%)
connection_other: 0.1411 mb (0.1%)
mnesia: 0.0908 mb (0.06%)
msg_index: 0.0305 mb (0.02%)
quorum_ets: 0.0247 mb (0.02%)
quorum_queue_dlx_procs: 0.0028 mb (0.0%)
stream_queue_procs: 0.0028 mb (0.0%)
stream_queue_replica_reader_procs: 0.0028 mb (0.0%)
connection_readers: 0.0 mb (0.0%)
connection_writers: 0.0 mb (0.0%)
connection_channels: 0.0 mb (0.0%)
queue_procs: 0.0 mb (0.0%)
queue_slave_procs: 0.0 mb (0.0%)
quorum_queue_procs: 0.0 mb (0.0%)
stream_queue_coordinator_procs: 0.0 mb (0.0%)
allocated_unused: 0.0 mb (0.0%)

[lukebakken]

Are you using virtualbox? Just curious.

[michaelklishin]

VMware Fusion

[hrw]

I am on vacations to the end of month. Will get back to the problem in September.

[mnasiadka]

So, let me add some things to that:
After just running rabbitmq - the memory usage is similar to what @michaelklishin pasted.
After deploying an OpenStack control plane (so effectively clients that connect) - the memory usage is as follows:

(rabbitmq)[rabbitmq@michal-rocky-9 /]$ rabbitmq-diagnostics memory_breakdown --unit=mb
Reporting memory breakdown on node rabbit@michal-rocky-9...
other_system: 1623.4243 mb (67.18%)
allocated_unused: 520.1108 mb (21.52%)
binary: 194.5488 mb (8.05%)
code: 35.5923 mb (1.47%)
other_proc: 18.8867 mb (0.78%)
connection_other: 5.1595 mb (0.21%)
plugins: 4.2841 mb (0.18%)
other_ets: 3.6209 mb (0.15%)
connection_readers: 2.7638 mb (0.11%)
queue_procs: 2.507 mb (0.1%)
mgmt_db: 1.9964 mb (0.08%)
atom: 1.4339 mb (0.06%)
metrics: 0.9213 mb (0.04%)
connection_channels: 0.5021 mb (0.02%)
mnesia: 0.3771 mb (0.02%)
connection_writers: 0.1683 mb (0.01%)
msg_index: 0.0308 mb (0.0%)
quorum_ets: 0.0237 mb (0.0%)
stream_queue_procs: 0.0027 mb (0.0%)
stream_queue_replica_reader_procs: 0.0027 mb (0.0%)
queue_slave_procs: 0.0 mb (0.0%)
quorum_queue_procs: 0.0 mb (0.0%)
stream_queue_coordinator_procs: 0.0 mb (0.0%)
reserved_unallocated: 0.0 mb (0.0%)

ps output:

42439      27437  0.0  0.0 224764  3356 ?        S    10:25   0:00 /bin/sh /usr/lib/rabbitmq/bin/rabbitmq-server
42439      27460  1.6 10.5 4927092 1727208 ?     Sl   10:25   0:16 /usr/lib64/erlang/erts-12.3.2/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 512 -MHlmbcs 512 -MMmcs 30 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -sbwt none -sbwtdcpu none -sbwtdio none -S 2:2 -sbwt none -sbwtdcpu none -sbwtdio none -B i -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -epmd_port 4369 -- -pa  -noshell -noinput -s rabbit boot -boot start_sasl -kernel inetrc '/etc/rabbitmq/erl_inetrc' -syslog logger [] -syslog syslog_error_logger false -kernel prevent_overlapping_partitions false

Log attached
rabbitmq.log
.

[lukebakken]

(so effectively clients that connect)

What does that mean? What clients, how many, what sort of workload?

[michaelklishin]

According to the log there were 74 accepted connections. In my case, there were none. While 74 is not a large number per se, they can consume enough with all defaults to trigger a larger block allocation.
In many cases a much smaller TCP buffer size would be enough, decreasing per-connection footprint and decreasing per-connection throughput roughly proportionally, which may be perfectly fine for some services.

[mnasiadka]

What clients? OpenStack API services, 74 channels (74 connections), max 2-3 consumers per channel, one authenticated user, one vhost, small messages.
Attached rabbitmqctl report output (if helpful).
rabbitmq_report.txt

Changing tcp_listen_options.sndbuf/rcvbuf to 8192 doesn't really help (binary seems to have went down, but other_system not really):

Reporting memory breakdown on node rabbit@michal-rocky-9...
other_system: 1623.3944 mb (73.15%)
allocated_unused: 516.8002 mb (23.29%)
code: 35.5915 mb (1.6%)
other_proc: 19.7732 mb (0.89%)
connection_other: 4.5993 mb (0.21%)
plugins: 3.9775 mb (0.18%)
other_ets: 3.6588 mb (0.16%)
connection_readers: 2.7359 mb (0.12%)
queue_procs: 2.4018 mb (0.11%)
mgmt_db: 1.5298 mb (0.07%)
binary: 1.4887 mb (0.07%)
atom: 1.4257 mb (0.06%)
metrics: 0.9148 mb (0.04%)
connection_channels: 0.4848 mb (0.02%)
mnesia: 0.3771 mb (0.02%)
connection_writers: 0.1507 mb (0.01%)
msg_index: 0.0308 mb (0.0%)
quorum_ets: 0.0237 mb (0.0%)
stream_queue_procs: 0.0027 mb (0.0%)
stream_queue_replica_reader_procs: 0.0027 mb (0.0%)
queue_slave_procs: 0.0 mb (0.0%)
quorum_queue_procs: 0.0 mb (0.0%)
stream_queue_coordinator_procs: 0.0 mb (0.0%)
reserved_unallocated: 0.0 mb (0.0%)

[lukebakken]

@mnasiadka see these items:

rabbitmq/cluster-operator#959 (comment)

docker-library/rabbitmq#545

There is a chance that the maximum number of file descriptors available to the Erlang VM is very high, in which case the VM will request a large amount of memory.

While RabbitMQ is running and consuming this memory, please run this command on the same VM:

rabbitmqctl eval 'os:cmd("ulimit -Hn").'

[lukebakken]

Where in the world did the 1073741816 value come from, if you can determine that?

[mnasiadka]

There's an article on RedHat website - https://access.redhat.com/solutions/1479623 - which points to this systemd commit systemd/systemd@a8b627a

[yoctozepto]

Does it mean we should utilise memcg now? Can that solve this problem too?

[lukebakken]

@yoctozepto I doubt it. The Erlang VM pre-allocates memory for the port table based on the number of available file descriptors to it. You must ensure that the number of available descriptors is a reasonable value, either via Pam, systemd, or some other system limiting function.

[yoctozepto]

Ack, thanks, @lukebakken