Failed to run on CentOS 8 due to page size differences

[DocLM]

Vector Version

0.10.0

Vector Configuration File

Debug Output

Expected Behavior

Vector run on AWS t4g instance

Actual Behavior

Vector fail to run with error

<jemalloc>: Unsupported system page size
memory allocation of 32 bytes failed
Aborted

Example Data

Additional Context

Installed Vector from archive or RPM package on AWS t4g instance (Graviton 2 ARM64) processor.
AMI: Centos 8

References

[jamtur01]

Hi @DocLM - thanks for reporting! Are you using the 0.10 release or the nightly builds or...?

[jszwedko]

I am able to reproduce this with vector 0.10.0 on Centos 8 (ami-0c7ec43a152e9c107) using ARM64. Interestingly, I do not get this error using Amazon Linux 2 with ARM64 so I'm wondering if it is something specific to Centos8.

[jszwedko]

It looks like Amazon Linux has page sizes of 4 kB while Centos 8 has 64 kB pages:

Amazon Linux:

cd /proc/1 && sudo grep -i pagesize smaps | sort | uniq
KernelPageSize:        4 kB
MMUPageSize:           4 kB

Centos:

 cd /proc/1 && sudo grep -i pagesize smaps | sort | uniq
KernelPageSize:       64 kB
MMUPageSize:          64 kB

It looks like we may need to compile vector with larger page sizes? It looks like, by default, jemallac uses the page sizes of the build system:

JEMALLOC_SYS_WITH_LG_PAGE=: Specify the base 2 log of the allocator page size, which must in turn be at least as large as the system page size. By default the configure script determines the host's page size and sets the allocator page size equal to the system page size, so this option need not be specified unless the system page size may change between configuration and execution, e.g. when cross compiling.

https://github.com/gnzlbg/jemallocator/blob/01b2dc89238a35b38f8d7b62c2ef876ed7641b2f/jemalloc-sys/README.md#environment-variables

It's unclear to me if we should do that generally or specifically for CentOS builds.

[DocLM]

Hi @DocLM - thanks for reporting! Are you using the 0.10 release or the nightly builds or...?

@jamtur01 I tried v0.10.0 and also nightly but the outcome is the same.

It's unclear to me if we should do that generally or specifically for CentOS builds.

@jszwedko Honestly I don't know. Yesterday I built it from source as a workaround but if bigger pages also works on systems with a lower default it can be a more inclusive approach.
Obviously an official rpm package is my preferred choice.

[jamtur01]

@jszwedko Not sure either. Maybe @Hoverbear knows?

[Hoverbear]

#3813 just closed a few hours ago. Do you think you could try with one of the next nightlies or from an updated main branch again? Otherwise I can take a look at this tomorrow. :) Thank you for reporting it.

[DocLM]

#3813 just closed a few hours ago. Do you think you could try with one of the next nightlies or from an updated main branch again? Otherwise I can take a look at this tomorrow. :) Thank you for reporting it.

I tried a moment ago (7.30 AM CET) with the latest nightly I found on the site (2020-10-08) without luck.

[jamtur01]

@DocLM Sorry - we've got a hiccup with our nightlies post #3813. I'll ping you when we have new builds. Hopefully soon'ish.

[jamtur01]

@DocLM New nightly images here: https://github.com/timberio/vector/actions/runs/294876413

[DocLM]

@DocLM New nightly images here: https://github.com/timberio/vector/actions/runs/294876413

@jamtur01 I tried a moment ago. It's not working with the same errors.

[DocLM]

I tried the official rpm for Vector 0.14.0 on Centos 8 with Graviton 2 and now works out of the box.

[jszwedko]

Thanks for following up @DocLM ! This must have changed with 0.14.0 when we dropped jemalloc as the default allocator.

[sinisterstumble]

@DocLM this is still happening on Centos 8 Stream with vector-0.19.0.

# uname -a
Linux ip-172-16-30-81.eu-central-1.compute.internal 4.18.0-358.el8.aarch64 #1 SMP Mon Jan 10 13:06:14 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
# /bin/vector
<jemalloc>: Unsupported system page size
memory allocation of 32 bytes failed
Aborted (core dumped)

[jszwedko]

Ah, doh, I completely forgot about this Centos 8 issue when we re-enabled jemalloc in 0.19.0. I'll re-open this for tracking.

[jszwedko]

I confirmed that building Vector with 64 KiB pages seems to run on systems with both 4 KiB pages and 64 KiB pages. I'm not sure what trade-offs are, though, or if they warrant a separate build specifically with large pages.

[sinisterstumble]

@jszwedko another thing to consider: EL9, Centos Stream 9, RHEL 9 Beta &c. have moved back to 4 kB.

[jszwedko]

That is good to know, thank you @markmartirosian !

Just for reference, in case users want to compile Vector themselves for systems with 64 K page sizes, you can use:

JEMALLOC_SYS_WITH_LG_PAGE=16 make build-aarch64-unknown-linux-gnu

once #10940 is merged.

[sinisterstumble]

@jszwedko thank you, that will help with building my own rpm.

[shumphrey]

I would love to see a precompiled version that works for EL8 (centos8/rockylinux8) on aarch64

[jszwedko]

Just a note that I am spiking into this via #18481 to just increase the page size generally since Vector compiled with a large page size also runs on systems with a smaller default page size. We just want to verify this doesn't negatively impact performance or memory usage.