Out of Memory error while running address sanitizer

[jrobert1]

I am running a C++ program that will allocate about 50-60GB RAM on a server that has 96GB of RAM. I we get the error below when our process allocates about 30GB of resident memory:

Out of memory. Dying.
The process has exhausted 8192MB for size class 2304.

Below are various system settings:

GCC version:
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.4-2ubuntu114.04' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu114.04)

$ ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 71679
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 32000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 770126
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

$ sudo sysctl -a | grep vm
fs.binfmt_misc.llvm-3/4-runtime/binfmt = enabled
fs.binfmt_misc.llvm-3/4-runtime/binfmt = interpreter /usr/bin/lli-3.4
fs.binfmt_misc.llvm-3/4-runtime/binfmt = flags:
fs.binfmt_misc.llvm-3/4-runtime/binfmt = offset 0
fs.binfmt_misc.llvm-3/4-runtime/binfmt = magic 4243
fs.binfmt_misc.jar = interpreter /usr/lib/jvm/java-7-oracle-1.7.0.80/jre/lib/jexec
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 0
vm.extfrag_threshold = 500
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 256 256 32
vm.max_map_count = 65530
vm.memory_failure_early_kill = 0
vm.memory_failure_recovery = 1
vm.min_free_kbytes = 90112
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.mmap_min_addr = 65536
vm.nr_hugepages = 0
vm.nr_hugepages_mempolicy = 0
vm.nr_overcommit_hugepages = 0
vm.nr_pdflush_threads = 0
vm.numa_zonelist_order = default
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 300
vm.page-cluster = 3
vm.panic_on_oom = 1
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 1
vm.user_reserve_kbytes = 131072
vm.vfs_cache_pressure = 100
vm.zone_reclaim_mode = 0

[kcc]

gcc version 4.8.4 is quite old.
Does this happen with the fresh clang? (it shouldn't)

[jrobert1]

Yes migrated to clang3.6 and that fixed the issue

[codetaku]

I'm having this problem and don't have the latitude to migrate the entire project to use clang. Is there a way to get a gcc version for which this issue will not happen in Ubuntu 12.04?

[kcc]

Just build the fresh GCC from sources, see gcc.gnu.org

[codetaku]

I was worried that that would be the answer. Thanks.

[ghost]

BTW you can probably try (unofficial) GCC 4.9 for Pangolin (see e.g. http://askubuntu.com/questions/428198/getting-installing-gcc-g-4-9-on-ubuntu).

[mrichtarsky]

I am getting this error when running a large stress test. I think it is a real exhaustion of this allocator. The machine has 256GB RAM.

AddressSanitizer: Out of memory. Dying. The process has exhausted 65536MB for size class 81920.
=================================================================

This is clang version 3.7.0 (tags/RELEASE_370/final).

My usual workaround of starting the involved process freshly at the beginning does not work here, the error still occurs after the test has been running for a couple of minutes.

Looking at the ASan runtime, changing the size of the size class will definitely need a rebuild of clang. Still it does not look straightforward - or is it sufficient to tweak only one constant?

[Brook108]

I am also getting this error when having large available memory with gcc 4.8.5, but i need to use gcc 4.8.5 version. How can i fix the issue on gcc 4.8.5.