Something is locking the memory mappings

[Stebalien]

So, badger mmaps files and leaves them mapped. While I can't find anything in badger that explicitly locks these mappings into memory, something is locking ranges (go?) and it's killing my memory.

To reproduce, write a bunch of data (gigabytes) into a badger datastore and then read them out.

[Stebalien]

Interestingly, I can't find anything in go-ipfs locking pages into memory. However, /proc/$PID/smaps clearly shows a bunch of locked badgerdb mmaps.

[Stebalien]

And /proc/$PID/status is showing 0 kB locked...

[Stebalien]

I give up. Even after hours of inactivity, something's still locking 1.3 GiB of mmaped badger memory and I can't find anything in the go compiler/runtime/libraries or any of our libraries that could possibly making the necessary syscalls (and nothing in my strace runs either).

[schomatis]

leaves them mapped.

Until the value log file is closed or GC'ed.

While I can't find anything in badger that explicitly locks these mappings into memory,

I doesn't (AFAIK).

However, /proc/$PID/smaps clearly shows a bunch of locked badgerdb mmaps.

In a normal ipfs add session with a badger datastore my smaps reports the log file mapped to memory but zero Locked bytes.

7f19b8000000-7f1a38000000 r--s 00000000 08:01 797983                     ~/.ipfs/badgerds/000000.vlog
Size:            2097152 kB
Rss:                  24 kB
Pss:                  24 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:        24 kB
Private_Dirty:         0 kB
Referenced:           24 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd sh mr mw me ms rr sd 

(For some reason Badger mmaps twice the value log file size, i.e., 2 GB.)

Even after hours of inactivity, something's still locking 1.3 GiB of mmaped badger memory

Could you provide me a script to deterministically reproduce your use case?

[Stebalien]

Doing an ipfs refs -r $some_large_hash on the latest master does it for me:

6878dfdb8000-68795fdb8000 r--s 00000000 00:1e 3165606                    /var/lib/ipfs/badgerds/000005.vlog
Size:            2097152 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:               26884 kB
Pss:               26884 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:     26884 kB
Private_Dirty:         0 kB
Referenced:         7392 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:            26884 kB
VmFlags: rd sh mr mw me ms rr 
68795fdb8000-68799fe38000 r--s 00000000 00:1e 3016919                    /var/lib/ipfs/badgerds/000004.vlog
Size:            1049088 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:              101972 kB
Pss:              101972 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:    101972 kB
Private_Dirty:         0 kB
Referenced:        23584 kB
Anonymous:             0 kB
LazyFree:              0 kB
AnonHugePages:         0 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:           101972 kB
VmFlags: rd mr me ms rr 

Rollup:

00400000-7dccb7b64000 ---p 00000000 00:00 0                              [rollup]
Rss:             1088980 kB
Pss:             1086850 kB
Shared_Clean:       2220 kB
Shared_Dirty:          0 kB
Private_Clean:    727540 kB
Private_Dirty:    359220 kB
Referenced:       430036 kB
Anonymous:        359220 kB
LazyFree:              0 kB
AnonHugePages:      2048 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:          1086850 kB

Name:   ipfs
Umask:  0022
State:  S (sleeping)
Tgid:   5865
Ngid:   0
Pid:    5865
PPid:   1
TracerPid:      0
Uid:    989     989     989     989
Gid:    989     989     989     989
FDSize: 1024
Groups: 989 
NStgid: 5865
NSpid:  5865
NSpgid: 5865
NSsid:  5865
VmPeak: 12388256 kB
VmSize: 12388256 kB
VmLck:         0 kB                            <--- ???
VmPin:         0 kB
VmHWM:   4578724 kB
VmRSS:   1074884 kB
RssAnon:          349880 kB
RssFile:          725004 kB
RssShmem:              0 kB
VmData:  2978592 kB
VmStk:       132 kB
VmExe:     21496 kB
VmLib:      2596 kB
VmPTE:     11652 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
Threads:        290
SigQ:   0/63336
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: fffffffe3bfa3a00
SigIgn: 0000000000000000
SigCgt: ffffffffffc1feff
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003ff7fdffff
CapAmb: 0000000000000000
NoNewPrivs:     1
Seccomp:        2
Speculation_Store_Bypass:       vulnerable
Cpus_allowed:   f
Cpus_allowed_list:      0-3
Mems_allowed:   1
Mems_allowed_list:      0
voluntary_ctxt_switches:        689
nonvoluntary_ctxt_switches:     21

[schomatis]

@Stebalien Sorry, I still can't reproduce this, could you provide me a specific hash (or set of commands to produce that hash). Also, I have no idea what the rollup is.

[Stebalien]

The rollup is just /proc/pid/smaps_rollup (showing totals). ipfs refs -r /ipns/dist.ipfs.io/ >/dev/null. However, I don't think it's hash specific. I think it's my kernel. I'm running arch with the linux-hardened kernel and probably have some buggy memory hardening patch.

[Stebalien]

My kernel appears to be unrelated. Could you try on tmpfs? Thinking this might be due to my filesystem setup (btrfs with CoW and compression), I tested IPFS in /tmp. However, now I'm thinking that tmpfs may have the same issue.

[schomatis]

Still can't reproduce. I've mounted the IPFS_PATH on a tmpfs. I don't have an smaps_rollup file, I'm using an (old) Ubuntu 16.04 with a vanilla kernel,

Linux 4.13.0-36-generic #40~16.04.1-Ubuntu SMP Fri Feb 16 23:25:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

but none of the Locked: values reported by smaps ever gets up from 0 kB.

Knowing now the smaps_rollup variant and looking at your previous outputs again, the smaps is showing that the .vlog files are only locking a small part of their sizes which seems reasonable (although I can't reproduce it) but the rollup is showing a bigger locked memory size, were those outputs taken during the same run, if so, which mapping has the big locked memory difference?

[Stebalien]

were those outputs taken during the same run, if so, which mapping has the big locked memory difference?

IIRC, I had other vlogs mapped as well that were using more memory (those were just a sample). I'm running 4.16 so that may make a difference.

[Stebalien]

So, I'm no longer seeing this. Most pages now "private clean". Not exactly sure what happened.