This repository has been archived by the owner on Oct 16, 2020. It is now read-only.
High kernel "SUnreclaimable" memory usage after around 5-7 days of uptime #2637
Comments
|
Thanks for the report. Is there a previous Container Linux version where you weren't seeing this problem? |
I can't remember any. IIRC I've seen this problem since I initially set up the VM since July/August. |
|
We have the same issue with some Container Linux VMs on VMware. # uptime
# cat /proc/meminfo
# slabtop --sort c -o
@bgilbert: The problems started when we upgraded from 2135.6.0 to 2191.4.1.
|
|
I have the same problem on |
|
This problem seems still to exist on
I have another instance running, which has much higher load (number of processes) but it does not suffer from the problem. The only configuration difference between the affected and non-affected instance is that the affected one listens for If I firewall port 443 on the affected machine the This looks like a kernel bug to me. Or can With the remaining lifetime of CoreOS I am not holding my breath to get this fixed. But I don't think this is really a distro bug, it's probably upstream somewhere. If somebody can provide a link to any upstream bug/fix that would be useful information. |
|
Same issue here, my release is: and my slabinfo: The |
|
Maybe the guys from @flatcar-linux can have a look on this issue? |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Issue Report
Bug
The kernel's memory usage on a DigitalOcean's 1 GB droplet with CoreOS rises up to an unreasonable level after leaving the machine on for around 5-7 days, making it unresponsive and sometimes trigger OOM.
The machine is not running Kubernetes but is a (single host) Docker swarm manager. It has only 3 containers running.
Container Linux Version
Environment
DigitalOcean cloud provider, Singapore POP, 1 GB machine.
Expected Behavior
The kernel uses a reasonable amount of memory. SClaimable may grow significantly, but SUnreclaimable shouldn't rise above 100 MB.
Actual Behavior
The kernel's unclaimable memory goes to almost 300 MB, while claimable memory stays around 57 MB.
sudo slabtop --sort cshows that the top consumer iscred_jarat around 74 MB.Reproduction Steps
cat /proc/meminfoandsudo slabtop --sort c.Other Information
This is the result of
cat /proc/meminfoandsudo slabtop --sort c:cat /proc/meminfo
sudo slabtop --sort c
Active / Total Objects (% used) : 3054935 / 3075343 (99.3%) Active / Total Slabs (% used) : 69313 / 69313 (100.0%) Active / Total Caches (% used) : 101 / 130 (77.7%) Active / Total Size (% used) : 345078.60K / 351209.45K (98.3%) Minimum / Average / Maximum Object : 0.01K / 0.11K / 23.25K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 392847 392662 99% 0.19K 18707 21 74828K cred_jar 24855 24855 100% 1.06K 1657 15 26512K mm_struct 329856 329299 99% 0.06K 5154 64 20616K anon_vma_chain 32727 32389 98% 0.59K 2518 13 20144K inode_cache 79504 79362 99% 0.25K 4969 16 19876K filp 600064 600064 100% 0.03K 4688 128 18752K kmalloc-32 39360 39360 100% 0.38K 3936 10 15744K kmem_cache 7848 7836 99% 2.00K 981 8 15696K kmalloc-2048 159160 159160 100% 0.09K 3460 46 13840K anon_vma 18821 18811 99% 0.69K 1711 11 13688K sock_inode_cache 13448 13448 100% 1.00K 1681 8 13448K UNIX 96720 96720 100% 0.13K 3224 30 12896K kernfs_node_cache 58254 51782 88% 0.19K 2774 21 11096K dentry 2336 2334 99% 4.00K 292 8 9344K kmalloc-4096 15768 15764 99% 0.50K 1971 8 7884K kmalloc-512 53888 53888 100% 0.12K 1684 32 6736K pid 26816 26736 99% 0.25K 1676 16 6704K skbuff_head_cache 853504 853504 100% 0.01K 1667 512 6668K kmalloc-8 9506 7982 83% 0.57K 679 14 5432K radix_tree_node 5100 5099 99% 1.00K 638 8 5104K kmalloc-1024 4275 2390 55% 1.05K 285 15 4560K ext4_inode_cache 47292 47092 99% 0.09K 1126 42 4504K kmalloc-96 279 254 91% 9.25K 93 3 2976K task_struct 39488 39488 100% 0.06K 617 64 2468K kmem_cache_node 36416 36272 99% 0.06K 569 64 2276K kmalloc-64 7938 6512 82% 0.19K 378 21 1512K kmalloc-192 1705 1408 82% 0.69K 155 11 1240K shmem_inode_cache 10335 9796 94% 0.10K 265 39 1060K buffer_head 1404 1264 90% 0.66K 118 12 944K proc_inode_cache 3584 3584 100% 0.25K 224 16 896K kmalloc-256 21522 15903 73% 0.04K 211 102 844K numa_policy 96 75 78% 8.00K 24 4 768K kmalloc-8192 300 300 100% 2.06K 20 15 640K sighand_cache 5024 5024 100% 0.12K 157 32 628K kmalloc-128 19210 19210 100% 0.02K 113 170 452K avtab_node 390 390 100% 1.06K 26 15 416K signal_cache 52 34 65% 8.00K 13 4 416K biovec-max 26368 26368 100% 0.02K 103 256 412K kmalloc-16 588 415 70% 0.66K 49 12 392K ovl_inode 910 853 93% 0.38K 91 10 364K mnt_cache 4032 4032 100% 0.06K 63 64 252K ebitmap_node 98 98 100% 2.25K 7 14 224K TCPv6 1029 1029 100% 0.19K 49 21 196K proc_dir_entry 48 24 50% 4.00K 6 8 192K names_cache 242 242 100% 0.69K 22 11 176K files_cache 3485 3485 100% 0.05K 41 85 164K ftrace_event_field 75 75 100% 2.12K 5 15 160K TCP 3570 3162 88% 0.04K 35 102 140K ext4_extent_status 1380 1380 100% 0.09K 30 46 120K trace_event_file 870 870 100% 0.13K 29 30 116K dm_bufio_buffer-4 168 168 100% 0.62K 14 12 112K dio 98 98 100% 1.12K 7 14 112K RAWv6 1344 1344 100% 0.07K 24 56 96K Acpi-Operand 110 110 100% 0.75K 11 10 88K task_group 1120 1120 100% 0.07K 20 56 80K eventpoll_pwq 60 60 100% 1.25K 5 12 80K UDPv6 110 110 100% 0.69K 10 11 80K bio-2 216 170 78% 0.31K 18 12 72K nf_conntrack 204 156 76% 0.31K 17 12 68K xfrm_dst_cache 256 256 100% 0.25K 16 16 64K pool_workqueue 10 10 100% 5.50K 2 5 64K net_namespace 26 26 100% 2.40K 2 13 64K request_queue 3328 3328 100% 0.02K 13 256 52K selinux_file_security 416 416 100% 0.12K 13 32 52K scsi_sense_cache 408 408 100% 0.12K 12 34 48K jbd2_journal_head 1122 1122 100% 0.04K 11 102 44K pde_opener 88 88 100% 0.50K 11 8 44K skbuff_fclone_cache 231 231 100% 0.19K 11 21 44K dmaengine-unmap-16 187 187 100% 0.23K 11 17 44K tw_sock_TCP 187 187 100% 0.23K 11 17 44K tw_sock_TCPv6 40 40 100% 0.94K 5 8 40K RAW 459 459 100% 0.08K 9 51 36K inotify_inode_mark 38 27 71% 0.81K 2 19 32K bdev_cache 15 15 100% 2.06K 1 15 32K dmaengine-unmap-256 8 8 100% 4.00K 1 8 32K sgpool-128 91 91 100% 0.30K 7 13 28K blkdev_requests 196 196 100% 0.14K 7 28 28K ext4_groupinfo_4k 24 24 100% 1.00K 3 8 24K biovec-64 438 438 100% 0.05K 6 73 24K mbcache 15 15 100% 1.06K 1 15 16K dmaengine-unmap-128 8 8 100% 2.00K 1 8 16K biovec-128 52 52 100% 0.30K 4 13 16K request_sock_TCP 128 128 100% 0.12K 4 32 16K secpath_cache 512 512 100% 0.03K 4 128 16K fscrypt_info 22 22 100% 0.72K 2 11 16K fat_inode_cache 153 153 100% 0.08K 3 51 12K Acpi-State 219 219 100% 0.05K 3 73 12K Acpi-Parse 117 117 100% 0.10K 3 39 12K blkdev_ioc 192 192 100% 0.06K 3 64 12K ext4_io_end 24 24 100% 0.32K 2 12 8K taskstats 40 40 100% 0.20K 2 20 8K file_lock_cache 13 13 100% 0.61K 1 13 8K hugetlbfs_inode_cache 8 8 100% 0.94K 1 8 8K mqueue_inode_cache 9 9 100% 0.43K 1 9 4K uts_namespace 25 25 100% 0.16K 1 25 4K sigqueue 16 16 100% 0.25K 1 16 4K dquot 19 19 100% 0.20K 1 19 4K ip4-frags 13 13 100% 0.30K 1 13 4K request_sock_TCPv6 85 85 100% 0.05K 1 85 4K fscrypt_ctx 256 256 100% 0.02K 1 256 4K jbd2_revoke_table_s 32 32 100% 0.12K 1 32 4K ext4_allocation_context 0 0 0% 0.09K 0 42 0K dma-kmalloc-96 0 0 0% 0.19K 0 21 0K dma-kmalloc-192 0 0 0% 0.01K 0 512 0K dma-kmalloc-8 0 0 0% 0.02K 0 256 0K dma-kmalloc-16 0 0 0% 0.03K 0 128 0K dma-kmalloc-32 0 0 0% 0.06K 0 64 0K dma-kmalloc-64 0 0 0% 0.12K 0 32 0K dma-kmalloc-128 0 0 0% 0.25K 0 16 0K dma-kmalloc-256 0 0 0% 0.50K 0 8 0K dma-kmalloc-512 0 0 0% 1.00K 0 8 0K dma-kmalloc-1024 0 0 0% 2.00K 0 8 0K dma-kmalloc-2048 0 0 0% 4.00K 0 8 0K dma-kmalloc-4096 0 0 0% 8.00K 0 4 0K dma-kmalloc-8192 0 0 0% 0.03K 0 128 0K avc_xperms_data 0 0 0% 0.12K 0 34 0K iint_cache 0 0 0% 0.45K 0 17 0K user_namespace 0 0 0% 0.94K 0 8 0K PING 0 0 0% 0.23K 0 17 0K posix_timers_cache 0 0 0% 0.03K 0 128 0K dnotify_struct 0 0 0% 0.12K 0 34 0K cfq_io_cq 0 0 0% 0.44K 0 9 0K bfq_queue 0 0 0% 1.12K 0 14 0K PINGv6 0 0 0% 2.57K 0 12 0K dm_uevent 0 0 0% 3.23K 0 9 0K kcopyd_job 0 0 0% 0.16K 0 24 0K kvm_mmu_page_header 0 0 0% 23.25K 0 1 0K kvm_vcpu 0 0 0% 0.13K 0 30 0K kvm_async_pf 0 0 0% 0.04K 0 102 0K fat_cache 0 0 0% 0.21K 0 18 0K nf_conntrack_expectAnd this is kernel messages, captured about 6 hours later compared to the 2 logs above:
dmesg
The text was updated successfully, but these errors were encountered: