Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

oom killed #555

Open
MidnightRAT opened this issue Jun 21, 2022 · 23 comments
Open

oom killed #555

MidnightRAT opened this issue Jun 21, 2022 · 23 comments

Comments

@MidnightRAT
Copy link

started with ulimit -n 65535

[101236.553584] app invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[101236.649900] CPU: 0 PID: 5641 Comm: app Kdump: loaded Not tainted 5.4.17-2136.307.3.4.el8uek.x86_64 #2
[101236.651576] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.5.1 06/16/2021
[101236.749970] Call Trace:
[101236.751757] dump_stack+0x6d/0x8b
[101236.752512] dump_header+0x4f/0x1e1
[101236.848687] oom_kill_process.cold.33+0xb/0x10
[101236.852238] out_of_memory+0x1bf/0x551
[101236.948560] __alloc_pages_slowpath+0xd40/0xe24
[101236.949738] __alloc_pages_nodemask+0x2df/0x326
[101236.950962] alloc_pages_current+0x81/0xd6
[101236.952591] __page_cache_alloc+0x6a/0x94
[101236.953354] pagecache_get_page+0xbe/0x2d9
[101237.048339] filemap_fault+0x7b5/0xa87
[101237.049089] ? __mod_memcg_lruvec_state+0x27/0x102
[101237.050099] ? unlock_page_memcg+0x12/0x14
[101237.050857] ? _cond_resched+0x19/0x29
[101237.051543] ? down_read+0x12/0x98
[101237.052204] __xfs_filemap_fault+0x6f/0x210 [xfs]
[101237.053046] ? filemap_map_pages+0x28d/0x3a7
[101237.053849] xfs_filemap_fault+0x37/0x40 [xfs]
[101237.148929] __do_fault+0x3c/0xd6
[101237.149586] __handle_mm_fault+0xa71/0xd5d
[101237.150355] handle_mm_fault+0xc9/0x1f0
[101237.151064] __do_page_fault+0x1f7/0x4b7
[101237.151791] do_page_fault+0x36/0x11a
[101237.152499] do_async_page_fault+0x64/0xbe
[101237.248368] ? prepare_exit_to_usermode+0xa5/0xce
[101237.249245] async_page_fault+0x13d/0x142
[101237.249935] RIP: 0033:0x464a40
[101237.250536] Code: Bad RIP value.
[101237.251116] RSP: 002b:000000c000047f30 EFLAGS: 00010202
[101237.252079] RAX: 0000000000002710 RBX: 0000000000002710 RCX: 0000000000004e20
[101237.253373] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00005c12a210c7f5
[101237.348751] RBP: 000000c000047f90 R08: 0000000000016159 R09: 000000c000047f00
[101237.350025] R10: 0000000000000000 R11: 0000000000000212 R12: 000000c000047958
[101237.351335] R13: 000000c000038000 R14: 000000c0000024e0 R15: 00007f4462f98134
[101237.352680] Mem-Info:
[101237.448297] active_anon:60964 inactive_anon:61225 isolated_anon:0
active_file:439 inactive_file:554 isolated_file:17
unevictable:3092 dirty:0 writeback:0 unstable:0
slab_reclaimable:6330 slab_unreclaimable:16484
mapped:3385 shmem:57 pagetables:3784 bounce:0
free:7784 free_pcp:343 free_cma:0
[101237.751974] Node 0 active_anon:243860kB inactive_anon:244900kB active_file:1264kB inactive_file:2716kB unevictable:12368kB isolated(anon):0kB isolated(file):0kB mapped:11972kB dirty:0kB writeback:0kB shmem:228kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 2048kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[101237.952274] Node 0 DMA free:2876kB min:712kB low:888kB high:1064kB active_anon:4256kB inactive_anon:6220kB active_file:32kB inactive_file:104kB unevictable:0kB writepending:0kB present:15004kB managed:14372kB mlocked:0kB kernel_stack:64kB pagetables:196kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[101238.052569] lowmem_reserve[]: 0 563 563 563 563
[101238.149378] Node 0 DMA32 free:28436kB min:28620kB low:35772kB high:42924kB active_anon:239604kB inactive_anon:238680kB active_file:1744kB inactive_file:2876kB unevictable:12368kB writepending:0kB present:1027952kB managed:660636kB mlocked:12368kB kernel_stack:3904kB pagetables:14944kB bounce:0kB free_pcp:1240kB local_pcp:716kB free_cma:0kB
[101238.348628] lowmem_reserve[]: 0 0 0 0 0
[101238.349898] Node 0 DMA: 624kB (UM) 318kB (UM) 2116kB (UM) 932kB (UM) 064kB 0128kB 1256kB (M) 1512kB (M) 11024kB (M) 02048kB 04096kB = 2912kB
[101238.448614] Node 0 DMA32: 4354kB (UME) 4968kB (UME) 32116kB (UME) 9732kB (ME) 4064kB (UME) 27128kB (UME) 12256kB (UM) 8512kB (UM) 21024kB (UM) 02048kB 04096kB = 29180kB
[101238.452727] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[101238.550119] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[101238.552333] 3818 total pagecache pages
[101238.553539] 72 pages in swap cache
[101238.649312] Swap cache stats: add 90806805, delete 90806621, find 2572123/47328139
[101238.651240] Free swap = 0kB
[101238.652337] Total swap = 1397756kB
[101238.653517] 260739 pages RAM
[101238.748902] 0 pages HighMem/MovableOnly
[101238.750411] 91987 pages reserved
[101238.751559] 0 pages cma reserved
[101238.752723] 0 pages hwpoisoned
[101238.849090] Tasks state (memory values in pages):
[101238.850776] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[101238.856406] [ 1042] 0 1042 26645 961 217088 280 0 systemd-journal
[101238.950800] [ 1080] 0 1080 27270 823 225280 987 -1000 systemd-udevd
[101239.049048] [ 1377] 32 1377 16805 964 172032 176 0 rpcbind
[101239.051282] [ 1378] 0 1378 18749 388 147456 151 -1000 auditd
[101239.148525] [ 1403] 996 1403 4404 359 65536 31 0 lsmd
[101239.151268] [ 1404] 997 1404 408066 580 344064 2584 0 polkitd
[101239.153828] [ 1405] 0 1405 12570 578 143360 265 0 smartd
[101239.250177] [ 1408] 992 1408 13123 317 135168 132 0 chronyd
[101239.252243] [ 1410] 81 1410 21283 835 167936 131 -900 dbus-daemon
[101239.254408] [ 1412] 0 1412 100563 998 401408 365 0 sssd
[101239.349668] [ 1418] 0 1418 31223 419 135168 130 0 irqbalance
[101239.351845] [ 1448] 0 1448 131786 1284 520192 6036 0 firewalld
[101239.353960] [ 1455] 0 1455 23584 991 204800 507 0 systemd-logind
[101239.450418] [ 1496] 0 1496 150418 1278 389120 1069 0 NetworkManager
[101239.452658] [ 1502] 0 1502 155985 1059 425984 3036 0 tuned
[101239.549978] [ 1508] 0 1508 77894 393 176128 173 0 gssproxy
[101239.552237] [ 1851] 0 1851 20996 1190 196608 497 0 dhclient
[101239.649791] [ 1979] 0 1979 14345 3072 159744 0 -17 iscsid
[101239.748291] [ 1980] 990 1980 309935 543 196608 2539 0 updater
[101239.750463] [ 1982] 0 1982 59010 381 90112 43 0 OSWatcher
[101239.752550] [ 1984] 991 1984 310101 690 208896 2964 0 agent
[101239.754577] [ 1987] 0 1987 57421 487 225280 530 0 rsyslogd
[101239.850123] [ 1994] 0 1994 23102 1076 204800 201 -1000 sshd
[101239.948190] [ 2059] 0 2059 56498 339 69632 32 0 agetty
[101239.953909] [ 2060] 0 2060 56588 307 69632 31 0 agetty
[101240.051365] [ 2062] 0 2062 61687 293 114688 169 0 crond
[101240.149554] [ 2064] 991 2064 328809 1331 233472 2364 0 gomon
[101240.151524] [ 2067] 0 2067 11008 464 114688 73 0 atd
[101240.249618] [ 2152] 0 2152 89327 711 311296 273 0 sudo
[101240.251498] [ 2219] 0 2219 22377 757 212992 365 0 systemd
[101240.348609] [ 2228] 0 2228 93053 1 311296 1375 0 (sd-pam)
[101240.350696] [ 2254] 0 2254 2165 207 65536 38 0 osms-agent
[101240.448427] [ 2257] 0 2257 392057 0 634880 22502 0 osms-agent
[101240.451436] [ 2582] 0 2582 89327 675 303104 272 0 sudo
[101240.454929] [ 2587] 989 2587 22350 846 204800 366 0 systemd
[101240.549370] [ 2589] 989 2589 96642 1 315392 1383 0 (sd-pam)
[101240.552724] [ 2595] 989 2595 310399 645 208896 2490 0 runcommand
[101240.648889] [ 5361] 0 5361 41480 1324 327680 332 0 sshd
[101240.754820] [ 5367] 1000 5367 22379 770 212992 372 0 systemd
[101240.849961] [ 5370] 1000 5370 133508 1 331776 1393 0 (sd-pam)
[101240.851812] [ 5376] 1000 5376 41536 614 311296 378 0 sshd
[101240.949089] [ 5377] 1000 5377 58857 428 98304 481 0 bash
[101240.951001] [ 5515] 1000 5515 58857 65 77824 481 0 bash
[101241.051001] [ 5522] 1000 5522 473989 59914 2592768 168092 0 db1000n
[101241.149031] [ 5633] 1000 5633 177109 828 143360 1523 0 launcher-disbal
[101241.248209] [ 5640] 1000 5640 177639 661 118784 1264 0 app
[101241.348277] [ 332172] 0 332172 360142 55754 2113536 116225 0 dnf
[101241.453130] [ 333542] 0 333542 23450 414 204800 18 0 sssd_nss
[101241.549421] [ 333546] 0 333546 58944 391 90112 12 0 nfssub
[101241.653268] [ 333560] 0 333560 25050 433 217088 19 0 sssd_be
[101241.655693] [ 333565] 0 333565 60051 800 106496 54 0 nfsiostat
[101241.749907] [ 333605] 0 333605 18564 571 180224 15 0 sshd
[101241.753146] [ 333615] 0 333615 18564 517 176128 7 0 sshd
[101241.850937] [ 333619] 0 333619 68537 598 147456 172 0 crond
[101241.948481] [ 333632] 0 333632 68537 528 147456 158 0 crond
[101242.049578] [ 333635] 0 333635 55630 266 77824 1 0 sh
[101242.051493] [ 333639] 0 333639 55630 55 69632 2 0 sh
[101242.149457] [ 333641] 0 333641 59952 699 106496 16 0 ksplice
[101242.151602] [ 333646] 0 333646 58944 326 90112 0 0 oswnet
[101242.250548] [ 333653] 0 333653 18564 480 172032 0 0 sshd
[101242.350970] [ 333656] 0 333656 58944 290 90112 0 0 iosub
[101242.352850] [ 333658] 0 333658 15206 158 143360 0 0 sshd
[101242.449099] [ 333659] 0 333659 58944 68 73728 0 0 oswnet
[101242.450996] [ 333660] 0 333660 68537 94 131072 152 0 crond
[101242.452898] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-3.scope,task=db1000n,pid=5522,uid=1000
[101242.649253] Out of memory: Killed process 5522 (db1000n) total-vm:1895956kB, anon-rss:239656kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:2532kB oom_score_adj:0
[101243.552711] oom_reaper: reaped process 5522 (db1000n), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[417432.812358] TCP: too many orphaned sockets
@arriven
Copy link
Owner

arriven commented Jun 22, 2022

How much memory do you have?

@deputinizer
Copy link
Contributor

deputinizer commented Jun 22, 2022 •
edited
Loading

I have 2 GB of RAM and still getting OOM after a few days...

info    attacking       {"target": "http://xxie"}
info    attacking       {"target": "http://xxsa"}
info    attacking       {"target": "http://xxtil/"}
Killed

Version 0.9.9

@arriven
Copy link
Owner

arriven commented Jun 22, 2022

A few days will be interesting to debug

@deputinizer
Copy link
Contributor

deputinizer commented Jun 23, 2022 •
edited
Loading

Next day... next kill... i have to pprof before it gets killed

 |    --- |                --- |           --- |                --- |       --- |
 |  Total |           13608405 |      13336567 |            4016925 | 724.90 MB |

Killed
ubuntu@vps-123456789:~/stoppropaganda$

Latest version 0.9.10

@deputinizer
Copy link
Contributor

Also, I got a very uncomfortable email...

Is there a way to lower the rate of TCP SYN / per second ?

Dear Customer,

Abnormal activity has been detected on your VPS vps1234.ovh.net.


As this constitutes a breach of contract, your virtual server vps1234.ovh.net
has been blocked.

You will find the logs brought up by our system below, which led to this alert.

- START OF ADDITIONAL INFORMATION -

Attack detail : 8Kpps/4Mbps
dateTime srcIp:srcPort dstIp:dstPort protocol flags bytes reason
2022.06.18 02:30:52 CEST vps1234:55046 84.252.129.113:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:36158 195.128.49.210:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:59934 172.67.145.235:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:48778 195.128.137.3:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:38424 31.31.198.231:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:38466 104.21.89.179:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:58508 84.42.111.137:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:54342 31.31.198.225:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:41018 31.31.196.201:443 TCP SYN 56 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:37354 104.21.21.95:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:55052 84.252.129.113:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:58170 37.140.192.179:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:59990 172.67.145.235:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:46116 37.140.192.54:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:58364 172.67.180.134:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:41332 37.140.192.89:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:42900 185.26.122.77:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:37134 77.222.56.143:443 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:58306 172.67.180.134:80 TCP SYN 60 ATTACK:TCP_SYN
2022.06.18 02:30:52 CEST vps1234:39082 194.105.131.43:80 TCP SYN 60 ATTACK:TCP_SYN

@deputinizer
Copy link
Contributor

deputinizer commented Jun 23, 2022 •
edited
Loading

And yet another panic... funny it's on a single-thread machine 😆 so it's not race condition 😟

... 6969 more lines ...
goroutine 20488 [sleep]:
time.Sleep(0x14f46b0400)
        /opt/hostedtoolcache/go/1.18.3/x64/src/runtime/time.go:194 +0x12e
github.com/valyala/fasthttp.(*HostClient).connsCleaner(0xc00db6ad00)
        /home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1711 +0x58
created by github.com/valyala/fasthttp.(*HostClient).acquireConn
        /home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1602 +0x218

Next time running ./db1000n 2> errors.txt

@arriven
Copy link
Owner

arriven commented Jun 23, 2022

Also, I got a very uncomfortable email...

Is there a way to lower the rate of TCP SYN / per second ?

Interesting, there isn't any tcp syn attacks implemented in the app because they are generally blocked by the provider. The only thing that could be treated like this is the amount of attempts to build a connection with the target. You can try using the general rate limiting to work around that but it'll decrease the overall performance of the app (--min-interval )

@deputinizer
Copy link
Contributor

deputinizer commented Jun 24, 2022 •
edited
Loading

amount of attempts to build a connection

well that's called Dial

In stoppropaganda i've implmented rate limiting
https://github.com/erkexzcx/stoppropaganda#dialspersecond

Edit: Here was the PR 025104a#diff-f1c4adfa861d40deef193a963cc05fe1180e22fa8d2c4195c04b40b989fba4f3R238
Yeah, this PR was a little bloated 😜

@arriven
Copy link
Owner

arriven commented Jun 26, 2022

Yeah, there's already a rate limiting in place, I just didn't treat dial as a special case but maybe I should

@arriven
Copy link
Owner

arriven commented Jun 26, 2022

@deputinizer thinking about it more, I think you can already achieve the same results with existing configuration options. There's exponential backoff present on errors which you can configure to have higher initial timeout (--backoff-timeout, default is 1 microsecond) and adapt backoff-limit/backoff-multiplier accordingly so that it doesn't spin to thousands of seconds

@arriven
Copy link
Owner

arriven commented Jun 26, 2022

regarding the actual memory issue, I've been debugging todays config for couple of hours and the memory usage was only fluctuating for couple of MBs. Testing it out on my benchmark server revealed that it's possible to increase the memory usage for one of the jobs by increasing the response size from the target. We were already planning to replace that job with a more effective one, I'll update this thread once we do that

@deputinizer
Copy link
Contributor

[Spoiler] fatal error: runtime: out of memory

./db1000n 2> stderr.txt

...
info	attacking	{"target": "http://smev74.gosuslugi.ru"}
fatal error: runtime: out of memory

runtime stack:
runtime.throw({0xcc1d6f?, 0x10000000?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/panic.go:992 +0x71
runtime.sysMap(0xc020800000, 0xc00006be90?, 0xc00006bef8?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/mem_linux.go:189 +0x11b
runtime.(*mheap).grow(0x13e4980, 0x8000?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/mheap.go:1404 +0x225
runtime.(*mheap).allocSpan(0x13e4980, 0x8000, 0x0, 0x1)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/mheap.go:1170 +0x171
runtime.(*mheap).alloc.func1()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/mheap.go:912 +0x65
runtime.systemstack()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:469 +0x49

goroutine 1 [running]:
runtime.systemstack_switch()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:436 fp=0xc000651230 sp=0xc000651228 pc=0x468a40
runtime.(*mheap).alloc(0x10000000?, 0x8000?, 0xe0?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/mheap.go:906 +0x65 fp=0xc000651278 sp=0xc000651230 pc=0x428de5
runtime.(*mcache).allocLarge(0xc000651368?, 0x10000000, 0x1)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/mcache.go:213 +0x85 fp=0xc0006512c8 sp=0xc000651278 pc=0x417105
runtime.mallocgc(0x10000000, 0xb907e0, 0x1)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/malloc.go:1096 +0x5a5 fp=0xc000651340 sp=0xc0006512c8 pc=0x40d505
runtime.makeslice(0x7fbd3d8010c0?, 0x0?, 0x7fbd65a6a108?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/slice.go:103 +0x52 fp=0xc000651368 sp=0xc000651340 pc=0x44f6b2
golang.org/x/crypto/scrypt.Key({0xc009095bf0?, 0x2c, 0x30}, {0xc0187e7480, 0x2c, 0x40}, 0x40000, 0x2?, 0x1, 0x20)
	/home/runner/go/pkg/mod/golang.org/x/[email protected]/scrypt/scrypt.go:204 +0x18a fp=0xc000651420 sp=0xc000651368 pc=0x82ed2a
filippo.io/age.(*ScryptIdentity).unwrap(0xc006bafa00, 0xc00e8d5600)
	/home/runner/go/pkg/mod/filippo.io/[email protected]/scrypt.go:159 +0x2b0 fp=0xc0006514e0 sp=0xc000651420 pc=0x8302b0
filippo.io/age.(*ScryptIdentity).unwrap-fm(0x203001?)
	<autogenerated>:1 +0x2c fp=0xc000651500 sp=0xc0006514e0 pc=0x83068c
filippo.io/age.multiUnwrap(0xc000651580, {0xc0072594f0, 0x1, 0x18?})
	/home/runner/go/pkg/mod/filippo.io/[email protected]/age.go:227 +0x63 fp=0xc000651560 sp=0xc000651500 pc=0x82f823
filippo.io/age.(*ScryptIdentity).Unwrap(0xe19120?, {0xc0072594f0?, 0x0?, 0x203001?})
	/home/runner/go/pkg/mod/filippo.io/[email protected]/scrypt.go:130 +0xab fp=0xc0006515a0 sp=0xc000651560 pc=0x82ffab
filippo.io/age.Decrypt({0xe19120?, 0xc004720de0?}, {0xc000651700, 0x1, 0xc00e442030?})
	/home/runner/go/pkg/mod/filippo.io/[email protected]/age.go:194 +0x25e fp=0xc0006516b8 sp=0xc0006515a0 pc=0x82f2fe
github.com/Arriven/db1000n/src/utils.decrypt({0xc00f32c000, 0x54bb5, 0x56000}, {0xcd71a2?, 0x5ea6ec?})
	/home/runner/work/db1000n/db1000n/src/utils/crypto.go:100 +0x165 fp=0xc000651720 sp=0xc0006516b8 pc=0x87bcc5
github.com/Arriven/db1000n/src/utils.Decrypt({0xc00f32c000, 0x54bb5, 0x56000})
	/home/runner/work/db1000n/db1000n/src/utils/crypto.go:81 +0x1e7 fp=0xc0006517c0 sp=0xc000651720 pc=0x87ba07
github.com/Arriven/db1000n/src/job/config.fetchAndDecrypt(0xc000116000?, {0xe273e0, 0x48}, 0xc000651a60?, 0x0)
	/home/runner/work/db1000n/db1000n/src/job/config/config.go:96 +0x42d fp=0xc0006519f8 sp=0xc0006517c0 pc=0x87e02d
github.com/Arriven/db1000n/src/job/config.fetch(0xe273e0?, {0xc0046fa750, 0x3, 0x1?}, 0x0?, 0x1?)
	/home/runner/work/db1000n/db1000n/src/job/config/config.go:68 +0xa5 fp=0xc000651ab0 sp=0xc0006519f8 pc=0x87daa5
github.com/Arriven/db1000n/src/job/config.FetchRawMultiConfig(...)
	/home/runner/work/db1000n/db1000n/src/job/config/config.go:175
github.com/Arriven/db1000n/src/job.(*Runner).Run(0xc00006ebc0, {0xe20a98?, 0xc0001bbb40?}, 0xc0001bbb40?)
	/home/runner/work/db1000n/db1000n/src/job/runner.go:97 +0x28f fp=0xc000651c28 sp=0xc000651ab0 pc=0xada32f
main.main()
	/home/runner/work/db1000n/db1000n/main.go:104 +0xb25 fp=0xc000651f80 sp=0xc000651c28 pc=0xb2e145
runtime.main()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/proc.go:250 +0x212 fp=0xc000651fe0 sp=0xc000651f80 pc=0x43ad92
runtime.goexit()
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc000651fe8 sp=0xc000651fe0 pc=0x46ab21

goroutine 43 [IO wait]:
internal/poll.runtime_pollWait(0x7fbd3e2af080, 0x77)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000efa680?, 0xca63a5?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitWrite(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).WaitWrite(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:741
net.(*netFD).connect(0xc000efa680, {0xe20ad0?, 0xc0000340b0}, {0xca5c03?, 0xe202a0?}, {0xe1af80?, 0xc00040faa0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:141 +0x716
net.(*netFD).dial(0xc000efa680, {0xe20ad0, 0xc0000340b0}, {0xe24488?, 0x0?}, {0xe24488?, 0xc000ef7e00}, 0xc0002e9fe0?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/sock_posix.go:149 +0x394
net.socket({0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, 0x2, 0x1, 0xc010f312d8?, 0xa7?, {0xe24488, 0x0}, ...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/sock_posix.go:70 +0x2b2
net.internetSocket({0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, {0xe24488, 0x0}, {0xe24488, 0xc000ef7e00}, 0xc000e66c80?, 0x0, ...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/ipsock_posix.go:142 +0xf8
net.(*sysDialer).doDialTCP(0xc000efa600, {0xe20ad0, 0xc0000340b0}, 0x0, 0xc?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:65 +0xa5
net.(*sysDialer).dialTCP(0x1?, {0xe20ad0?, 0xc0000340b0?}, 0xf8393a?, 0x6c6?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:61 +0x59
net.(*sysDialer).dialSingle(0xc000efa600, {0xe20ad0, 0xc0000340b0}, {0xe1dbc8?, 0xc000ef7e00})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:583 +0x28b
net.(*sysDialer).dialSerial(0xc000efa600, {0xe20ad0, 0xc0000340b0}, {0xc000870260?, 0x1, 0xca6990?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:551 +0x312
net.(*Dialer).DialContext(0xc000d61860, {0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, {0xc000e66b00, 0xf})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:428 +0x736
net.(*Dialer).Dial(0x10?, {0xca63a5?, 0x7fbd3e291380?}, {0xc000e66b00?, 0x1?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:351 +0x45
github.com/Arriven/db1000n/src/core/http.dialViaProxyFunc.func1({0xc000e66b00?, 0x1?})
	/home/runner/work/db1000n/db1000n/src/core/http/http.go:163 +0x37
github.com/valyala/fasthttp.dialAddr({0xc000e66b00?, 0xc000e66b00?}, 0x13a0f40?, 0x0?, 0x0, 0x0, 0x14f46b0400)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:2015 +0x62
github.com/valyala/fasthttp.(*HostClient).dialHostHard(0xc000ebc820)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1938 +0x172
github.com/valyala/fasthttp.(*HostClient).acquireConn(0xc000ebc820, 0x0, 0x54?)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1605 +0x228
github.com/valyala/fasthttp.(*HostClient).doNonNilReqResp(0xc000ebc820, 0xc000e9ce00, 0xc010f1c300)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1406 +0x29e
github.com/valyala/fasthttp.(*HostClient).do(0x0?, 0x0?, 0x0)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1357 +0x79
github.com/valyala/fasthttp.(*HostClient).Do(0xc000ebc820, 0xc000e9ce00, 0xc0007df9c0?)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1304 +0x91
github.com/Arriven/db1000n/src/job.fastHTTPJob({0xe20b40?, 0xc0009f0510?}, 0x0?, 0xc0000e7080, 0xc0007df980, 0x0?)
	/home/runner/work/db1000n/db1000n/src/job/http.go:167 +0x5ce
github.com/Arriven/db1000n/src/job.(*Runner).runJobs.func1(0x2)
	/home/runner/work/db1000n/db1000n/src/job/runner.go:204 +0x193
created by github.com/Arriven/db1000n/src/job.(*Runner).runJobs
	/home/runner/work/db1000n/db1000n/src/job/runner.go:201 +0x88b

goroutine 7456 [IO wait]:
internal/poll.runtime_pollWait(0x7fbd3d511108, 0x77)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc00ea06580?, 0xca63a5?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitWrite(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).WaitWrite(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:741
net.(*netFD).connect(0xc00ea06580, {0xe20ad0?, 0xc0000340b0}, {0xca5c03?, 0xe202a0?}, {0xe1af80?, 0xc00e9d0920?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:141 +0x716
net.(*netFD).dial(0xc00ea06580, {0xe20ad0, 0xc0000340b0}, {0xe24488?, 0x0?}, {0xe24488?, 0xc00ea049f0}, 0xc00e9e2cd8?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/sock_posix.go:149 +0x394
net.socket({0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, 0x2, 0x1, 0x0?, 0x8?, {0xe24488, 0x0}, ...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/sock_posix.go:70 +0x2b2
net.internetSocket({0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, {0xe24488, 0x0}, {0xe24488, 0xc00ea049f0}, 0xc00e9ecfe0?, 0x0, ...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/ipsock_posix.go:142 +0xf8
net.(*sysDialer).doDialTCP(0xc00ea06500, {0xe20ad0, 0xc0000340b0}, 0x0, 0xd?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:65 +0xa5
net.(*sysDialer).dialTCP(0x1?, {0xe20ad0?, 0xc0000340b0?}, 0x1?, 0x0?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:61 +0x59
net.(*sysDialer).dialSingle(0xc00ea06500, {0xe20ad0, 0xc0000340b0}, {0xe1dbc8?, 0xc00ea049f0})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:583 +0x28b
net.(*sysDialer).dialSerial(0xc00ea06500, {0xe20ad0, 0xc0000340b0}, {0xc00e9fd720?, 0x1, 0xca6990?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:551 +0x312
net.(*Dialer).DialContext(0xc00ea027e0, {0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, {0xc01871c318, 0x11})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:428 +0x736
net.(*Dialer).Dial(0xbc47a0?, {0xca63a5?, 0xc00e9e2cd8?}, {0xc01871c318?, 0xc00ea027e0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:351 +0x45
github.com/Arriven/db1000n/src/core/packetgen.openNetConn({0xe20a98?, 0xc00ea080c0}, {{0xca63a5, 0x3}, {0xc01871c318, 0x11}, 0x0, {{0x0, 0x0}, {0x0, ...}, ...}, ...}, ...)
	/home/runner/work/db1000n/db1000n/src/core/packetgen/connection.go:153 +0x17e
github.com/Arriven/db1000n/src/core/packetgen.OpenConnection({0xe20a98, 0xc00ea080c0}, {{0xca6279?, 0xc00e48d7a0?}, 0xc01312d050?, 0xc014443e00?})
	/home/runner/work/db1000n/db1000n/src/core/packetgen/connection.go:60 +0x15a
github.com/Arriven/db1000n/src/job.sendPacket({0xe20a98?, 0xc014443d40?}, 0xc014443d40?, 0xc000180380, 0x0)
	/home/runner/work/db1000n/db1000n/src/job/packetgen.go:74 +0xaa
github.com/Arriven/db1000n/src/job.packetgenJob({0xe20a98?, 0xc01872d940?}, 0xc0117c8680?, 0xc0000e7080, 0x3?, 0x43b156?)
	/home/runner/work/db1000n/db1000n/src/job/packetgen.go:59 +0x216
github.com/Arriven/db1000n/src/job.rawnetJob({0xe20a98, 0xc01872d940}, {0xca63a5?, 0x3?}, 0x1000000004fd4c7?, 0x0?, 0xc012f665a0?, 0xc0179d14f8?)
	/home/runner/work/db1000n/db1000n/src/job/rawnet.go:54 +0x8a
github.com/Arriven/db1000n/src/job.tcpJob({0xe20a98?, 0xc01872d940?}, 0x405d80?, 0xc0179d14a0?, 0xc0117c8738?, 0x40571d?)
	/home/runner/work/db1000n/db1000n/src/job/rawnet.go:38 +0x3b
github.com/Arriven/db1000n/src/job.parallelJob.func1(0x9673a6?, 0xc017a09440?)
	/home/runner/work/db1000n/db1000n/src/job/complex.go:100 +0x78
created by github.com/Arriven/db1000n/src/job.parallelJob
	/home/runner/work/db1000n/db1000n/src/job/complex.go:99 +0x7c5

goroutine 41 [IO wait]:
internal/poll.runtime_pollWait(0x7fbd3e2af260, 0x77)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000dc9f80?, 0xca63a5?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitWrite(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).WaitWrite(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:741
net.(*netFD).connect(0xc000dc9f80, {0xe20ad0?, 0xc0000340b0}, {0xca5c03?, 0xe202a0?}, {0xe1af80?, 0xc00040f9e0?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:141 +0x716
net.(*netFD).dial(0xc000dc9f80, {0xe20ad0, 0xc0000340b0}, {0xe24488?, 0x0?}, {0xe24488?, 0xc000ef61b0}, 0xc0002e9e30?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/sock_posix.go:149 +0x394
net.socket({0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, 0x2, 0x1, 0xc000ed72d8?, 0xa7?, {0xe24488, 0x0}, ...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/sock_posix.go:70 +0x2b2
net.internetSocket({0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, {0xe24488, 0x0}, {0xe24488, 0xc000ef61b0}, 0xc000e667c0?, 0x0, ...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/ipsock_posix.go:142 +0xf8
net.(*sysDialer).doDialTCP(0xc000dc9f00, {0xe20ad0, 0xc0000340b0}, 0x0, 0xc?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:65 +0xa5
net.(*sysDialer).dialTCP(0x1?, {0xe20ad0?, 0xc0000340b0?}, 0xf8393a?, 0x80?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:61 +0x59
net.(*sysDialer).dialSingle(0xc000dc9f00, {0xe20ad0, 0xc0000340b0}, {0xe1dbc8?, 0xc000ef61b0})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:583 +0x28b
net.(*sysDialer).dialSerial(0xc000dc9f00, {0xe20ad0, 0xc0000340b0}, {0xc00086aec0?, 0x1, 0xca6990?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:551 +0x312
net.(*Dialer).DialContext(0xc000d61620, {0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, {0xc000e66640, 0xf})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:428 +0x736
net.(*Dialer).Dial(0x10?, {0xca63a5?, 0x7fbd3e291380?}, {0xc000e66640?, 0x1?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:351 +0x45
github.com/Arriven/db1000n/src/core/http.dialViaProxyFunc.func1({0xc000e66640?, 0x1?})
	/home/runner/work/db1000n/db1000n/src/core/http/http.go:163 +0x37
github.com/valyala/fasthttp.dialAddr({0xc000e66640?, 0xc000e66640?}, 0x13a0f40?, 0x0?, 0x0, 0x0, 0x14f46b0400)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:2015 +0x62
github.com/valyala/fasthttp.(*HostClient).dialHostHard(0xc000ebc4e0)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1938 +0x172
github.com/valyala/fasthttp.(*HostClient).acquireConn(0xc000ebc4e0, 0x0, 0x54?)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1605 +0x228
github.com/valyala/fasthttp.(*HostClient).doNonNilReqResp(0xc000ebc4e0, 0xc000e9c700, 0xc000dc7b00)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1406 +0x29e
github.com/valyala/fasthttp.(*HostClient).do(0x0?, 0x0?, 0x0)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1357 +0x79
github.com/valyala/fasthttp.(*HostClient).Do(0xc000ebc4e0, 0xc000e9c700, 0xc0007df300?)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1304 +0x91
github.com/Arriven/db1000n/src/job.fastHTTPJob({0xe20b40?, 0xc0009f0450?}, 0xc000052e30?, 0xc0000e7080, 0xc0007df2c0, 0x0?)
	/home/runner/work/db1000n/db1000n/src/job/http.go:167 +0x5ce
github.com/Arriven/db1000n/src/job.(*Runner).runJobs.func1(0x0)
	/home/runner/work/db1000n/db1000n/src/job/runner.go:204 +0x193
created by github.com/Arriven/db1000n/src/job.(*Runner).runJobs
	/home/runner/work/db1000n/db1000n/src/job/runner.go:201 +0x88b

goroutine 22 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fbd3ec7a778, 0x72)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000424c80?, 0x0?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Accept(0xc000424c80)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:614 +0x22c
net.(*netFD).accept(0xc000424c80)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:172 +0x35
net.(*TCPListener).accept(0xc0002fac30)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:139 +0x28
net.(*TCPListener).Accept(0xc0002fac30)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock.go:288 +0x3d
net/http.(*Server).Serve(0xc000364e00, {0xe20210, 0xc0002fac30})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:3039 +0x385
net/http.(*Server).ListenAndServe(0xc000364e00)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/http/server.go:2968 +0x7d
github.com/Arriven/db1000n/src/utils/metrics.serveMetrics({0xe20a98?, 0xc0001bbb40}, 0xc000070fc0, {0xca72af, 0x5})
	/home/runner/work/db1000n/db1000n/src/utils/metrics/serve.go:41 +0x225
github.com/Arriven/db1000n/src/utils/metrics.ExportPrometheusMetrics({0xe20a98, 0xc0001bbb40}, 0x0?, {0x0?, 0x0?}, {0xca72af, 0x5})
	/home/runner/work/db1000n/db1000n/src/utils/metrics/prometheus.go:164 +0x50
created by github.com/Arriven/db1000n/src/utils/metrics.InitOrFail
	/home/runner/work/db1000n/db1000n/src/utils/metrics/prometheus.go:102 +0x12a

goroutine 42 [IO wait]:
internal/poll.runtime_pollWait(0x7fbd3e2af170, 0x77)
	/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000efa300?, 0xca63a5?, 0x0)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitWrite(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).WaitWrite(...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/internal/poll/fd_unix.go:741
net.(*netFD).connect(0xc000efa300, {0xe20ad0?, 0xc0000340b0}, {0xca5c03?, 0xe202a0?}, {0xe1af80?, 0xc00040fa40?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/fd_unix.go:141 +0x716
net.(*netFD).dial(0xc000efa300, {0xe20ad0, 0xc0000340b0}, {0xe24488?, 0x0?}, {0xe24488?, 0xc000ef7020}, 0xc0002e9f08?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/sock_posix.go:149 +0x394
net.socket({0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, 0x2, 0x1, 0xc000ed32d8?, 0xa7?, {0xe24488, 0x0}, ...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/sock_posix.go:70 +0x2b2
net.internetSocket({0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, {0xe24488, 0x0}, {0xe24488, 0xc000ef7020}, 0xc000e66a20?, 0x0, ...)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/ipsock_posix.go:142 +0xf8
net.(*sysDialer).doDialTCP(0xc000efa280, {0xe20ad0, 0xc0000340b0}, 0x0, 0xc?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:65 +0xa5
net.(*sysDialer).dialTCP(0x1?, {0xe20ad0?, 0xc0000340b0?}, 0xf8393a?, 0x6c6?)
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/tcpsock_posix.go:61 +0x59
net.(*sysDialer).dialSingle(0xc000efa280, {0xe20ad0, 0xc0000340b0}, {0xe1dbc8?, 0xc000ef7020})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:583 +0x28b
net.(*sysDialer).dialSerial(0xc000efa280, {0xe20ad0, 0xc0000340b0}, {0xc00086b8a0?, 0x1, 0xca6990?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:551 +0x312
net.(*Dialer).DialContext(0xc000d61740, {0xe20ad0, 0xc0000340b0}, {0xca63a5, 0x3}, {0xc000e668a0, 0x10})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:428 +0x736
net.(*Dialer).Dial(0x413585?, {0xca63a5?, 0xc28160?}, {0xc000e668a0?, 0x5d00000000000018?})
	/opt/hostedtoolcache/go/1.18.3/x64/src/net/dial.go:351 +0x45
github.com/Arriven/db1000n/src/core/http.dialViaProxyFunc.func1({0xc000e668a0?, 0xbc37e0?})
	/home/runner/work/db1000n/db1000n/src/core/http/http.go:163 +0x37
github.com/valyala/fasthttp.dialAddr({0xc000e668a0?, 0xc000e668a0?}, 0x10?, 0x0?, 0x1, 0xc010f1c000, 0x14f46b0400)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:2015 +0x62
github.com/valyala/fasthttp.(*HostClient).dialHostHard(0xc000ebc680)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1938 +0x172
github.com/valyala/fasthttp.(*HostClient).acquireConn(0xc000ebc680, 0x0, 0x54?)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1605 +0x228
github.com/valyala/fasthttp.(*HostClient).doNonNilReqResp(0xc000ebc680, 0xc000e9ca80, 0xc000dc7e00)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1406 +0x29e
github.com/valyala/fasthttp.(*HostClient).do(0x0?, 0x0?, 0x0)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1357 +0x79
github.com/valyala/fasthttp.(*HostClient).Do(0xc000ebc680, 0xc000e9ca80, 0xc0007df680?)
	/home/runner/go/pkg/mod/github.com/valyala/[email protected]/client.go:1304 +0x91
github.com/Arriven/db1000n/src/job.fastHTTPJob({0xe20b40?, 0xc0009f04b0?}, 0x0?, 0xc0000e7080, 0xc0007df640, 0x0?)
	/home/runner/work/db1000n/db1000n/src/job/http.go:167 +0x5ce
github.com/Arriven/db1000n/src/job.(*Runner).runJobs.func1(0x1)
	/home/runner/work/db1000n/db1000n/src/job/runner.go:204 +0x193
created by github.com/Arriven/db1000n/src/job.(*Runner).runJobs
	/home/runner/work/db1000n/db1000n/src/job/runner.go:201 +0x88b

@arriven
Copy link
Owner

arriven commented Jun 30, 2022

@MidnightRAT @deputinizer should be fixed now, poke me if you still encounter it

@MidnightRAT
Copy link
Author 

[ 3449.257924] systemd-journal invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=-250
[ 3449.257933] CPU: 0 PID: 405 Comm: systemd-journal Not tainted 5.13.0-1036-oracle #43~20.04.1-Ubuntu
[ 3449.257936] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.5.1 06/16/2021
[ 3449.257938] Call Trace:
[ 3449.257940]  <TASK>
[ 3449.257943]  dump_stack+0x7d/0x9c
[ 3449.257949]  dump_header+0x4f/0x1f6
[ 3449.257953]  oom_kill_process.cold+0xb/0x10
[ 3449.257957]  out_of_memory+0x1cf/0x520
[ 3449.257961]  __alloc_pages_slowpath.constprop.0+0xc86/0xd50
[ 3449.257967]  __alloc_pages+0x2ce/0x320
[ 3449.257970]  alloc_pages+0x87/0xe0
[ 3449.257973]  __page_cache_alloc+0x89/0xb0
[ 3449.257977]  pagecache_get_page+0x218/0x4d0
[ 3449.257980]  filemap_fault+0x5bf/0x9b0
[ 3449.257981]  ? do_set_pte+0xc8/0x140
[ 3449.257984]  ? next_uptodate_page+0x160/0x270
[ 3449.257987]  ext4_filemap_fault+0x32/0x50
[ 3449.257991]  __do_fault+0x3e/0xc0
[ 3449.257994]  do_fault+0x1e6/0x400
[ 3449.257996]  __handle_mm_fault+0x619/0x8e0
[ 3449.257999]  handle_mm_fault+0xda/0x2b0
[ 3449.258001]  do_user_addr_fault+0x1bb/0x650
[ 3449.258005]  exc_page_fault+0x7d/0x170
[ 3449.258008]  ? asm_exc_page_fault+0x8/0x30
[ 3449.258012]  asm_exc_page_fault+0x1e/0x30
[ 3449.258015] RIP: 0033:0x7f1e00753f7b
[ 3449.258022] Code: Unable to access opcode bytes at RIP 0x7f1e00753f51.
[ 3449.258024] RSP: 002b:00007ffc2419f118 EFLAGS: 00010246
[ 3449.258027] RAX: 0000000000000000 RBX: 0000000000000578 RCX: 0000000000000006
[ 3449.258029] RDX: 0000000000000006 RSI: 00007f1e008b11bc RDI: 00007f1dff64f5b8
[ 3449.258030] RBP: 000055eb7e86d260 R08: 00000000000000e8 R09: 000055eb7e83df40
[ 3449.258032] R10: 00007f1dff6442e0 R11: 00007f1dfc104db0 R12: 000000000039d5b8
[ 3449.258033] R13: 00007ffc2419f148 R14: 00000000020dad70 R15: 00007f1dff64f5b8
[ 3449.258036]  </TASK>
[ 3449.258036] Mem-Info:
[ 3449.258038] active_anon:251 inactive_anon:173988 isolated_anon:0
                active_file:0 inactive_file:1883 isolated_file:32
                unevictable:5292 dirty:0 writeback:0
                slab_reclaimable:6046 slab_unreclaimable:16938
                mapped:2379 shmem:271 pagetables:1079 bounce:0
                free:14684 free_pcp:63 free_cma:0
[ 3449.258043] Node 0 active_anon:1004kB inactive_anon:695952kB active_file:0kB inactive_file:7532kB unevictable:21168kB isolated(anon):0kB isolated(file):0kB mapped:9516kB dirty:0kB writeback:0kB shmem:1084kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 278528kB writeback_tmp:0kB kernel_stack:3104kB pagetables:4316kB all_unreclaimable? yes
[ 3449.258048] Node 0 DMA free:4308kB min:680kB low:848kB high:1016kB reserved_highatomic:0KB active_anon:0kB inactive_anon:9552kB active_file:0kB inactive_file:180kB unevictable:0kB writepending:0kB present:15004kB managed:14372kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 3449.258053] lowmem_reserve[]: 0 913 913 913 913
[ 3449.258057] Node 0 DMA32 free:54428kB min:44372kB low:55464kB high:66556kB reserved_highatomic:10240KB active_anon:1004kB inactive_anon:686400kB active_file:164kB inactive_file:7036kB unevictable:21168kB writepending:0kB present:1027952kB managed:974964kB mlocked:21168kB bounce:0kB free_pcp:252kB local_pcp:248kB free_cma:0kB
[ 3449.258063] lowmem_reserve[]: 0 0 0 0 0
[ 3449.258066] Node 0 DMA: 4*4kB (U) 4*8kB (U) 5*16kB (UE) 3*32kB (UE) 6*64kB (UME) 3*128kB (UME) 3*256kB (UME) 1*512kB (E) 2*1024kB (UE) 0*2048kB 0*4096kB = 4320kB
[ 3449.258081] Node 0 DMA32: 2440*4kB (UME) 1846*8kB (UME) 590*16kB (UME) 173*32kB (ME) 85*64kB (ME) 38*128kB (ME) 20*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54928kB
[ 3449.258095] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 3449.258097] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 3449.258098] 4551 total pagecache pages
[ 3449.258099] 0 pages in swap cache
[ 3449.258100] Swap cache stats: add 0, delete 0, find 0/0
[ 3449.258101] Free swap  = 0kB
[ 3449.258102] Total swap = 0kB
[ 3449.258102] 260739 pages RAM
[ 3449.258103] 0 pages HighMem/MovableOnly
[ 3449.258104] 13405 pages reserved
[ 3449.258104] 0 pages hwpoisoned
[ 3449.258105] Tasks state (memory values in pages):
[ 3449.258106] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 3449.258108] [    405]     0   405    28618     1092   225280        0          -250 systemd-journal
[ 3449.258112] [    428]     0   428     4943      783    61440        0         -1000 systemd-udevd
[ 3449.258116] [    566]     0   566    70051     4500    86016        0         -1000 multipathd
[ 3449.258119] [    657]     0   657      624      128    45056        0             0 bpfilter_umh
[ 3449.258122] [    659]   112   659     1777      720    53248        0             0 rpcbind
[ 3449.258125] [    660]   102   660    22493      687    77824        0             0 systemd-timesyn
[ 3449.258128] [    714]   100   714     6645      677    73728        0             0 systemd-network
[ 3449.258152] [    716]   101   716     5965     1681    90112        0             0 systemd-resolve
[ 3449.258154] [    762]     0   762    60325      970   102400        0             0 accounts-daemon
[ 3449.258157] [    763]   103   763     1951      805    53248        0          -900 dbus-daemon
[ 3449.258160] [    769]     0   769    20458      605    65536        0             0 irqbalance
[ 3449.258162] [    771]     0   771     7404     2848    90112        0             0 networkd-dispat
[ 3449.258164] [    773]     0   773    59107      635    90112        0             0 polkitd
[ 3449.258166] [    774]   104   774    56088      364    77824        0             0 rsyslogd
[ 3449.258169] [    778] 584788   778   178846     1559   114688        0             0 updater
[ 3449.258171] [    780] 584788   780   178881     1591   118784        0             0 agent
[ 3449.258174] [    783]     0   783     4155      887    69632        0             0 systemd-logind
[ 3449.258176] [    784]     0   784    98718     1360   139264        0             0 udisksd
[ 3449.258178] [    799]     0   799     1279       54    45056        0             0 iscsid
[ 3449.258181] [    800]     0   800     1405     1365    49152        0           -17 iscsid
[ 3449.258183] [    816]     0   816     2136      579    49152        0             0 cron
[ 3449.258185] [    822]     0   822     3045      709    69632        0         -1000 sshd
[ 3449.258188] [    824]     0   824      950      519    40960        0             0 atd
[ 3449.258190] [    842]     0   842     1840      424    45056        0             0 agetty
[ 3449.258192] [    848]     0   848     1459      386    53248        0             0 agetty
[ 3449.258194] [    865]     0   865    27028     2777   110592        0             0 unattended-upgr
[ 3449.258197] [    868]     0   868    78615     1103   110592        0             0 ModemManager
[ 3449.258199] [   1068] 584788  1068   179221     2099   118784        0             0 gomon
[ 3449.258201] [   1079]     0  1079     3451     1019    65536        0             0 sshd
[ 3449.258204] [   1082]  1001  1082     4592     1190    77824        0             0 systemd
[ 3449.258206] [   1083]  1001  1083    25832      875    98304        0             0 (sd-pam)
[ 3449.258208] [   1179]  1001  1179     3485      368    65536        0             0 sshd
[ 3449.258211] [   1180]  1001  1180     2515      899    53248        0             0 bash
[ 3449.258213] [   1192]  1001  1192     2515      575    49152        0             0 bash
[ 3449.258215] [   1199]  1001  1199   387627   152709  1671168        0             0 db1000n
[ 3449.258217] [   1240]  1001  1240   177110     1254   143360        0             0 launcher-disbal
[ 3449.258220] [   1295]  1001  1295   177640     1683   114688        0             0 app
[ 3449.258222] [   1534]     0  1534     8513     1082    61440        0          -900 snapd
[ 3449.258225] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1001.slice/session-1.scope,task=db1000n,pid=1199,uid=1001
[ 3449.258263] Out of memory: Killed process 1199 (db1000n) total-vm:1550508kB, anon-rss:610836kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:1632kB oom_score_adj:0
[ 3449.359951] oom_reaper: reaped process 1199 (db1000n), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 3455.762261] loop10: detected capacity change from 0 to 8
root@instance-20220226-1719:~# free
              total        used        free      shared  buff/cache   available
Mem:         989336      190572      491268        1092      307496      655840
Swap:             0           0           0

@heximcz
Copy link

heximcz commented Jul 3, 2022 

[ 1937.576964] db1000n invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[ 1937.576996] CPU: 2 PID: 850 Comm: db1000n Tainted: G         C        5.10.103-v7l+ #1529
[ 1937.577000] Hardware name: BCM2711
[ 1937.577006] Backtrace: 
[ 1937.577041] [<c0b875f0>] (dump_backtrace) from [<c0b87984>] (show_stack+0x20/0x24)
[ 1937.577050]  r7:ffffffff r6:00000000 r5:60000013 r4:c12e6fd8
[ 1937.577059] [<c0b87964>] (show_stack) from [<c0b8bd74>] (dump_stack+0xcc/0xf8)
[ 1937.577067] [<c0b8bca8>] (dump_stack) from [<c0b8a004>] (dump_header+0x64/0x208)
[ 1937.577073]  r10:c12051c8 r9:00000cc0 r8:00000000 r7:c0e3c824 r6:c40b6d00 r5:c1ac1f00
[ 1937.577078]  r4:cb9e9cd0 r3:6e4df213
[ 1937.577097] [<c0b89fa0>] (dump_header) from [<c03ba5f8>] (oom_kill_process+0x1b4/0x1c0)
[ 1937.577102]  r7:c0e3c824 r6:cb9e9cd0 r5:c1ac2480 r4:c1ac1f00
[ 1937.577112] [<c03ba444>] (oom_kill_process) from [<c03bb170>] (out_of_memory+0x2b8/0x390)
[ 1937.577117]  r7:c1208600 r6:c1205048 r5:c1ac1f00 r4:cb9e9cd0
[ 1937.577127] [<c03baeb8>] (out_of_memory) from [<c0410030>] (__alloc_pages_nodemask+0x7ec/0x1184)
[ 1937.577133]  r7:c1338550 r6:00001000 r5:0001494e r4:00000000
[ 1937.577147] [<c040f844>] (__alloc_pages_nodemask) from [<c09fc540>] (skb_page_frag_refill+0xe4/0x10c)
[ 1937.577154]  r10:7aa42cec r9:0000006c r8:d2925b00 r7:cb9e9e60 r6:00000040 r5:00000cc0
[ 1937.577158]  r4:c3e863d0
[ 1937.577165] [<c09fc45c>] (skb_page_frag_refill) from [<c09fcdb4>] (sk_page_frag_refill+0x24/0x78)
[ 1937.577169]  r5:d2925c04 r4:d2925b00
[ 1937.577181] [<c09fcd90>] (sk_page_frag_refill) from [<c0ab2ae4>] (tcp_sendmsg_locked+0x4c0/0xd98)
[ 1937.577186]  r5:d2925c04 r4:c869e000
[ 1937.577195] [<c0ab2624>] (tcp_sendmsg_locked) from [<c0ab33f4>] (tcp_sendmsg+0x38/0x4c)
[ 1937.577201]  r10:c4484500 r9:c4342540 r8:cb9e9ef8 r7:00000000 r6:000000a9 r5:cb9e9e60
[ 1937.577205]  r4:d2925b00
[ 1937.577214] [<c0ab33bc>] (tcp_sendmsg) from [<c0aeba24>] (inet_sendmsg+0x48/0x54)
[ 1937.577219]  r7:00000000 r6:000000a9 r5:cb9e9e60 r4:d2925b00
[ 1937.577228] [<c0aeb9dc>] (inet_sendmsg) from [<c09f8390>] (sock_sendmsg+0x44/0x54)
[ 1937.577234]  r7:00000000 r6:00000000 r5:c4484500 r4:cb9e9e60
[ 1937.577241] [<c09f834c>] (sock_sendmsg) from [<c09f8444>] (sock_write_iter+0xa4/0xf8)
[ 1937.577245]  r5:c1205048 r4:cb9e9ee0
[ 1937.577255] [<c09f83a0>] (sock_write_iter) from [<c0440184>] (vfs_write+0x3ec/0x464)
[ 1937.577262]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:c1205048 r5:000000a9
[ 1937.577266]  r4:c4342540
[ 1937.577272] [<c043fd98>] (vfs_write) from [<c04403c4>] (ksys_write+0xcc/0xe0)
[ 1937.577278]  r10:000000a9 r9:04f1c4d0 r8:c4342540 r7:00000004 r6:03aecf48 r5:c4342541
[ 1937.577282]  r4:c1205048
[ 1937.577289] [<c04402f8>] (ksys_write) from [<c04403f0>] (sys_write+0x18/0x1c)
[ 1937.577294]  r10:00000004 r9:cb9e8000 r8:c0200204 r7:00000004 r6:03aecf48 r5:00000000
[ 1937.577298]  r4:00000000
[ 1937.577308] [<c04403d8>] (sys_write) from [<c02001e4>] (__sys_trace_return+0x0/0x1c)
[ 1937.577313] Exception stack(0xcb9e9fa8 to 0xcb9e9ff0)
[ 1937.577319] 9fa0:                   00000000 00000000 000001af 04f1c4d0 000000a9 00000000
[ 1937.577326] 9fc0: 00000000 00000000 03aecf48 00000004 00000005 00000020 03aecf00 00ecaf68
[ 1937.577332] 9fe0: 04f1c4d3 0adc6be4 000129c0 0009d3ac
[ 1937.577339] Mem-Info:
[ 1937.577355] active_anon:96 inactive_anon:80587 isolated_anon:0
                active_file:11029 inactive_file:30902 isolated_file:0
                unevictable:4 dirty:6 writeback:0
                slab_reclaimable:3288 slab_unreclaimable:10544
                mapped:30547 shmem:2169 pagetables:735 bounce:0
                free:775436 free_pcp:15 free_cma:80385
[ 1937.577370] Node 0 active_anon:384kB inactive_anon:322348kB active_file:44116kB inactive_file:123608kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:122188kB dirty:24kB writeback:0kB shmem:8676kB writeback_tmp:0kB kernel_stack:1464kB all_unreclaimable? no
[ 1937.577382] DMA free:336620kB min:16384kB low:20480kB high:24576kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:188kB inactive_file:224kB unevictable:0kB writepending:0kB present:786432kB managed:666356kB mlocked:0kB pagetables:248kB bounce:0kB free_pcp:60kB local_pcp:44kB free_cma:321540kB
[ 1937.577390] lowmem_reserve[]: 0 0 3188 3188
[ 1937.577428] HighMem free:2765124kB min:512kB low:20576kB high:40640kB reserved_highatomic:0KB active_anon:384kB inactive_anon:322348kB active_file:43928kB inactive_file:123572kB unevictable:16kB writepending:24kB present:3264512kB managed:3264512kB mlocked:16kB pagetables:2692kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[ 1937.577434] lowmem_reserve[]: 0 0 0 0
[ 1937.577467] DMA: 441*4kB (UMEC) 319*8kB (UMEC) 222*16kB (UMEC) 25*32kB (UMEC) 12*64kB (MEC) 4*128kB (UMEC) 4*256kB (MEC) 5*512kB (MC) 4*1024kB (UMC) 0*2048kB 78*4096kB (C) = 337116kB
[ 1937.577596] HighMem: 1339*4kB (UM) 743*8kB (UM) 368*16kB (UM) 183*32kB (UM) 115*64kB (UM) 39*128kB (UM) 21*256kB (UM) 119*512kB (UM) 71*1024kB (UM) 13*2048kB (UM) 626*4096kB (M) = 2765124kB
[ 1937.577729] 44110 total pagecache pages
[ 1937.577741] 0 pages in swap cache
[ 1937.577749] Swap cache stats: add 0, delete 0, find 0/0
[ 1937.577755] Free swap  = 102396kB
[ 1937.577761] Total swap = 102396kB
[ 1937.577767] 1012736 pages RAM
[ 1937.577773] 816128 pages HighMem/MovableOnly
[ 1937.577779] 30019 pages reserved
[ 1937.577785] 81920 pages cma reserved
[ 1937.577792] Tasks state (memory values in pages):
[ 1937.577799] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[ 1937.577830] [    137]     0   137     5302     1728   135168        0             0 systemd-journal
[ 1937.577842] [    165]     0   165     4499      957    40960        0         -1000 systemd-udevd
[ 1937.577863] [    337]   100   337     5596      707    57344        0             0 systemd-timesyn
[ 1937.577874] [    399]     0   399     1987      557    40960        0             0 cron
[ 1937.577882] [    403]   108   403     1442      644    32768        0             0 avahi-daemon
[ 1937.577891] [    406]     0   406     3253     1439    49152        0             0 systemd-logind
[ 1937.577900] [    409]     0   409      923      188    36864        0             0 alsactl
[ 1937.577911] [    416]   104   416     1639      751    40960        0          -900 dbus-daemon
[ 1937.577920] [    421]   108   421     1442       63    32768        0             0 avahi-daemon
[ 1937.577928] [    424]     0   424     2678      945    45056        0             0 wpa_supplicant
[ 1937.577936] [    425] 65534   425     1080      537    36864        0             0 thd
[ 1937.577946] [    427]     0   427     6914       20    40960        0             0 rngd
[ 1937.577955] [    428]     0   428     6376      700    49152        0             0 rsyslogd
[ 1937.577965] [    563]     0   563   228984     9177   180224        0          -999 containerd
[ 1937.577974] [    564]     0   564     1077      324    32768        0             0 agetty
[ 1937.577985] [    573]     0   573     2671     1347    45056        0         -1000 sshd
[ 1937.577995] [    582]     0   582   233188    13976   221184        0          -500 dockerd
[ 1937.578003] [    621]     0   621      535       32    28672        0             0 hciattach
[ 1937.578012] [    635]     0   635     2381      851    40960        0             0 bluetoothd
[ 1937.578023] [    707]     0   707     3134     1603    45056        0             0 sshd
[ 1937.578034] [    710]     0   710     3642     1750    49152        0             0 systemd
[ 1937.578043] [    711]     0   711     8807      881    65536        0             0 (sd-pam)
[ 1937.578052] [    727]     0   727     2090      912    40960        0             0 bash
[ 1937.578063] [    740]     0   740     2030      665    36864        0             0 tmux: client
[ 1937.578072] [    742]     0   742     2184      760    36864        0             0 tmux: server
[ 1937.578080] [    743]     0   743     2093      865    40960        0             0 bash
[ 1937.578087] [    756]     0   756   229885     8985   172032        0             0 docker
[ 1937.578097] [    787]     0   787   200854     2137    57344        0          -998 containerd-shim
[ 1937.578105] [    807]     0   807   270611    65936  1204224        0             0 db1000n
[ 1937.578115] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=f95378fa8d03d2d2754e3f85c98f8d2beb7f61d5af410b7f358ed7a08904fc66,mems_allowed=0,global_oom,task_memcg=/,task=db1000n,pid=807,uid=0
[ 1937.578194] Out of memory: Killed process 807 (db1000n) total-vm:1082444kB, anon-rss:251128kB, file-rss:12616kB, shmem-rss:0kB, UID:0 pgtables:1176kB oom_score_adj:0
[ 1938.117713] oom_reaper: reaped process 807 (db1000n), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB


Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version                        Architecture Description
+++-=========================-==============================-============-========================================================
un  docker                    <none>                         <none>       (no description available)
ii  docker-ce                 5:20.10.17~3-0~raspbian-buster armhf        Docker: the open-source application container engine
ii  docker-ce-cli             5:20.10.17~3-0~raspbian-buster armhf        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras 5:20.10.17~3-0~raspbian-buster armhf        Rootless support for Docker.
ii  docker-compose-plugin     2.6.0~raspbian-buster          armhf        Docker Compose (V2) plugin for the Docker CLI.

| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-======================================
un  containerd     <none>       <none>       (no description available)
ii  containerd.io  1.6.6-1      armhf        An open and reliable container runtime

              total        used        free      shared  buff/cache   available
Mem:          3.7Gi        85Mi       3.5Gi       8.0Mi       196Mi       3.5Gi
Swap:          99Mi          0B        99Mi

@AlexUkr73
Copy link

AlexUkr73 commented Jul 16, 2022 •
edited
Loading

the same issue on Ubuntu VM with 2 GB RAM with latest and older db1000n builds:

Jul 16 07:13:11 azurea3 kernel: [64679.724233] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,gl obal_oom,task_memcg=/user.slice,task=db1000n,pid=10373,uid=1000 Jul 16 07:13:11 azurea3 kernel: [64679.724243] Out of memory: Killed process 10373 (db1000n) total-vm:1549092kB, anon-rss:374 792kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:1060kB oom_score_adj:0

Could you advice, please? - what syntax and value should be passed to option
-min-interval

regardless of this option's value /absent value - only help displays

Please advice or pont me any how-to with any other ways to reduce memory usage by utility in Ubuntu (e.g. with options examples, etc)

UPD:
this does not help
prlimit --pid <PID_ID> --as=<MEMORY_VALUE>

e.g.
prlimit --pid 10632 --as=1024000000

++++++++++++++++
UPD & UPD:
script below seems solved the problem by monitoring and re-running 'db1000n' process

#! /bin/bash
until $HOME/./db1000n; do
echo "'db1000n' is crashed with exit code $?. Respawning.." >&2
sleep 1
done

@arriven
Copy link
Owner

arriven commented Jul 16, 2022

I've spent couple of days trying to repro this on my machine but I never got it to consume more than 500 mb of RAM. Did someone monitor the memory usage of db1000n before the oom kill? I'm wondering whether it's a leak that accumulates somewhere or some kind of very weird spike in memory usage, knowing that would help to diagnose the problem

@AlexUkr73 as for -min-interval option something like -min-interval 5s should work

@AlexUkr73
Copy link

AlexUkr73 commented Jul 17, 2022 •
edited
Loading

@arriven
Regarding reproducing - I've noticed it is reproduced on VMs with only 1 CPU available and 1 GB RAM (Amazon AWS)

-min-interval 5s- tried everything but didn't expect adding letter after digit ;o)

It will be the good idea to add this statement/example to application's help

@MidnightRAT
Copy link
Author 

[11514.036137] db1000n invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[11514.036146] CPU: 1 PID: 2173 Comm: db1000n Not tainted 5.15.0-1013-oracle #17~20.04.1-Ubuntu
[11514.036150] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.5.1 06/16/2021
[11514.036151] Call Trace:
[11514.036154]  <TASK>
[11514.036156]  dump_stack_lvl+0x4a/0x5f
[11514.036163]  dump_stack+0x10/0x12
[11514.036165]  dump_header+0x53/0x221
[11514.036169]  oom_kill_process.cold+0xb/0x10
[11514.036172]  out_of_memory+0x1dc/0x530
[11514.036176]  __alloc_pages_slowpath.constprop.0+0xcd0/0xde0
[11514.036180]  ? __alloc_pages_slowpath.constprop.0+0xd84/0xde0
[11514.036184]  __alloc_pages+0x2a3/0x2f0
[11514.036187]  alloc_pages+0x90/0x110
[11514.036189]  __page_cache_alloc+0x83/0xb0
[11514.036192]  pagecache_get_page+0x150/0x510
[11514.036195]  ? page_cache_ra_unbounded+0x16a/0x210
[11514.036198]  filemap_fault+0x523/0xb60
[11514.036201]  ? filemap_map_pages+0x134/0x640
[11514.036205]  __do_fault+0x40/0x110
[11514.036208]  do_fault+0x1f5/0x410
[11514.036211]  __handle_mm_fault+0x635/0x850
[11514.036214]  handle_mm_fault+0xd8/0x2b0
[11514.036218]  do_user_addr_fault+0x1b9/0x650
[11514.036221]  exc_page_fault+0x77/0x160
[11514.036225]  ? asm_exc_page_fault+0x8/0x30
[11514.036228]  asm_exc_page_fault+0x1e/0x30
[11514.036231] RIP: 0033:0x455cf7
[11514.036238] Code: Unable to access opcode bytes at RIP 0x455ccd.
[11514.036239] RSP: 002b:000000c00000b8e8 EFLAGS: 00010216
[11514.036242] RAX: 000000000006a7fc RBX: fffffffffffffffa RCX: 00000000012e8380
[11514.036245] RDX: 00000000000004ea RSI: 00000000000046f4 RDI: 0000000001134a60
[11514.036246] RBP: 000000c00000b8f8 R08: 00000000000004eb R09: 00000000011371b8
[11514.036248] R10: 00007fffa8f3e090 R11: 00000000001f77da R12: 000000c00000b9a0
[11514.036249] R13: ffffffffffffffff R14: 000000c000002000 R15: 0000000000000000
[11514.036252]  </TASK>
[11514.036253] Mem-Info:
[11514.036254] active_anon:293 inactive_anon:165819 isolated_anon:0
                active_file:9 inactive_file:447 isolated_file:32
                unevictable:1396 dirty:0 writeback:0
                slab_reclaimable:5947 slab_unreclaimable:17616
                mapped:1087 shmem:259 pagetables:877 bounce:0
                kernel_misc_reclaimable:0
                free:12584 free_pcp:314 free_cma:0
[11514.036260] Node 0 active_anon:1172kB inactive_anon:663276kB active_file:36kB inactive_file:1788kB unevictable:5584kB isolated(anon):0kB isolated(file):128kB mapped:4348kB dirty:0kB writeback:0kB shmem:1036kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 270336kB writeback_tmp:0kB kernel_stack:2208kB pagetables:3508kB all_unreclaimable? yes
[11514.036265] Node 0 DMA free:4212kB min:696kB low:868kB high:1040kB reserved_highatomic:0KB active_anon:0kB inactive_anon:9712kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15004kB managed:14368kB mlocked:0kB bounce:0kB free_pcp:8kB local_pcp:8kB free_cma:0kB
[11514.036271] lowmem_reserve[]: 0 894 894 894 894
[11514.036275] Node 0 DMA32 free:46124kB min:44356kB low:55444kB high:66532kB reserved_highatomic:2048KB active_anon:1172kB inactive_anon:653564kB active_file:16kB inactive_file:1648kB unevictable:5584kB writepending:0kB present:1027952kB managed:974684kB mlocked:5584kB bounce:0kB free_pcp:1248kB local_pcp:764kB free_cma:0kB
[11514.036281] lowmem_reserve[]: 0 0 0 0 0
[11514.036284] Node 0 DMA: 3*4kB (E) 9*8kB (UME) 4*16kB (UME) 6*32kB (UME) 4*64kB (UME) 5*128kB (UME) 4*256kB (UME) 2*512kB (U) 1*1024kB (E) 0*2048kB 0*4096kB = 4308kB
[11514.036299] Node 0 DMA32: 68*4kB (ME) 93*8kB (UME) 69*16kB (UE) 49*32kB (UME) 22*64kB (UME) 12*128kB (UME) 5*256kB (UM) 29*512kB (M) 23*1024kB (UM) 0*2048kB 0*4096kB = 46312kB
[11514.036314] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[11514.036316] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[11514.036318] 1835 total pagecache pages
[11514.036319] 0 pages in swap cache
[11514.036319] Swap cache stats: add 0, delete 0, find 0/0
[11514.036321] Free swap  = 0kB
[11514.036321] Total swap = 0kB
[11514.036322] 260739 pages RAM
[11514.036323] 0 pages HighMem/MovableOnly
[11514.036323] 13476 pages reserved
[11514.036324] 0 pages hwpoisoned
[11514.036325] Tasks state (memory values in pages):
[11514.036325] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[11514.036328] [    349]     0   349    28900      805   225280        0          -250 systemd-journal
[11514.036333] [    376]     0   376     5454      408    65536        0         -1000 systemd-udevd
[11514.036337] [    558]   107   558     1777      459    53248        0             0 rpcbind
[11514.036340] [    559]     0   559      624      128    45056        0             0 bpfilter_umh
[11514.036342] [    560]   101   560    22493      568    69632        0             0 systemd-timesyn
[11514.036345] [    600]   102   600     6645      232    73728        0             0 systemd-network
[11514.036347] [    602]   103   602     5967     1003    86016        0             0 systemd-resolve
[11514.036350] [    651]   105   651     1862      470    53248        0          -900 dbus-daemon
[11514.036352] [    652]     0   652     1279       55    45056        0             0 iscsid
[11514.036355] [    653]     0   653     1405     1357    49152        0           -17 iscsid
[11514.036376] [    658]     0   658     6631     2561    86016        0             0 networkd-dispat
[11514.036379] [    660] 584788   660   178782     1207   114688        0             0 updater
[11514.036382] [    661] 584788   661   178945     1227   122880        0             0 agent
[11514.036384] [    664]     0   664     4165      789    73728        0             0 systemd-logind
[11514.036386] [    688]     0   688     3045      768    69632        0         -1000 sshd
[11514.036389] [    714]     0   714      950      516    45056        0             0 atd
[11514.036391] [    730]     0   730      717      395    40960        0             0 agetty
[11514.036393] [    734]     0   734     1098      435    49152        0             0 agetty
[11514.036395] [    741]     0   741    26287     2484   102400        0             0 unattended-upgr
[11514.036397] [    782] 584788   782   179285     1863   126976        0             0 gomon
[11514.036400] [   1535]     0  1535   218640     3823   249856        0          -900 snapd
[11514.036402] [   2068]     0  2068     3449      872    65536        0             0 sshd
[11514.036405] [   2071]  1001  2071     4594      994    73728        0             0 systemd
[11514.036407] [   2072]  1001  2072    42119      797    94208        0             0 (sd-pam)
[11514.036409] [   2158]  1001  2158     3482      779    65536        0             0 sshd
[11514.036411] [   2159]  1001  2159     1529      612    45056        0             0 bash
[11514.036413] [   2173]  1001  2173   370474   148369  1593344        0             0 db1000n
[11514.036415] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1001.slice/session-3.scope,task=db1000n,pid=2173,uid=1001
[11514.036438] Out of memory: Killed process 2173 (db1000n) total-vm:1481896kB, anon-rss:593476kB, file-rss:0kB, shmem-rss:0kB, UID:1001 pgtables:1556kB oom_score_adj:0

@vitich
Copy link

vitich commented Jul 18, 2022 •
edited
Loading 

sudo sysctl -a | grep "vm.overcommit_memory"

And if !=2 do

sysctl -w vm.overcommit_memory=2
sysctl -w vm.overcommit_ratio=80

@seriyps
Copy link

seriyps commented Dec 30, 2022

It looks like there is some kind of memory leak. This is how my RAM consumption graph over 24h looks
image

@rkoten
Copy link

rkoten commented Jan 26, 2023

Getting OOM killed on 0.9.20 running on a Raspberry Pi 3B+ with 1GB RAM. Before this version I used to run various releases from around May 2022 and never got this issue.

@pashagolub
Copy link

Probably the same issue on Windows causing OS to reboot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@seriyps @heximcz @pashagolub @rkoten @vitich @arriven @MidnightRAT @AlexUkr73 @deputinizer