Database crash after remove dockershim

[anatoly-kryzhanovsky]

i have k8s cluster (1.22.3) with harbor installation 2.5.0 (via helm package 1.9.0)
we use internal pgsql database and all works fine

but some time ago we remove docker from all nodes and replace it with containerd (prepare for migration to new version of k8s)

now i can see that pod with database restart occasionally. in logs i see that:

2022-04-26 09:26:35.794 UTC [1] LOG:  database system is ready to accept connections
2022-04-26 09:31:42.391 UTC [1] LOG:  server process (PID 361) exited with exit code 141
2022-04-26 09:31:42.391 UTC [1] LOG:  terminating any other active server processes
2022-04-26 09:31:42.391 UTC [374] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.391 UTC [374] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.391 UTC [374] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.391 UTC [364] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.391 UTC [364] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.391 UTC [364] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.391 UTC [245] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.391 UTC [245] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.391 UTC [245] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.391 UTC [157] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.391 UTC [157] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.391 UTC [157] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.391 UTC [22] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.391 UTC [22] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.391 UTC [22] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.391 UTC [123] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.391 UTC [123] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.391 UTC [123] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.391 UTC [244] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.391 UTC [244] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.391 UTC [244] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.392 UTC [243] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.392 UTC [243] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.392 UTC [243] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.392 UTC [246] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.392 UTC [246] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.392 UTC [246] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:42.432 UTC [69] WARNING:  terminating connection because of crash of another server process
2022-04-26 09:31:42.432 UTC [69] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-04-26 09:31:42.432 UTC [69] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-04-26 09:31:43.031 UTC [375] FATAL:  the database system is in recovery mode
2022-04-26 09:31:43.532 UTC [376] LOG:  PID 243 in cancel request did not match any process
2022-04-26 09:31:46.992 UTC [1] LOG:  all server processes terminated; reinitializing
2022-04-26 09:31:47.545 UTC [377] LOG:  database system was interrupted; last known up at 2022-04-26 09:26:35 UTC
2022-04-26 09:31:47.545 UTC [378] LOG:  PID 245 in cancel request did not match any process
2022-04-26 09:31:50.472 UTC [388] FATAL:  the database system is in recovery mode
2022-04-26 09:31:50.505 UTC [398] FATAL:  the database system is in recovery mode
2022-04-26 09:31:52.283 UTC [399] FATAL:  the database system is in recovery mode
2022-04-26 09:31:56.528 UTC [400] LOG:  PID 246 in cancel request did not match any process
2022-04-26 09:31:58.357 UTC [377] LOG:  database system was not properly shut down; automatic recovery in progress
2022-04-26 09:31:59.367 UTC [377] LOG:  redo starts at 0/63EFC050
2022-04-26 09:31:59.385 UTC [377] LOG:  invalid record length at 0/63F6D038: wanted 24, got 0
2022-04-26 09:31:59.385 UTC [377] LOG:  redo done at 0/63F6D000
2022-04-26 09:32:00.480 UTC [410] FATAL:  the database system is in recovery mode
2022-04-26 09:32:00.511 UTC [420] FATAL:  the database system is in recovery mode
2022-04-26 09:32:00.523 UTC [1] LOG:  received smart shutdown request
2022-04-26 09:32:04.946 UTC [1] LOG:  abnormal database system shutdown
2022-04-26 09:32:05.139 UTC [1] LOG:  database system is shut down

can anybody helm me with that issue?

[zyyw]

exit code 141 means pipe failed. Is it possible that you environment has some mis-configuration?

[anatoly-kryzhanovsky]

hm, i don't know what can be broken.. before i remove dockershim all works fine.
may be you can give me some advise to start investigation?

[anatoly-kryzhanovsky]

may be i can enable some kind of additional logging?
because i still cannot use my installation?

[anatoly-kryzhanovsky]

i turn on all logging in pgsql (level debug5) but i can't see anything new...

2022-05-11 11:21:27.103 UTC [150] STATEMENT:  SELECT count(*) FROM INFORMATION_SCHEMA.tables WHERE table_name = $1 AND table_type = 'BASE TABLE'
2022-05-11 11:21:27.565 UTC [5573] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
2022-05-11 11:21:27.569 UTC [5812] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
2022-05-11 11:21:27.569 UTC [5822] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
2022-05-11 11:21:27.569 UTC [5812] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
2022-05-11 11:21:27.569 UTC [5812] STATEMENT:  SELECT 1
2022-05-11 11:21:27.569 UTC [5822] DEBUG:  StartTransaction(1) name: unnamed; blockState: DEFAULT; state: INPROGRESS, xid/subid/cid: 0/1/0
2022-05-11 11:21:27.569 UTC [5822] STATEMENT:  SELECT 1
2022-05-11 11:21:27.570 UTC [5812] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
2022-05-11 11:21:27.570 UTC [5812] STATEMENT:  SELECT 1
2022-05-11 11:21:27.570 UTC [5822] DEBUG:  CommitTransaction(1) name: unnamed; blockState: STARTED; state: INPROGRESS, xid/subid/cid: 0/1/0
2022-05-11 11:21:27.570 UTC [5822] STATEMENT:  SELECT 1
2022-05-11 11:21:27.570 UTC [5812] DEBUG:  shmem_exit(0): 1 before_shmem_exit callbacks to make
2022-05-11 11:21:27.570 UTC [5812] DEBUG:  shmem_exit(0): 6 on_shmem_exit callbacks to make
2022-05-11 11:21:27.570 UTC [5812] DEBUG:  proc_exit(0): 3 callbacks to make
2022-05-11 11:21:27.570 UTC [5812] DEBUG:  exit(0)
2022-05-11 11:21:27.570 UTC [5812] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:27.570 UTC [5812] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:27.570 UTC [5812] DEBUG:  proc_exit(-1): 0 callbacks to make
2022-05-11 11:21:27.570 UTC [5822] DEBUG:  unexpected EOF on client connection
2022-05-11 11:21:27.570 UTC [5822] DEBUG:  shmem_exit(0): 1 before_shmem_exit callbacks to make
2022-05-11 11:21:27.570 UTC [5822] DEBUG:  shmem_exit(0): 6 on_shmem_exit callbacks to make
2022-05-11 11:21:27.570 UTC [5822] DEBUG:  proc_exit(0): 3 callbacks to make
2022-05-11 11:21:27.570 UTC [5822] DEBUG:  exit(0)
2022-05-11 11:21:27.570 UTC [5822] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:27.570 UTC [5822] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:27.571 UTC [5822] DEBUG:  proc_exit(-1): 0 callbacks to make
2022-05-11 11:21:27.571 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:27.571 UTC [1] DEBUG:  server process (PID 5819) exited with exit code 141
2022-05-11 11:21:27.571 UTC [1] LOG:  server process (PID 5819) exited with exit code 141
2022-05-11 11:21:27.571 UTC [1] LOG:  terminating any other active server processes
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 35
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 5822
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 5812
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 5575
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 5574
2022-05-11 11:21:27.571 UTC [5822] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:27.571 UTC [5822] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:27.571 UTC [5822] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 5573
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 5563
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 4716
2022-05-11 11:21:27.571 UTC [5575] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:27.571 UTC [5575] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:27.571 UTC [5575] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 151
2022-05-11 11:21:27.571 UTC [5574] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:27.571 UTC [5574] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:27.571 UTC [5574] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:27.571 UTC [5812] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:27.571 UTC [5812] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:27.571 UTC [5812] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 150
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 31
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 30
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 32
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 33
2022-05-11 11:21:27.571 UTC [1] DEBUG:  sending SIGQUIT to process 34
2022-05-11 11:21:27.571 UTC [5573] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:27.571 UTC [5573] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:27.571 UTC [5573] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:27.571 UTC [4716] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:27.571 UTC [4716] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:27.571 UTC [4716] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:27.572 UTC [5563] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:27.572 UTC [5563] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:27.572 UTC [5563] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:27.573 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:27.573 UTC [1] DEBUG:  server process (PID 5573) exited with exit code 2
2022-05-11 11:21:27.573 UTC [1] DETAIL:  Failed process was running: SELECT T0."id", T0."digest", T0."content_type", T0."size", T0."status", T0."update_time", T0."version", T0."creation_time" FROM "blob" T0 WHERE T0."digest" IN ($1)
2022-05-11 11:21:27.573 UTC [1] DEBUG:  server process (PID 5812) exited with exit code 2
2022-05-11 11:21:27.573 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:27.573 UTC [151] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:27.573 UTC [151] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:27.573 UTC [151] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:27.573 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:28.344 UTC [150] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:28.344 UTC [150] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:28.344 UTC [150] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:28.344 UTC [150] STATEMENT:  SELECT count(*) FROM INFORMATION_SCHEMA.tables WHERE table_name = $1 AND table_type = 'BASE TABLE'
2022-05-11 11:21:28.345 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:29.586 UTC [1] DEBUG:  forked new backend, pid=5828 socket=9
2022-05-11 11:21:29.587 UTC [5828] FATAL:  the database system is in recovery mode
2022-05-11 11:21:29.587 UTC [5828] DEBUG:  shmem_exit(1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:29.587 UTC [5828] DEBUG:  shmem_exit(1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:29.587 UTC [5828] DEBUG:  proc_exit(1): 1 callbacks to make
2022-05-11 11:21:29.587 UTC [5828] DEBUG:  exit(1)
2022-05-11 11:21:29.587 UTC [5828] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:29.587 UTC [5828] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:29.587 UTC [5828] DEBUG:  proc_exit(-1): 0 callbacks to make
2022-05-11 11:21:29.589 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:29.589 UTC [1] DEBUG:  server process (PID 5828) exited with exit code 1
2022-05-11 11:21:29.972 UTC [34] DEBUG:  writing stats file "pg_stat/global.stat"
2022-05-11 11:21:29.973 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:29.973 UTC [1] DEBUG:  server process (PID 5822) exited with exit code 2
2022-05-11 11:21:30.759 UTC [33] WARNING:  terminating connection because of crash of another server process
2022-05-11 11:21:30.759 UTC [33] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-05-11 11:21:30.759 UTC [33] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2022-05-11 11:21:30.764 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:30.765 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:30.765 UTC [1] DEBUG:  server process (PID 5563) exited with exit code 2
2022-05-11 11:21:30.765 UTC [1] DETAIL:  Failed process was running: SELECT T0."id", T0."project_id", T0."name", T0."value", T0."creation_time", T0."update_time" FROM "project_metadata" T0 WHERE T0."project_id" = $1
2022-05-11 11:21:30.765 UTC [1] DEBUG:  server process (PID 5574) exited with exit code 2
2022-05-11 11:21:30.765 UTC [1] DETAIL:  Failed process was running: SELECT "project_id", "owner_id", "name", "creation_time", "update_time", "deleted", "registry_id" FROM "project" WHERE "project_id" = $1 AND "deleted" = $2
2022-05-11 11:21:30.765 UTC [1] DEBUG:  server process (PID 5575) exited with exit code 2
2022-05-11 11:21:30.765 UTC [1] DETAIL:  Failed process was running: SELECT "project_id", "owner_id", "name", "creation_time", "update_time", "deleted", "registry_id" FROM "project" WHERE "name" = $1 AND "deleted" = $2
2022-05-11 11:21:30.766 UTC [1] DEBUG:  forked new backend, pid=5830 socket=9
2022-05-11 11:21:30.767 UTC [5830] LOG:  PID 5575 in cancel request did not match any process
2022-05-11 11:21:30.767 UTC [5830] DEBUG:  shmem_exit(0): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:30.767 UTC [5830] DEBUG:  shmem_exit(0): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:30.767 UTC [5830] DEBUG:  proc_exit(0): 1 callbacks to make
2022-05-11 11:21:30.767 UTC [5830] DEBUG:  exit(0)
2022-05-11 11:21:30.767 UTC [5830] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:30.767 UTC [5830] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:30.767 UTC [5830] DEBUG:  proc_exit(-1): 0 callbacks to make
2022-05-11 11:21:30.767 UTC [34] DEBUG:  writing stats file "pg_stat/db_16384.stat"
2022-05-11 11:21:30.768 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:30.768 UTC [1] DEBUG:  server process (PID 5830) exited with exit code 0
2022-05-11 11:21:31.479 UTC [1] DEBUG:  forked new backend, pid=5831 socket=9
2022-05-11 11:21:31.479 UTC [5831] LOG:  PID 5563 in cancel request did not match any process
2022-05-11 11:21:31.480 UTC [5831] DEBUG:  shmem_exit(0): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:31.480 UTC [5831] DEBUG:  shmem_exit(0): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:31.480 UTC [5831] DEBUG:  proc_exit(0): 1 callbacks to make
2022-05-11 11:21:31.480 UTC [5831] DEBUG:  exit(0)
2022-05-11 11:21:31.480 UTC [5831] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:31.480 UTC [5831] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:31.480 UTC [5831] DEBUG:  proc_exit(-1): 0 callbacks to make
2022-05-11 11:21:31.482 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:31.482 UTC [1] DEBUG:  server process (PID 5831) exited with exit code 0
2022-05-11 11:21:31.602 UTC [1] DEBUG:  forked new backend, pid=5836 socket=9
2022-05-11 11:21:31.604 UTC [5836] FATAL:  the database system is in recovery mode
2022-05-11 11:21:31.604 UTC [5836] DEBUG:  shmem_exit(1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:31.604 UTC [5836] DEBUG:  shmem_exit(1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:31.604 UTC [5836] DEBUG:  proc_exit(1): 1 callbacks to make
2022-05-11 11:21:31.604 UTC [5836] DEBUG:  exit(1)
2022-05-11 11:21:31.604 UTC [5836] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:31.604 UTC [5836] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:31.604 UTC [5836] DEBUG:  proc_exit(-1): 0 callbacks to make
2022-05-11 11:21:31.605 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:31.606 UTC [1] DEBUG:  server process (PID 5836) exited with exit code 1
2022-05-11 11:21:32.364 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:32.364 UTC [1] DEBUG:  sending signal 9 to process 4716
2022-05-11 11:21:32.364 UTC [1] DEBUG:  sending signal 9 to process 151
2022-05-11 11:21:32.364 UTC [1] DEBUG:  sending signal 9 to process 150
2022-05-11 11:21:33.156 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:33.156 UTC [1] LOG:  statistics collector process (PID 34) was terminated by signal 9: Killed
2022-05-11 11:21:33.615 UTC [1] DEBUG:  forked new backend, pid=5842 socket=9
2022-05-11 11:21:33.617 UTC [5842] FATAL:  the database system is in recovery mode
2022-05-11 11:21:33.617 UTC [5842] DEBUG:  shmem_exit(1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:33.617 UTC [5842] DEBUG:  shmem_exit(1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:33.617 UTC [5842] DEBUG:  proc_exit(1): 1 callbacks to make
2022-05-11 11:21:33.617 UTC [5842] DEBUG:  exit(1)
2022-05-11 11:21:33.617 UTC [5842] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2022-05-11 11:21:33.617 UTC [5842] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2022-05-11 11:21:33.617 UTC [5842] DEBUG:  proc_exit(-1): 0 callbacks to make
2022-05-11 11:21:33.619 UTC [1] DEBUG:  reaping dead processes
2022-05-11 11:21:33.619 UTC [1] DEBUG:  server process (PID 5842) exited with exit code 1

[anatoly-kryzhanovsky]

so because o didnt find any solution i decide to move to external pgsql - so i deploy stateful set with exactly same parameters but based on official pgsql 13.6 image... and all works fine!
this is kind of magic (