Runaway process on Guardian start (v.2.27.1)

[marko-zeroproxy]

Note: Originally logged as Discussion #4403: #4403

Problem description

Running a Guardian instance on EC2, 8GB ram, 8GB swap file. Suddenly started consuming 100% swap, memory and CPU. When we reboot the image or the entire server, once we restart the instance it immediately runs back up within 2-3 mins. All we see a increasing number of node processes that are eating up all resources until the server crashes within 5-7 minutes.

We see dozens of instances of one issue before it crashes. We also see a lot of 500 error popups in the UI before it crashes. This started after we initialed a dozen policy imports with about 10 documents per policy via the API, using one at a time.

Step to reproduce

Steps to reproduce the behavior:

1. Start Guardian
2. Onboard ~10 hedera/guardian accounts
3. Create ~5 VC documents per account using an existing Guardian policy

Expected behavior

System resource utilization should not continue or increase when there is no UI, API or other activity

Screenshots