Distributed installation sensor retains memory then aborts.

[mstannh]

We have a distributed Security Onion V 2.3.140 installation running on VirtualBox 6.1.36 r152435 {Qt5.9.7) extensions are not install
Nodes are Manager, Search, Sensor.
The installation is working fine except that the sensor is retaining memory and eventually aborts after a couple of hours
The system is receiving packets about every minute however there is a cycle where it is extremely busy and then greatly reduced traffic
Peak times start on the half hour and run for 30 minutes. During peak periods the the sensor ingests approximately 450 GB of data. during reduced traffic periods the sensor ingests 500MB of data.
We are filtering port 443 and ARP.

Symptoms are:

• Each peak period uses approx 16 -20 GB of RAM which remains allocated until the next peak period
• After approximately 3 Peak periods (3 hours) all RAM is consumed and SWAP is engaged and
• On the Senor console we see out of memory errors (reference attachment

Please reference the attached images that show consumption of resources an error messages.

[isaacgolding]

What are your hardware specs?

[mstannh]

Sure........
2 x Intel Xeon E5-2699 18-Core 2.3 Ghz CPUs
256 Gbytes memory
10 Gbase T ethernet
NFS mounted disks - Oracle ZS5-4 DE3-24C
and we are monitoring inbound / outbound traffic from a mirrored port on an Oracle ES2-64 switch.

[isaacgolding]

Sorry I wasn’t detailed enough. What are the specs of the VM that SO is installed in?

On Mon, Aug 22, 2022 at 2:09 PM mstannh ***@***.***> wrote: Sure........ 2 x Intel Xeon E5-2699 18-Core 2.3 Ghz CPUs 256 Gbytes memory 10 Gbase T ethernet NFS mounted disks - Oracle ZS5-4 DE3-24C and we are monitoring inbound / outbound traffic from a mirrored port on an Oracle ES2-64 switch. — Reply to this email directly, view it on GitHub <#8567 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AY3AYRKCNMWG5NUBS3PC2TTV2O67NANCNFSM57HXIP5Q> . You are receiving this because you commented.Message ID: <Security-Onion-Solutions/securityonion/repo-discussions/8567/comments/3450048 @github.com>

-- Isaac Golding Owner Isaac Golding Services Co. <https://isaacgolding.com/>

[mstannh]

VM Specs are:

• VirtualBox 6.1.36 r152435 {Qt5.9.7)
• 18 Cores, 64 Gbytes Memory.
• Chipset: PIIX3
• Processor Extended Features enabled: PAE/NX, Nexted VT-x/AMD-V
• storage: SATA Port 0, AHCI Type Device, Host I/O/ Cache enabled.
• Networks: Paravirtualized Network (virtio-net)

[isaacgolding]

So my specs are similar to yours.... however I'm only getting the same issue about once a day. I know chasing down a memory problem can take a lot of time so I've been waiting on the next release to see if the problem goes away before dedicating hours/days to chasing the issue.

And on that note version 2.3.150 just dropped today. I'm running soup right now and assuming everything goes well I can tell you tomorrow if this "runaway" memory issue goes away.

[mstannh]

Thanks for that input.
We are also executing SOUP at present.
If the upgrade does not help the memory leak...........I am not sure what we have as a plan.
Re-booting the sensor node every 2.5 hours is not a good plan.
What are your thoughts on a massive increase SWAP resources such as ...in our environment increasing from 8 GB to 80-120-GB.

[defensivedepth]

What does disk wait look like? (Builtin Grafana dashboards will show you that)

My initial thought is the NFS is causing issues - from the docs: https://docs.securityonion.net/en/2.3/hardware.html#storage

We only support local storage. Remote storage like SAN/iSCSI/FibreChannel/NFS increases complexity and points of failure, and has serious performance implications. You may be able to make remote storage work, but we do not provide any support for it. By using local storage, you keep everything self-contained and you don’t have to worry about competing for resources. Local storage is usually the most cost efficient solution as well.

[mstannh]

It appears that the main users of the memory are:

• Suricata-main
• Stenographer
• Zeek (to a lesser extent)

[mstannh]

Latest occurrence

• abort at 1022
• restart at 1042
• fresh run at 1048
• run ends at 1104 and memory is retained until next run
• next run start at 1131 compounding memory usage
  

[dougburks]

What is the average and peak traffic in terms of Mbps?

How many Suricata workers are running?

How many Zeek workers are running?

How many NIDS rules are enabled?

Are you running any custom Zeek scripts?

Have you tried increasing RAM?

Is there additional traffic that you can filter out?

Have you tried without NFS?

Have you tried running the sensor on bare metal instead of VirtualBox?

[mstannh]

Thanks for replying Doug:

Q -What is the average and peak traffic in terms of Mbps?
A- AVG: 2,500Mbps Peak: 9,300 Mbps
Note: Sensor is receiving in/out-bound traffic across a 10Gb interface.

Q- How many Suricata workers are running?
A- 8

Q - How many Zeek workers are running?
A - 8

Q - How many NIDS rules are enabled?
A- we are using the Emerging all rule set so about 35K

Q- Are you running any custom Zeek scripts?
A- No

Q- Have you tried increasing RAM?
A- Yes - currently configured with 64 Gbytes

Q- Is there additional traffic that you can filter out?
A- Must have all packets. For testing we filtered out arp and 443 but it did not remove the issue

Q- Have you tried without NFS?
A- The Virtual Machine guests do not use NFS.
NFS is used by the host machine only.

Q- Have you tried running the sensor on bare metal instead of VirtualBox?
A- Considered, but not attempted.

[dougburks]

Q -What is the average and peak traffic in terms of Mbps?
A- AVG: 2,500Mbps Peak: 9,300 Mbps
Q- How many Suricata workers are running?
A- 8
Q - How many Zeek workers are running?
A - 8

Depending on the actual processor and what other VMs are running, you may need more CPU cores so that you can enable more Suricata and Zeek workers:
https://docs.securityonion.net/en/2.3/hardware.html#cpu

One option might be to switch from Zeek metadata to Suricata metadata as this should lower your CPU load:
https://docs.securityonion.net/en/2.3/suricata.html#metadata

Q - How many NIDS rules are enabled?
A- we are using the Emerging all rule set so about 35K

You might need to tune your ruleset to lower your system load:
https://docs.securityonion.net/en/2.3/managing-alerts.html#identifying-rule-categories

Q- Have you tried increasing RAM?
A- Yes - currently configured with 64 Gbytes

You may need more RAM:
https://docs.securityonion.net/en/2.3/hardware.html#ram

Q- Have you tried without NFS?
A- The Virtual Machine guests do not use NFS.
NFS is used by the host machine only.

NFS can cause issues even if only used for the host:
https://docs.securityonion.net/en/2.3/hardware.html#storage

Q- Have you tried running the sensor on bare metal instead of VirtualBox?
A- Considered, but not attempted.

From https://docs.securityonion.net/en/2.3/hardware.html#virtualization:
We recommend dedicated physical hardware (especially if you’re monitoring lots of traffic) to avoid competing for resources. Sensors can be virtualized, but you’ll have to ensure that they are allocated sufficient resources.

[isaacgolding]

For what it's worth I'm still seeing the same or similar error. I'll start digging into logs and try to find who/what is getting killed.

[mstannh]

Can you elaborate on this image showing min max average and node type?

[mstannh]

I am continuing this thread after a bit of a break as the problem still exists.
We went to Augusta training and conference and talked to some knowledgeable folks but we are still experience the issues as stated previously.
I will restate here.
We have 1 forward node / sensor running as a VM on VirtualBox. It monitors inbound and outbound traffic on a mirrored port of an Oracle switch.
Port 53 and the 3 Highest talking Hosts are filtered out at the global.sls. That filters out about 75% of the traffic on the switch.
There are two issues (not sure if they are related or need to be on separate discussions):
1- Our MBTF for the sensor is about 5 hours as the memory reaches >95%. It aborts and we have to reboot.

2 - During the time the sensor is on line we have extremely high Steno packet drops

Any suggestions as to

• Could these symptoms be related?
• Do they need to be separate discussions?
• Where should I start trouble shooting this?

[dougburks]

Have you considered my recommendations above?
#8567 (reply in thread)

You could also try disabling Stenographer temporarily to see how that affects system performance.

[mstannh]

I looked at /opt/so/log/stenographer/ to examine Diagnostic logging for Stenographer, There was nothing there. Do i need to configure a value or run a commend to run/collect steno diagnostic data?

[mstannh]

Please disregard my last comment as I was on the wrong node.
I found the Steno logs on the sensor (duh) and am looking through them, but not sure what I am looking for, however I see the drops.