kubernetes deployment - inotify issue

[mmguero]

Sub-issue of #149.

A lot of the code in Malcolm that deals with processing PCAP, processing logs, processing extracted files, relies on inotify (here is one good example). However, with a PersistentVolume over NFS, it doesn't appear we're getting those events. That would make sense, as from what I understand inotify is a kernel thing and it may not be propogated over NFS. We need to figure out if this is something we can "fix" magically and make work everywhere with all types of persistentvolumes (which, unfortunately, I kind of doubt) or come up with something else (probably some kind of polling). Grepping inotify in the source code will show where we need to do it.

[mmguero]

This is interesting, and I think this makes sense. I think we can do some kind of polling without actually having to rewrite all of our stuff that uses inotify.

I set up this stupid test:

1. Get inside of the pcap-capture container on a Kubernetes worker inside a VM
2. Run: inotifywait -m -e close_write --format '%w%f' "${PCAP_UPLOAD_DIR}" | while read NEWFILE; do echo "$NEWFILE"; done
3. Outside of kubernetes (just on the host that's exporting NFS share itself) I write a file. No notifications are triggered.
4. Inside of the same kubernetes pod in another shell, I write a file: the inotify notification is triggered
5. Inside of a different kubernetes container on the same node in a shell, I write a file: the inotify notification is triggered
6. Inside of a different kubernetes container on a different same node in a shell, I write a file: the inotify notification is NOT triggered

So this makes sense: if you're in the same kernel space as the thing doing the inotify when you make the changes, the changes are seen.

I think what this means is this: I do need to introduce polling, but I can do it in a generic way abstracted from the processes that are using inotify themselves. If I set up some script to run in the background of these containers that are doing the inotify stuff to be given a list of directories to watch and some parameters to determine that a file is "done" being written to (I can't rely on inotify, as we've determined, and I can see that lsof has the same issue), so maybe how long ago is the modify time, etc., then just simply "touch" the file, then the rest of the inotify stuff will work as designed.

[mmguero]

The python library watchdog (github) might be a good choice for this without reinventing the wheel. It specifically has a pollingobserver designed for this exact scenario.

[mmguero]

Files that need adjusted:

• filebeat/scripts/filebeat-watch-zeeklogs-uploads-folder.sh
• pcap-monitor/scripts/watch-pcap-uploads-folder.sh
• shared/bin/pcap_watcher.py
• shared/bin/zeek_carve_watcher.py

I think I will use watchdog directly in the .py scripts, and figure out a way to hook into what it's doing polling-wise and touch the files for the bash stuff.

[mmguero]

Hmm, although there might be some issue with the fact the close event doesn't happen with the polling observer.

[mmguero]

This has been a bit of a hassle due to trying to distinguish between "modify file attributes" (like access time) and "modify file contents" events (see gorakhargosh/watchdog#800 and gorakhargosh/watchdog#260) but I think I've got it working. It seems to all work in both polling and non-polling mode now. I will test it out in k3s with NFS storage using polling next week.