kubernetes - document how to get running on Amazon EKS

[mmguero]

The Kubernetes deployment (#149) has been released, but I still need to figure out how to get it working on Amazon AWS EKS.

I've got a work in progress document in my development fork where I'm putting the steps I've taken as I've tried to figure it out.

So far I've actually gotten Malcolm to deploy and start up okay, but have run into issues figuring out the right approach for the shared storage for the persistent volumes (a more complicated way of saying "the file systems the various Malcolm containers need to mount in order to share"). I was able to get a little bit further along using the gp2 storage type (which is like an ECS instance's default) but found that it didn't support multi-attach, which (barring some major architectural changes to Malcolm as a bunch of containers share mount point for zeek logs, pcap files, suricata logs, etc.) is a pretty firm requirement. I tried switching over to the io2 storage class, but in creating my storage volumes I ran into an error like "An error occurred (MaxIOPSLimitExceeded) when calling the CreateVolume operation: You have exceeded your maximum io2 IOPS limit of 100000 IOPS in this region. Please contact AWS Support to request an Elastic Block Store service limit increase" so I stopped that pretty quick and deleted the volumes. I tried it with io1 as well, but got a similar error about MULTI_NODE_MULTI_WRITER not supported for the io1 storage, and something about an instance not existing for the gp2 storage (which i created for the opensearch volumes as they don't require readwritemany).

So right now I'm kind of stuck on the storage side of it. I'm thinking that maybe I should try AWS NFS File Shares with EFS (?).

[mmguero]

This blog post says:

Comparing EFS to another popular Amazon service, EBS, the major advantage of EFS is that it offers shared storage. EBS has higher performance, but it is attached to a specific EC2 machine instance and cannot be shared across multiple machines.

Which seems to indicate that EFS seems to be what I need to look into.

[mmguero]

Just wanted to happily report that I was able to figure out getting Malcolm running on EKS (and even accessible from the internet). Using ALB (instead of Ingress-NGINX) when deploying on EKS seemed to work well.

I've been documenting everything in the Malcolm documentation in my development fork, the latest and greatest docs regarding Malcolm in K8S (and now EKS) can be found here as they've yet to be merged into the main Malcolm repos:

• Deploying Malcolm with Kubernetes: Ingress Controllers
  • I've added a section here where I describe and give examples for using Ingress-NGINX for self-hosting or other non-AWS Kubernetes installations, as that has worked well for us in-house. I also give an example manifest for and suggesting using the ALB controller for EKS deployment.
• Deploying Malcolm on Amazon Elastic Kubernetes Service (EKS)
  • I've updated my instructions here for how I got my cluster and EFS configured to run Malcolm on EKS, including the manifest example and references for ALB. A lot of this is still pretty manual, @piercema and I are going to look at Terraform to do some of the cluster provisioning and whatnot. There are a few changes I need to make to some of my containers' initialization to ensure the necessary subdirectories get created underneath some of the persistent volume mount points on startup, but that won't be difficult.

One thing I still need to figure out is how (if?) ALB can handle a couple of the other services Malcolm uses to receive logs. I've got ALB working fine for the main HTTPS endpoint at 443/tcp and for the OpenSearch REST API endpoint at 9200/tcp, but traditionally Malcolm's Logstash instance accepts connections on 5044/tcp to receive logs from Beats. This is TLS-encrypted but not HTTPS. Can the ALB ingress controller be configured to allow me to accept arbitrary TCP socket connections that are not HTTP(S)-based?

[mmguero]

Using the instructions I've outlined in the last comment I've got it up and running on EKS, connecting to HTTPS for 443 and 9200 and with fluent-bit to 5044 and 5045 for logs and metadata. There will surely be improvements but I'm going to mark as closed for now.