Helm install for falco not working on EKS 1.16 -- Unable to download precompiled falco-probe module for 4.14.173-137.229.amzn2.x86_64

[ianhundere]

Describe the bug
We recently updated from EKS 1.15 to 1.16. We were using 0.19.0 of Falco before this without issue.
We tried using the latest falcosecurity/falco:master image, but that did not resolve the issue.

Error! echo
Your kernel headers for kernel 4.14.173-137.229.amzn2.x86_64 cannot be found at
/lib/modules/4.14.173-137.229.amzn2.x86_64/build or /lib/modules/4.14.173-137.229.amzn2.x86_64/source.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/a259b4bf49c3330d9ad6c3eed9eb1a31954259a6/build/make.log
* Trying to load a system falco-probe, if present
* Trying to find precompiled falco-probe for 4.14.173-137.229.amzn2.x86_64
Found kernel config at /host/boot/config-4.14.173-137.229.amzn2.x86_64
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/falco-probe-a259b4bf49c3330d9ad6c3eed9eb1a31954259a6-x86_64-4.14.173-137.229.amzn2.x86_64-f0c8ced41ae4d0e71aa715068964ce9f.ko
curl: (22) The requested URL returned error: 404 Not Found
Download failed, consider compiling your own falco-probe and loading it or getting in touch with the Falco community
Tue May  5 15:09:32 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Tue May  5 15:09:32 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Tue May  5 15:09:32 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue May  5 15:09:33 2020: Unable to load the driver. Exiting.
Tue May  5 15:09:33 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco-probe module is loaded.. Exiting.

How to reproduce it

helm install falcohelm stable/falco in the EKS 1.16 env.

Expected behaviour

No errors when Falco pods spin up.

Environment

• Falco version:

0.18.0 / 0.19.0 / 0.22.0 / master

• Cloud provider or hardware configuration: EKS cluster (1.16)
• OS: Amazon Linux 2
• Installation method: Helm Install

[leogr]

Hi @ianhundere

the driver loader in the current master branch should download drivers from https://dl.bintray.com/falcosecurity/driver.

Since in your description I see Trying to download precompiled module from https://s3.amazonaws.com/download.draios..., I'm not sure which Falco version produced the log that you have attached.

Could you also attach the log you get when using falcosecurity/falco:master, please?

Thank you in advance.

[ianhundere]

Sure thing.

* Setting up /usr/src links from host
* Unloading falco module, if present
* Running dkms build failed, couldn't find /var/lib/dkms/falco/47374b2b73734d509f3c99890c80be5242021c3d/build/make.log
* Trying to load a system falco driver, if present
Detected an unsupported target system, please get in touch with the Falco community
* Trying to find a prebuilt falco module for kernel 4.14.173-137.229.amzn2.x86_64
Thu May  7 13:29:53 2020: Falco initialized with configuration file /etc/falco/falco.yaml
Thu May  7 13:29:53 2020: Loading rules from file /etc/falco/falco_rules.yaml:
Thu May  7 13:29:54 2020: Loading rules from file /etc/falco/falco_rules.local.yaml:
Thu May  7 13:29:55 2020: Unable to load the driver. Exiting.
Thu May  7 13:29:55 2020: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

[leogr]

Tnx!
I believe the driver for that has not built by the driverkit build grid.
@leodido @fntlnz should we move this issue to test-infra ?

[leodido]

These two lines

Detected an unsupported target system, please get in touch with the Falco community
* Trying to find a prebuilt falco module for kernel 4.14.173-137.229.amzn2.x86_64

make me think there's a problem in the get_target_id function of the falco-driver-loader script.

Given the kernel 4.14.173-137.229.amzn2.x86_64, first of all it'd have detected the target is amazonlinux2.

[leogr]

make me think there's a problem in the get_target_id function of the falco-driver-loader script.

Or just the helm chart is not mounting /host/etc (see the "if" condition)?

[leodido]

Yes, definitely

[moserke]

I can confirm that when I mount /host/etc falco starts for 4.14.173-137.229.amzn2.x86_64.

[leogr]

See also falcosecurity/falco#1203

[leogr]

Hi @ianhundere

as temporary workaround, you can clone the helm chart locally and modify it to always mount /host/etc. Basically you have to remove both the "if"s here and here.

Let me know if it works for you. Anyway, I'm transferring this issue to the new charts repo so it can be fixed in future releases.