Kernel module isn't installed to module dir when using falco-driver-loader --download

[johananl]

Describe the bug

When using falco-driver-loader --download, the kernel module is downloaded and loaded using insmod with an absolute path. However, the module isn't installed/symlinked in /lib/modules/$(uname -r). As a result, modprobe falco fails which in turn causes falco-kmod-inject.service to fail which is a dependency of falco-kmod.service.

Interestingly, when running falco-driver-loader --compile (or when the download fails and the script falls back to local compilation), the module is symlinked because dkms is used in this case and does the linking automatically.

How to reproduce it

curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
  sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg

echo 'deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main' | sudo tee /etc/apt/sources.list.d/falcosecurity.list

sudo apt update
sudo apt install -y dkms make linux-headers-$(uname -r)
sudo FALCO_FRONTEND=noninteractive apt-get install -y falco

sudo falco-driver-loader
lsmod | grep falco # module loaded
sudo systemctl start falco-kmod # fails
sudo systemctl start falco-kmod-inject # fails

Root cause:

$ sudo modprobe falco
modprobe: FATAL: Module falco not found in directory /lib/modules/5.15.0-72-generic

Workaround:

sudo ln -s /root/.falco/4.0.0+driver/x86_64/falco_ubuntu-generic_5.15.0-72-generic_79.ko /lib/modules/$(uname -r)/falco.ko
sudo depmod

Now systemctl start falco-kmod works.

Expected behaviour

I expected to get a consistent behavior, regardless of whether the kernel module was downloaded or compiled locally.

Screenshots

Environment

• Falco version:

Wed May 24 18:41:37 2023: Falco version: 0.34.1 (x86_64)
Wed May 24 18:41:37 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Falco version: 0.34.1
Libs version:  0.10.4
Plugin API:    2.0.0
Engine:        16
Driver:
  API version:    3.0.0
  Schema version: 2.0.0
  Default driver: 4.0.0+driver

• System info:

{
  "machine": "x86_64",
  "nodename": "snappy-ewe",
  "release": "5.15.0-72-generic",
  "sysname": "Linux",
  "version": "#79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023"
}

• Cloud provider or hardware configuration:
  n/a
• OS:

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

• Kernel:
  Linux snappy-ewe 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Installation method:
  Deb package

Additional context

[Andreagit97]

ehi @johananl thank you for reporting this, we will take a look with @FedeDP

[FedeDP]

Hi!

sudo ln -s /root/.falco/4.0.0+driver/x86_64/falco_ubuntu-generic_5.15.0-72-generic_79.ko /lib/modules/$(uname -r)/falco.ko

Yep this is something known; i tried to address it but had to revert it for reasons: #2388, that reverted 3087362.

/cc @leogr

[leogr]

.deb and .rpm packages are intended to work with --compile. Is any compelling reason to use --download in such cases? 🤔

[johananl]

.deb and .rpm packages are intended to work with --compile. Is any compelling reason to use --download in such cases? 🤔

AFAICT - no, as long as the dependencies for local compilation exist. But --download does seem to be the default when running the script without flags, so we may want to make sure it works.

[leogr]

But --download does seem to be the default when running the script without flags, so we may want to make sure it works.

Actually, both options are enabled by default:

  --compile      try to compile the driver locally (default true)
  --download     try to download a prebuilt driver (default true)

The expected behavior is that the script first tries with --download, then if the download file, uses--compile as a fallback.

Moreover, in the case of the kmod, the .deb will only use --compile at the post-install step since it is intended to be used in conjunction with dkms:

falco/scripts/debian/postinst.in

Lines 80 to 84 in 79b9d0f

	"kmod")
	# Only compile for kmod, in this way we use dkms
	echo "[POST-INSTALL] Call 'falco-driver-loader --compile module':"
	falco-driver-loader --compile module
	;;

That being said, I believe the root cause of the problem is as follows:

Since you used FALCO_FRONTEND=noninteractive and then manually ran sudo falco-driver-loader, it only downloaded the driver to /root/.falco/4.0.0+driver/x86_64/falco_ubuntu-generic_5.15.0-72-generic_79.ko; that interfered with the subsequent invocation of systemctl start falco-kmod-inject which assumes the driver to be installed via dkms.

If it's confirmed, we should either:

• recommend only use falco-driver-loader --compile in a such a case
• make falco-kmod-inject resilient

cc @FedeDP @Andreagit97

[johananl]

Actually, both options are enabled by default

That's right, I meant that --download is what takes place in practice for deployments with internet connectivity, and argued that it makes sense for the Falco service to behave the same way regardless of whether the kmod was downloaded or compiled locally.

To elaborate, my use case is a scripted installation of Falco, therefore I can't rely on manually interacting with a graphical prompt. I used FALCO_FRONTEND=noninteractive only here in the reproduction instructions. My actual installation uses Ansible and does the following:

• Add the Falco GPG key.
• Add the Falco APT repo.
• Install the falco DEB package.
• Run falco-driver-loader --compile.

@leogr you're saying that installing the DEB package should be enough because the kmod gets compiled in postinstall? I'm pretty sure I didn't have the kernel module in place after installing the DEB package, which is what prompted me to run falco-driver-loader in the first place 😕 I'll double check this. Maybe when dkms isn't present the kmod simply doesn't get compiled.

[johananl]

I've just reproduced the problem on my end. After installing the DEB package, lsmod | grep falco gives no output and modprobe falco gives "module falco not found". I'm trying to check the APT output to see if the module was compiled in postinstall.

[johananl]

Looking at the postinstall script, I can see we're compiling the kmod only when chosen_driver is kmod, which defaults to empty and is set to kmod only using dialog:

falco/scripts/debian/postinst.in

Line 19 in 9a4205c

chosen_driver=

falco/scripts/debian/postinst.in

Line 48 in 9a4205c

chosen_driver="kmod"

falco/scripts/debian/postinst.in

Lines 79 to 83 in 9a4205c

	case "$chosen_driver" in
	"kmod")
	# Only compile for kmod, in this way we use dkms
	echo "[POST-INSTALL] Call 'falco-driver-loader --compile module':"
	falco-driver-loader --compile module

So, unless I'm missing something we aren't building the kmod in DEB postinstall when dialog isn't present. This means that one has to build the kmod by running falco-driver-loader manually not only during binary installation (the .tar.gz archive) but also during DEB installation without dialog. I'd say we could clarify this fact in the docs, or figure out a way to specify that we want to compile the kmod in postinstall also during non-interactive installations. I wonder if it's possible to configure a DEB installation using custom env vars Maybe we could do something like FALCO_MODE=kmod apt-get install -y falco and conditionally compile the kernel module based on the value of this var.

Lastly, would it make sense to compile the kmod by default, and just not insert it when using eBPF mode? Is there any harm in doing so other than a couple more dependencies during installation? I'm trying to figure out how the UX can become simpler and more straightforward by default: Right now it feels like you need internal knowledge of how Falco works just to get it running, at least in my case.

[johananl]

Now I see that this is a potential duplicate of #2431.

[Andreagit97]

. I wonder if it's possible to configure a DEB installation using custom env vars Maybe we could do something like FALCO_MODE=kmod apt-get install -y falco and conditionally compile the kernel module based on the value of this var.

Lastly, would it make sense to compile the kmod by default, and just not insert it when using eBPF mode? Is there any harm in doing so other than a couple more dependencies during installation? I'm trying to figure out how the UX can become simpler and more straightforward by default: Right now it feels like you need internal knowledge of how Falco works just to get it running, at least in my case.

These are all good points, our actual setup is not so intuitive for the end user. Before refactoring another time our systemd units I think we need 2 key points:

• Deprecate falco-driver-loader #2675
• new: driver selection in falco.yaml #2413

These 2 points should allow us to have a great intuitive design and maybe just a single configurable systemd unit!

Thanks @johananl for pointing this and other UX issues!

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[Andreagit97]

/remove-lifecycle stale

[Andreagit97]

we are working on that

[FedeDP]

I see 2 fixes for this:

• Add use of FALCO_DRIVER_CHOICE and FALCOCTL_ENABLED env vars #2773 -> to allow users without dialog package to still retain the possibility to customize the driver being automatically installed
• making falco-kmod.service smarter, ie: insmodding /root/.falco/$driver_version/$arch/foo.ko instead of just modprobeing the driver.

[FedeDP]

For the latter one, we need to make sure that the driver is always copied to /root/.falco/$driver_version/$arch/foo.ko. This is already done by the new falcoctl driver-loader implementation that will be alive with Falco 0.37.0.

[FedeDP]

Indeed, we cannot change falco-kmod.service: for host installation the kmod is always expected to be installed via dkms to ensure that it gets properly bumped in case the user upgrades its kernel.
In the end, the only real solution is the usage of FALCO_DRIVER_CHOICE : #2773
cc @leogr

[leogr]

In the end, the only real solution is the usage of FALCO_DRIVER_CHOICE : #2773

I agree 👍

[FedeDP]

FALCO_DRIVER_CHOICE and the new falcoctl based driver-loader are now on Falco master.
I think we can close this one.
/close

[poiana]

@FedeDP: Closing this issue.

In response to this:

FALCO_DRIVER_CHOICE and the new falcoctl based driver-loader are now on Falco master.
I think we can close this one.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.