Teleport RPMs won't allow enhanced session recording to work

[webvictim]

Description

What happened: Installing Teleport from either the RPM repo (https://rpm.releases.teleport.dev) or the RPMs on https://goteleport.com/download will result in enhanced session recording failing to load.

Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: 2022-04-01T13:18:35Z WARN [PROC:1]    Teleport process has exited with error. error:[
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: ERROR REPORT:
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: Original Error: *trace.BadParameterError operating system does not support enhanced session recording, check Teleport documentation for more details on supported operating systems, kernels, and configuration
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: Stack Trace:
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: /go/src/github.com/gravitational/teleport/lib/service/service.go:1865 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initSSH.func1
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:494 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:263 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: User Message: operating system does not support enhanced session recording, check Teleport documentation for more details on supported operating systems, kernels, and configuration] service:ssh.node service/supervisor.go:268
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: 2022-04-01T13:18:35Z DEBU [PROC:1]    Broadcasting event. event:ServiceExitedWithError service/supervisor.go:370
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: 2022-04-01T13:18:35Z DEBU [PROC:1]    Service is completed and removed. service:ssh.node service/supervisor.go:239
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: 2022-04-01T13:18:35Z ERRO [PROC:1]    Critical service ssh.node has exited with error operating system does not support enhanced session recording, check Teleport documentation for more details on supported operating systems, kernels, and configuration, aborting. service/signals.go:144
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: 2022-04-01T13:18:35Z DEBU [PROC:1]    Broadcasting event. event:TeleportExit service/supervisor.go:370
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: ERROR REPORT:
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: Original Error: *trace.BadParameterError operating system does not support enhanced session recording, check Teleport documentation for more details on supported operating systems, kernels, and configuration
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: Stack Trace:
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: /go/src/github.com/gravitational/teleport/lib/service/service.go:1865 github.com/gravitational/teleport/lib/service.(*TeleportProcess).initSSH.func1
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:494 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:263 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
Apr 01 13:18:35 ip-172-31-34-128.us-east-2.compute.internal teleport[3636]: User Message: operating system does not support enhanced session recording, check Teleport documentation for more details on supported operating systems, kernels, and configuration

When Teleport is installed from the tarball on https://goteleport.com/download, enhanced session recording works normally.

What you expected to happen: Enhanced session recording should also work when Teleport is installed via RPM.

I suspect the reason is that we're bundling CentOS 7 RPMs everywhere and these don't have BPF/BTF support properly compiled in.

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.

1. Install an RPM on an Amazon Linux 2 AMI running kernel 5.10.102-99.473.amzn2.x86_64 or similar (this is the default)
2. Enable enhanced session recording in Teleport ssh_service config
3. Observe failure to start

Server Details

• Teleport version (run teleport version): Teleport v9.0.3 git:v9.0.3-0-g1cf2b3e17 go1.17.7
• Server OS (e.g. from /etc/os-release):

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS EC2

[zmb3]

If I had to guess the fix for #10686 broke this.

[russjones]

I think once we drop the glibc version, we can put the regular binaries into the RPM again and it should work.

Or we add BPF support in when building on CentOS 7.

[webvictim]

I remember BPF support on CentOS 7 being too difficult to add at the time, but can't remember why. This was pre-BPF rewrite though so the situation is likely different now.

[russjones]

BPF functionality won't actually work on CentOS 7, requires a newer kernel to run, but I think we should be able to build it on CentOS 7.

[webvictim]

I'm pretty sure this has been fixed now? @jakule @russjones

[jakule]

All our releases are built on CentOS 7 and all 64-bit ones have BPF support. BPF still won't work on CentOS 7 as the kernel in CentOS 7 is just too old, but RPM installed on a supported system should work.