Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow clone3 syscall in seccomp filters #28117

Merged
merged 1 commit into from
Oct 11, 2021

Conversation

BlackYoup
Copy link
Contributor

@BlackYoup BlackYoup commented Sep 24, 2021

What does this PR do?

This PR allows the clone3 syscall to be used in the seccomp filters.

clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details

Why is it important?

This is important because it can lead to crashes in softwares using libbeat as a dependency, as it does for apm-server. As soon as glibc 2.34 hits the mainstream distributions, this might become a more encountered problem. Usage of this syscall only requires a glibc update, meaning that binaries compiled before the glibc update will also be impacted.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Should the syscall be allowed for ARM?

How to test this PR locally

I'm not too sure how to do that here. Any pointers would be greatly appreciated. Bare minimum is to have glibc 2.34 installed but I don't know how to trigger the bug directly from the beats project.

Related issues

Use cases

This PR allows an additional linux syscall, namely clone3, to be used to create new threads.

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 24, 2021
@mergify
Copy link
Contributor

mergify bot commented Sep 24, 2021

This pull request does not have a backport label. Could you fix it @BlackYoup? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 24, 2021
@BlackYoup
Copy link
Contributor Author

This pull request does not have a backport label. Could you fix it @BlackYoup? pray
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

* `backport-v./d./d./d` is the label to automatically backport to the `7./d` branch. `/d` is the digit

NOTE: backport-skip has been added to this pull request.

Sorry, I couldn't add a bug label (or didn't find how to) nor can add a label now.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 24, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-10-07T13:58:44.752+0000

  • Duration: 153 min 7 sec

  • Commit: 2855e04

Test stats 🧪

Test Results
Failed 0
Passed 53657
Skipped 5346
Total 59003

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

@simitt simitt added bug Team:Elastic-Agent Label for the Agent team labels Sep 29, 2021
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 29, 2021
@simitt simitt removed the backport-skip Skip notification from the automated backport with mergify label Sep 29, 2021
@simitt simitt requested a review from a team September 29, 2021 06:42
@mergify
Copy link
Contributor

mergify bot commented Sep 29, 2021

This pull request does not have a backport label. Could you fix it @BlackYoup? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

@mergify mergify bot added the backport-skip Skip notification from the automated backport with mergify label Sep 29, 2021
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. This is another case of where if go-seccomp-bpf supported argument filtering we would restrict what flags could be used with clone3 to limit the ability to start new processes.

@andrewkroh andrewkroh added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify libbeat and removed backport-skip Skip notification from the automated backport with mergify labels Sep 29, 2021
@andrewkroh
Copy link
Member

run tests

@simitt
Copy link
Contributor

simitt commented Oct 6, 2021

@ruflin or @andrewkroh is there anything blocking from merging this in?

@andrewkroh
Copy link
Member

It's ready to merge from my POV.

@mergify
Copy link
Contributor

mergify bot commented Oct 7, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b seccomp-clone3 upstream/seccomp-clone3
git merge upstream/master
git push upstream seccomp-clone3

clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details
@simitt simitt merged commit 82507fd into elastic:master Oct 11, 2021
@simitt
Copy link
Contributor

simitt commented Oct 11, 2021

Thank you @BlackYoup for the fix!

mergify bot pushed a commit that referenced this pull request Oct 11, 2021
clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details

(cherry picked from commit 82507fd)
mergify bot pushed a commit that referenced this pull request Oct 11, 2021
clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details

(cherry picked from commit 82507fd)
v1v added a commit to v1v/beats that referenced this pull request Oct 11, 2021
* upstream/master: (73 commits)
  Remove GCP support from Functionbeat (elastic#28253)
  Move labels and annotations under kubernetes.namespace. (elastic#27917)
  Update go release version 1.17.1 (elastic#27543)
  Osquerybeat: Runner and Fetcher unit tests (elastic#28290)
  Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289)
  seccomp: allow clone3 syscall for x86 (elastic#28117)
  packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297)
  [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288)
  [cloud][docker] use the private docker namespace (elastic#28286)
  Update aws-lambda-go library version to 1.13.3 (elastic#28236)
  Deprecate common.Float (elastic#28280)
  Filebeat: Change compatibility test stage to test against previous minor instead of 7.11 (elastic#28274)
  x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling (elastic#28124)
  Explicitly pass http config to doppler consumer (elastic#28277)
  processors/actions/add_fields: Do not panic if event.Fields is nil map (elastic#28219)
  Resolved timestamp for defender atp (elastic#28272)
  [Winlogbeat] Tolerate faults when Windows Event Log session is interrupted (elastic#28191)
  [elastic-agent] proxy requests to subprocesses to their metrics endpoints (elastic#28165)
  Build cloud docker images for elastic-agent (elastic#28134)
  Upgrade k8s go-client library (elastic#28228)
  ...
newly12 pushed a commit to newly12/beats that referenced this pull request Oct 13, 2021
clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details
fearful-symmetry pushed a commit that referenced this pull request Oct 20, 2021
* singleton sysinfo host to avoid frequently collecting host info

* add Host object to Stats object

* update changelog

* set procStats.host to nil if any error calling sysinfo.Host()

* Update aws-lambda-go library version to 1.13.3 (#28236)

* [cloud][docker] use the private docker namespace (#28286)

* [7.x] [DOCS] Update api_key example on elasticsearch output (#28288)

* packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (#28297)

* seccomp: allow clone3 syscall for x86 (#28117)

clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details

* Osquerybeat: Improve handling of osquery.autoload file, allow customizations (#28289)

Previously the osquery.autoload file was overwritten every time on
osquerybeat start and stamped with our extension.
After the change we check the content of the file and do not overwrite it on
each osquerybeat start. This allows the user to deploy their own
extensions if their want and start osquery with that.

* Osquerybeat: Runner and Fetcher unit tests (#28290)

* Runner and Fetcher unit tests

* Fix header formatting

* Tweak test

* Update go release version 1.17.1 (#27543)

* format of conditional build tags has changed
* matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123
* iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests
* go.mod file contains separate require directive for transitive dependencies

* Move labels and annotations under kubernetes.namespace. (#27917)

* Move labels and annotations under kubernetes.namespace.

* Remove GCP support from Functionbeat (#28253)

* Fix build tags for Go 1.17 (#28338)

* [Elastic Agent] Add ability to communicate with Kibana through service token (#28096)

* Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand.

* Add changelog entry.

* Fix go fmt.

* Add username to ASA Security negotiation log (#26975)

* Add username to ASA Security negotiation log

I added the username user.name field to ASA Security negotiation log line.

* adding support for both formats

* adding changelog entry

* updating geo fields in expected output files

* reverse formatting

* reverting to older version of file

* reverting formatting again

* regenrate golden files again

* remove formatting, ready for review

* fixing missing message due to no newline

* fix dissect pattern to fit correctly

Co-authored-by: Marius Iversen <[email protected]>

* x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (#28325)

* Redis: remove deprecated fields (#28246)

* Redis: remove deprecated fields

* Disable generator tests temporarily (#28362)

* Windows/perfmon metricset -  remove deprecated perfmon.counters configuration (#28282)

* remove deprecated config

* changelog

* [Filebeat] - S3 Input - Add support for only iterating/accessing only… (#28252)

* [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths

* Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (#28230)

* Breaking change for 8.0, namespace_annotations replaced by namespace.annotations

* Take care of namespace being nil

* [Heartbeat] Setuid to regular user / lower capabilities when possible (#27878)

partial fix for #27648 , this PR:

Detects if the user is running as root then:
Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present
Attempts to Setuid , Setgid and Setgroups to that user / groups
Invokes setcap to drop all privileges except NET_RAW+ep
This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that.

With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root.

* mage fmt

Co-authored-by: kaiyan-sheng <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
Co-authored-by: Ugo Sangiorgi <[email protected]>
Co-authored-by: Dan Kortschak <[email protected]>
Co-authored-by: Arnaud Lefebvre <[email protected]>
Co-authored-by: Aleksandr Maus <[email protected]>
Co-authored-by: apmmachine <[email protected]>
Co-authored-by: Michael Katsoulis <[email protected]>
Co-authored-by: Noémi Ványi <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: LaZyDK <[email protected]>
Co-authored-by: Marius Iversen <[email protected]>
Co-authored-by: Andrea Spacca <[email protected]>
Co-authored-by: Mariana Dima <[email protected]>
Co-authored-by: Andrew Cholakian <[email protected]>
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
* singleton sysinfo host to avoid frequently collecting host info

* add Host object to Stats object

* update changelog

* set procStats.host to nil if any error calling sysinfo.Host()

* Update aws-lambda-go library version to 1.13.3 (elastic#28236)

* [cloud][docker] use the private docker namespace (elastic#28286)

* [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288)

* packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297)

* seccomp: allow clone3 syscall for x86 (elastic#28117)

clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details

* Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289)

Previously the osquery.autoload file was overwritten every time on
osquerybeat start and stamped with our extension.
After the change we check the content of the file and do not overwrite it on
each osquerybeat start. This allows the user to deploy their own
extensions if their want and start osquery with that.

* Osquerybeat: Runner and Fetcher unit tests (elastic#28290)

* Runner and Fetcher unit tests

* Fix header formatting

* Tweak test

* Update go release version 1.17.1 (elastic#27543)

* format of conditional build tags has changed
* matching of * in regexes was fixed, thus breaking some of our code: golang/go#46123
* iproute package was missing from the new Golang Docker image, thus, we had to add it for our tests
* go.mod file contains separate require directive for transitive dependencies

* Move labels and annotations under kubernetes.namespace. (elastic#27917)

* Move labels and annotations under kubernetes.namespace.

* Remove GCP support from Functionbeat (elastic#28253)

* Fix build tags for Go 1.17 (elastic#28338)

* [Elastic Agent] Add ability to communicate with Kibana through service token (elastic#28096)

* Add ability to communicate with Kibana through service token. Add ability to pass service token to container subcommand.

* Add changelog entry.

* Fix go fmt.

* Add username to ASA Security negotiation log (elastic#26975)

* Add username to ASA Security negotiation log

I added the username user.name field to ASA Security negotiation log line.

* adding support for both formats

* adding changelog entry

* updating geo fields in expected output files

* reverse formatting

* reverting to older version of file

* reverting formatting again

* regenrate golden files again

* remove formatting, ready for review

* fixing missing message due to no newline

* fix dissect pattern to fit correctly

Co-authored-by: Marius Iversen <[email protected]>

* x-pack/filebeat/module/cisco: loosen time parsing and add group and session type capture (elastic#28325)

* Redis: remove deprecated fields (elastic#28246)

* Redis: remove deprecated fields

* Disable generator tests temporarily (elastic#28362)

* Windows/perfmon metricset -  remove deprecated perfmon.counters configuration (elastic#28282)

* remove deprecated config

* changelog

* [Filebeat] - S3 Input - Add support for only iterating/accessing only… (elastic#28252)

* [Filebeat] - S3 Input - Add support for only iterating/accessing only specific folders or datapaths

* Breaking change for 8.0, namespace_annotations replaced by namespace.annotations (elastic#28230)

* Breaking change for 8.0, namespace_annotations replaced by namespace.annotations

* Take care of namespace being nil

* [Heartbeat] Setuid to regular user / lower capabilities when possible (elastic#27878)

partial fix for elastic#27648 , this PR:

Detects if the user is running as root then:
Checks to see if an environment variable BEAT_SETUID_AS (set in our Docker.tmpl) is present
Attempts to Setuid , Setgid and Setgroups to that user / groups
Invokes setcap to drop all privileges except NET_RAW+ep
This PR also fixes the broken syscall filtering in heartbeat, some non-syscall strings were breaking that.

With the changes here elastic-agent can still run as root, but the subprocesses can lower their privileges ASAP. This should also make it possible for heartbeat to safely run ICMP pings and synthetics. Synthetics must run as non-root, but ICMP requires NET_RAW. This lets us be consistent in our docs with the recommendation that elastic-agent run as root.

* mage fmt

Co-authored-by: kaiyan-sheng <[email protected]>
Co-authored-by: Victor Martinez <[email protected]>
Co-authored-by: Ugo Sangiorgi <[email protected]>
Co-authored-by: Dan Kortschak <[email protected]>
Co-authored-by: Arnaud Lefebvre <[email protected]>
Co-authored-by: Aleksandr Maus <[email protected]>
Co-authored-by: apmmachine <[email protected]>
Co-authored-by: Michael Katsoulis <[email protected]>
Co-authored-by: Noémi Ványi <[email protected]>
Co-authored-by: Blake Rouse <[email protected]>
Co-authored-by: LaZyDK <[email protected]>
Co-authored-by: Marius Iversen <[email protected]>
Co-authored-by: Andrea Spacca <[email protected]>
Co-authored-by: Mariana Dima <[email protected]>
Co-authored-by: Andrew Cholakian <[email protected]>
hydrapolic added a commit to hydrapolic/gentoo-1 that referenced this pull request Nov 5, 2021
Glibc-2.34 patch taken from upstream:
elastic/beats#28117

Signed-off-by: Tomáš Mózes <[email protected]>
@rforberger
Copy link

It still happens to me on Fedora 35 with
metricbeat-7.15.2-1.x86_64
filebeat-7.15.2-1.x86_64
When is it going to be fixed there?

v1v pushed a commit that referenced this pull request Nov 11, 2021
clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details

(cherry picked from commit 82507fd)
gentoo-bot pushed a commit to gentoo/gentoo that referenced this pull request Nov 11, 2021
Glibc-2.34 patch taken from upstream:
elastic/beats#28117

Signed-off-by: Tomáš Mózes <[email protected]>
Signed-off-by: Sam James <[email protected]>
@der-eismann
Copy link

der-eismann commented Nov 12, 2021

Can confirm that beats 7.15.2 x64 are still broken on Fedora 35, however the workaround mentioned here works fine.

//Edit: I noticed that with the workaround there is still a warning saying the following, but at least it's running

found unknown syscalls for arch x86_64: clone3

@rforberger
Copy link

Even with
metricbeat-7.16.0-1.x86_64
filebeat-7.16.0-1.x86_64
on Fedora 35 amd64 I still have the problem, that the beats won't start.
I don't want to apply the workarround, because I think it's insecure.
Can anyone fix it? :)

@andrewkroh
Copy link
Member

In 7.16.0 the clone3 syscall is recognized by the seccomp filter config. So you could specify the exact same seccomp policy as is applied in the fixed branch via your config file.

  • Config format: https://www.elastic.co/guide/en/beats/filebeat/current/linux-seccomp.html
  • Fixed policy that includes clone3.
    Names: []string{
    "accept",
    "accept4",
    "access",
    "arch_prctl",
    "bind",
    "brk",
    "chmod",
    "chown",
    "clock_gettime",
    "clone",
    "clone3",
    "close",
    "connect",
    "dup",
    "dup2",
    "epoll_create",
    "epoll_create1",
    "epoll_ctl",
    "epoll_pwait",
    "epoll_wait",
    "exit",
    "exit_group",
    "fchdir",
    "fchmod",
    "fchmodat",
    "fchown",
    "fchownat",
    "fcntl",
    "fdatasync",
    "flock",
    "fstat",
    "fstatfs",
    "fsync",
    "ftruncate",
    "futex",
    "getcwd",
    "getdents",
    "getdents64",
    "geteuid",
    "getgid",
    "getpeername",
    "getpid",
    "getppid",
    "getrandom",
    "getrlimit",
    "getrusage",
    "getsockname",
    "getsockopt",
    "gettid",
    "gettimeofday",
    "getuid",
    "inotify_add_watch",
    "inotify_init1",
    "inotify_rm_watch",
    "ioctl",
    "kill",
    "listen",
    "lseek",
    "lstat",
    "madvise",
    "mincore",
    "mkdirat",
    "mmap",
    "mprotect",
    "munmap",
    "nanosleep",
    "newfstatat",
    "open",
    "openat",
    "pipe",
    "pipe2",
    "poll",
    "ppoll",
    "pread64",
    "pselect6",
    "pwrite64",
    "read",
    "readlink",
    "readlinkat",
    "recvfrom",
    "recvmmsg",
    "recvmsg",
    "rename",
    "renameat",
    "rt_sigaction",
    "rt_sigprocmask",
    "rt_sigreturn",
    "sched_getaffinity",
    "sched_yield",
    "sendfile",
    "sendmmsg",
    "sendmsg",
    "sendto",
    "set_robust_list",
    "setitimer",
    "setsockopt",
    "shutdown",
    "sigaltstack",
    "socket",
    "splice",
    "stat",
    "statfs",
    "sysinfo",
    "tgkill",
    "time",
    "tkill",
    "uname",
    "unlink",
    "unlinkat",
    "wait4",
    "waitid",
    "write",
    "writev",
    .

#28330 needs merged to get the fix into 7.16.

@rforberger
Copy link

@andrewkroh thanks, but I don't want to touch the config file.
I am waiting for the fix to come into 7.16. Thanks again.

andrewkroh pushed a commit that referenced this pull request Dec 12, 2021
* seccomp: allow clone3 syscall for x86 (#28117)

clone3 is a linux syscall that is now used by glibc starting version
2.34. It is used when pthread_create() gets called. Current seccomp
filters do not allow this syscall leading to crashes like
runtime/cgo: pthread_create failed: Operation not permitted

See elastic/apm-server#6238 for more details

(cherry picked from commit 82507fd)

Co-authored-by: Arnaud Lefebvre <[email protected]>
Co-authored-by: Jaime Soriano Pastor <[email protected]>
@rforberger
Copy link

It works now again with version 7.16.2-1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify bug libbeat Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants