Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Control Plane Fails to Start on Fedora 34 Silverblue #2521

Closed
adambkaplan opened this issue Nov 2, 2021 · 13 comments
Closed

Control Plane Fails to Start on Fedora 34 Silverblue #2521

adambkaplan opened this issue Nov 2, 2021 · 13 comments
Labels
area/provider/podman Issues or PRs related to podman area/rootless Issues or PRs related to rootless containers kind/bug Categorizes issue or PR as related to a bug.

Comments

@adambkaplan
Copy link

What happened:

When using the podman provider, the KinD control plane fails to come up on Fedora 34 Silverblue.

What you expected to happen:

Control plane to start using rootless Podman

How to reproduce it (as minimally and precisely as possible):

  1. Install Fedora 34 Silverblue
  2. Install KinD

Anything else we need to know?:

Environment:

  • kind version: (use kind version):

kind v0.11.0 go1.16.4 linux/amd64

  • Kubernetes version: (use kubectl version):

N/A

  • Docker version: (use docker info):
podman info
host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 12
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.9-200.fc34.x86_64
  linkmode: dynamic
  memFree: 2201493504
  memTotal: 33377628160
  ociRuntime:
    name: crun
    package: crun-1.1-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.1
      commit: 5b341a145c4f515f96f55e3e7760d1c79ec3cf1f
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 5126479872
  swapTotal: 8589930496
  uptime: 294h 53m 38.04s (Approximately 12.25 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/adkaplan/.config/containers/storage.conf
  containerStore:
    number: 26
    paused: 0
    running: 1
    stopped: 25
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.4
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.4
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/adkaplan/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 398
  runRoot: /run/user/1000/containers
  volumePath: /var/home/adkaplan/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Mon Aug 30 16:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1
  • OS (e.g. from /etc/os-release):
NAME=Fedora
VERSION="34.20211006.0 (Silverblue)"
ID=fedora
VERSION_ID=34
VERSION_CODENAME=""
PLATFORM_ID="platform:f34"
PRETTY_NAME="Fedora 34.20211006.0 (Silverblue)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:34"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-silverblue/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=34
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=34
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Silverblue"
VARIANT_ID=silverblue
OSTREE_VERSION='34.20211006.0'
@adambkaplan adambkaplan added the kind/bug Categorizes issue or PR as related to a bug. label Nov 2, 2021
@BenTheElder
Copy link
Member

I see fuse-overlayfs, is this rootless podman? https://kind.sigs.k8s.io/docs/user/rootless/

@BenTheElder
Copy link
Member

can you run with kind create cluster --retain (prevent cleanup on failure) and then capture / share kind export logs? you can then cleanup with kind delete cluster.

@adambkaplan
Copy link
Author

I see fuse-overlayfs, is this rootless podman? https://kind.sigs.k8s.io/docs/user/rootless/

This is rootless podman, and I believe I configured the delegate.conf and iptables.conf files in /etc per the instructions.

capture / share kind export logs

I have posted them at the gist here - https://gist.github.com/adambkaplan/13e99a24c4d1f98475739a672a7ae019
I see several filesystem mount errors - I wonder if fuse-overlayfs is the culprit.

@adambkaplan
Copy link
Author

Also added results of the systemctl hints from the main container process to the gist.

@BenTheElder BenTheElder added area/provider/podman Issues or PRs related to podman area/rootless Issues or PRs related to rootless containers labels Nov 3, 2021
@BenTheElder
Copy link
Member

I think we've had some other recent issues with fuse-overlayfs

@aojea
Copy link
Contributor

aojea commented Nov 3, 2021

I would try kind from master first, this was merged recently
#2278

@mrWinston
Copy link

mrWinston commented Nov 18, 2021

If it helps in any way, i am possibly experiencing the same issue as the OP on fedora 34 ( also with podman and rootless ).
Trying with kind installed from latest main ( at the time of writing commit c8b759f2e1444c0175259943899d768853e17241 ) results in the same error as before.
Here's the log export: https://gist.github.com/mrWinston/d1414bfe3e9c553631d9c4731efc4773

Let me know if i can help with debugging in any way.

@aojea
Copy link
Contributor

aojea commented Nov 18, 2021

Nov 18 08:23:01 kind-control-plane kubelet[3981]: W1118 08:23:01.649183 3981 fs.go:588] stat failed on /dev/mapper/luks-0d827a3f-2fc6-46cc-9f14-1c151674f80f with error: no such file or directory

@mrWinston you are hitting this problem #2411 , that is not really something KIND can fix :/

@mrWinston
Copy link

@aojea oh, yes. you're right, applying the workaround from the linked issue does indeed solve it for me. Thanks for the quick reply!

@mbana
Copy link

mbana commented Feb 12, 2022

This worked for me given this configuration (Fedora 35 with rootless Podman):

$ cat /etc/os-release                                                                
NAME="Fedora Linux"
VERSION="35 (Workstation Edition)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation
$ podman version          
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8
Built:        Wed Dec  8 22:45:07 2021
OS/Arch:      linux/amd64
$ podman info             
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 32
  distribution:
    distribution: fedora
    variant: workstation
    version: "35"
  eventLogger: journald
  hostname: mbana-pc3
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.16.7-200.fc35.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 40270614528
  memTotal: 67404099584
  ociRuntime:
    name: crun
    package: crun-1.4.2-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.2
      commit: f6fbc8f840df1a414f31a60953ae514fa497c748
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 8h 14m 58.74s (Approximately 0.33 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/mbana/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/mbana/.local/share/containers/storage
  graphStatus:
    Build Version: 'Btrfs v5.15.1 '
    Library Version: "102"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /home/mbana/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1638999907
  BuiltTime: Wed Dec  8 22:45:07 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/amd64
  Version: 3.4.4

$ kind --version     
kind version 0.11.1

Then

tee ~/.config/containers/storage.conf <<'EOF'
[storage]
driver = "btrfs"
EOF

systemctl restart --now podman
podman system reset

tee config.yml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraMounts:
  - hostPath: /dev/nvme0n1p1
    containerPath: /dev/nvme0n1p1
EOF

Run:

kubectl cluster-info --context kind-test-1
Kubernetes control plane is running at https://127.0.0.1:41285
CoreDNS is running at https://127.0.0.1:41285/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

@BenTheElder
Copy link
Member

Possibly #2584, certainly at least for this subthread #2521 (comment)

@BenTheElder
Copy link
Member

This has gotten dated, we can revisit this with contemporary fedora/kind versions if there remain issues.

@adambkaplan
Copy link
Author

Oh wow this was old. I can confirm that kind + rootless podman was working just fine on Fedora 37 Silverblue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/provider/podman Issues or PRs related to podman area/rootless Issues or PRs related to rootless containers kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

5 participants