Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman: panic: runtime error: invalid memory address or nil pointer dereference #20250

Open
petrkr opened this issue Jan 9, 2023 · 21 comments
Open

podman: panic: runtime error: invalid memory address or nil pointer dereference #20250

petrkr opened this issue Jan 9, 2023 · 21 comments

Comments

@petrkr
Copy link

petrkr commented Jan 9, 2023

Maintainer: Oskari Rauta
Environment: ARMv7 - Turris Omnia - OpenWRT 21.02 (Turris OS 6.2.1)
Description:

Seems same problem like this one containers/podman#10535

Just wonder if update to latest 3.4.x podman version (https://github.com/containers/podman/releases?q=v3.4&expanded=true) would fix that. It is hard for me to set-up whole build environment to rebuild just one package to test it

Package makefile: https://git.openwrt.org/?p=feed/packages.git;a=blob;f=utils/podman/Makefile;h=e8323d4022e0b332a00a0ef4c7ba17e3fae0debf;hb=refs/heads/openwrt-21.02

i've tried overlay and btrfs (as there is Btrfs filesystem) storage engine, both are same.

root@turris:~# podman run -it --rm arm32v7/busybox
✔ docker.io/arm32v7/busybox:latest
Trying to pull docker.io/arm32v7/busybox:latest...
Getting image source signatures
Copying blob 46758452d3ee done  
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x200 pc=0x58e0a0]

goroutine 87 [running]:
reflect.mapiterinit(0x1bce3c0, 0x3f2a4c0, 0x416a4b0)
        runtime/map.go:1373 +0x2c
github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
        github.com/modern-go/[email protected]/unsafe_map.go:112
github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0x3e88738, 0x3eec388)
        github.com/json-iterator/[email protected]/reflect_map.go:333 +0x34
github.com/json-iterator/go.(*structFieldEncoder).IsEmpty(0x3e88f90, 0x3eec300)
        github.com/json-iterator/[email protected]/reflect_struct_encoder.go:118 +0x50
github.com/json-iterator/go.(*structEncoder).Encode(0x3e89008, 0x3eec300, 0x416a4b0)
        github.com/json-iterator/[email protected]/reflect_struct_encoder.go:148 +0x63c
github.com/json-iterator/go.(*OptionalEncoder).Encode(0x3ee8838, 0x3ee8158, 0x416a4b0)
        github.com/json-iterator/[email protected]/reflect_optional.go:70 +0x104
github.com/json-iterator/go.(*sliceEncoder).Encode(0x4061fc0, 0x4028728, 0x416a4b0)
        github.com/json-iterator/[email protected]/reflect_slice.go:38 +0x37c
github.com/json-iterator/go.(*OptionalEncoder).Encode(0x3ee8860, 0x3ee88a0, 0x416a4b0)
        github.com/json-iterator/[email protected]/reflect_optional.go:70 +0x104
github.com/json-iterator/go.(*onePtrEncoder).Encode(0x3ee8868, 0x4028728, 0x416a4b0)
        github.com/json-iterator/[email protected]/reflect.go:219 +0x8c
github.com/json-iterator/go.(*Stream).WriteVal(0x416a4b0, {0x1b4ac50, 0x4028728})
        github.com/json-iterator/[email protected]/reflect.go:98 +0x1d4
github.com/json-iterator/go.(*frozenConfig).Marshal(0x3f9c540, {0x1b4ac50, 0x4028728})
        github.com/json-iterator/[email protected]/config.go:299 +0xb4
github.com/containers/storage.(*layerStore).saveLayers(0x4028700)
        github.com/containers/[email protected]/layers.go:484 +0x150
github.com/containers/storage.(*layerStore).Save(0x4028700)
        github.com/containers/[email protected]/layers.go:467 +0xd8
github.com/containers/storage.(*layerStore).Put(0x4028700, {0x413a327, 0x40}, 0x0, {0x0, 0x0, 0x0}, {0x0, 0x0}, 0x0, ...)
        github.com/containers/[email protected]/layers.go:799 +0x1834
github.com/containers/storage.(*store).PutLayer(0x3cc80c0, {0x413a327, 0x40}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0}, ...)
        github.com/containers/[email protected]/store.go:1057 +0xc38
github.com/containers/image/v5/storage.(*storageImageDestination).commitLayer(0x409e6e0, {0x1d5f65c, 0x3ec31a0}, {{{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, ...}, ...}, ...)
        github.com/containers/image/[email protected]/storage/storage_image.go:1010 +0xeb0
github.com/containers/image/v5/storage.(*storageImageDestination).queueOrCommit(0x409e6e0, {0x1d5f65c, 0x3ec31a0}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, ...}, ...)
        github.com/containers/image/[email protected]/storage/storage_image.go:837 +0x1dc
github.com/containers/image/v5/storage.(*storageImageDestination).PutBlobWithOptions(0x409e6e0, {0x1d5f65c, 0x3ec31a0}, {0x1d55dd4, 0x3eddd60}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, ...}, ...}, ...)
        github.com/containers/image/[email protected]/storage/storage_image.go:466 +0x11c
github.com/containers/image/v5/copy.(*copier).copyBlobFromStream(0x3eeea00, {0x1d5f65c, 0x3ec31a0}, {0x1d5779c, 0x3df4d20}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, ...}, ...}, ...)
        github.com/containers/image/[email protected]/copy/copy.go:1632 +0x1b34
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream(0x3ec2180, {0x1d5f65c, 0x3ec31a0}, {0x1d5779c, 0x3df4d20}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, ...}, ...}, ...)
        github.com/containers/image/[email protected]/copy/copy.go:1362 +0x348
github.com/containers/image/v5/copy.(*imageCopier).copyLayer.func3(0x3ec2180, 0x416ab40, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, {0x40e0640, ...}, ...}, ...)
        github.com/containers/image/[email protected]/copy/copy.go:1305 +0x19c
github.com/containers/image/v5/copy.(*imageCopier).copyLayer(0x3ec2180, {0x1d5f65c, 0x3ec31a0}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, ...}, ...)
        github.com/containers/image/[email protected]/copy/copy.go:1329 +0x7b0
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1(0x0, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, {0x40e0640, 0x31}, ...}, ...)
        github.com/containers/image/[email protected]/copy/copy.go:891 +0x338
created by github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2
        github.com/containers/image/[email protected]/copy/copy.go:928 +0x338
root@turris:~#
@petrkr
Copy link
Author

petrkr commented Jan 9, 2023

Including podman version/info

root@turris:~# podman version
Version:      3.4.1
API Version:  3.4.1
Go Version:   go1.18.9
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/arm

root@turris:~# podman info
host:
  arch: arm
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 2
  distribution:
    distribution: '"turrisos"'
    version: 6.2.1
  eventLogger: none
  hostname: turris
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.86
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 190943232
  memTotal: 2114068480
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/sbin/runc
    version: |-
      runc version 1.1.2
      spec: 1.0.2-dev
      go: go1.18.9
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: false
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 86h 2m 54.11s (Approximately 3.58 days)
plugins:
  log:
  - k8s-file
  - none
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /srv/containers/storage
  graphStatus:
    Build Version: 'Btrfs v5.11 '
    Library Version: "102"
  imageStore:
    number: 0
  runRoot: /tmp/run/containers/storage
  volumePath: /srv/containers/storage/volumes
version:
  APIVersion: 3.4.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.9
  OsArch: linux/arm
  Version: 3.4.1

@dangowrt
Copy link
Member

@oskarirauta

@oskarirauta
Copy link
Contributor

oskarirauta commented Jan 29, 2023 •
edited
Loading

©petrkr

Not sure, but didn't 3.4.1 fix this issue? Or some of the newer ones? My webserver (yes, my site's server runs on openwrt) - has been running with podman 3.4.4 ever since I updated to that version. There's weekly reboots on server, but it has nothing to do with podman.... Should update the whole system.

Btw. why is mine using cgroupv1 when you clearly have cgroupv2..?

Otherwise your setup seems just like mine that runs without issues except that is a bit older, and:

  • cgroup version
  • I have slirp ( but as we run these with root it doesn't have anything.. )
  • my storage driver is extfs instead of btrfs, but that's propably more related on your choice of filesystem.

One thing though pops... You are using runc as oci runtime. Would you mind trying first with crun, as it's recommended for podman over runc.

I have never updated anything but master versions, current is running with 4.3.0 and I should be today, updating the build to 4.3.1. My main router is running with 4.3.0 as far as I know- without bigger problem, with basic alpine:latest and tmux keeps it up..

EDIT:
Sorry, I seem to be instead running with 4.3.1 and I already have updated in master to 4.3.1 - I just got a notification about 4.4.0-rc3 so no update coming even to master at the moment, as I try to avoid updates to RC versions, unless their changelog reveals that they fix real issues, and atleast for rc2 there wasn't changelog listed at all.. Sorry, for misleading information, I just woke up.. But really, try switching to crun first.

@oskarirauta
Copy link
Contributor

But try switch to crun first.

@petrkr
Copy link
Author

petrkr commented Jan 29, 2023

I did not mentioned it, but I've triee both, runc and crun, same results

@oskarirauta
Copy link
Contributor

Ok, so you need a updated version then. I can verify that 3.4.4 works fine.

@petrkr
Copy link
Author

petrkr commented Jan 29, 2023

Ok, so you need a updated version then. I can verify that 3.4.4 works fine.

Can not update as package is not compiled. That is what I wrote on original issue.. if someone have working build environment, maybe can put and build 3.4.4 and I can test. But in 21.02 repo is still 3.4.1

@oskarirauta
Copy link
Contributor

oskarirauta commented Jan 29, 2023 •
edited
Loading

@petrkr

I have, but wrong arch- mine is x86-64.. There's though even newer version available, though I am not sure if other system depencies are filled-

Snapshots: podman_4.3.1-1

Double check the download path, there were alternatives for arm cortex a9 and I know nothing about turris.
Other variants were neon and vfpv3-d16, most likely though, linked file is for your arch...

@petrkr
Copy link
Author

petrkr commented Jan 30, 2023 

# opkg install /root/podman_4.3.1-1_arm_cortex-a9.ipk 
Package podman (3.4.1-1) installed in root is up to date.
Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency libncursesw for libreadline
 * pkg_hash_fetch_best_installation_candidate: Packages for libreadline found, but incompatible with the architectures configured

yes, there will be dependency hell as this package will be probably from 22/master branch.

@oskarirauta
Copy link
Contributor

That is possible. I pushed it few weeks ago to master.

@paper42
Copy link
Contributor

paper42 commented Jan 30, 2023

Updating to 3.4.7 helps, but I am still having issues:

root@turris:~# podman run -it --rm arm32v7/busybox
✔ docker.io/arm32v7/busybox:latest
Trying to pull docker.io/arm32v7/busybox:latest...
Getting image source signatures
Copying blob 46758452d3ee done  
Copying config 1d57ab16f6 done  
Writing manifest to image destination
Storing signatures
Error: OCI runtime error: prctl: Invalid argument

and this on any subsequent runs:

Error: container create failed (no logs from conmon): EOF

I didn't manage to build 4.3.1 on OpenWrt 21.02 even though gpg-error.pc is in build_dir/target-arm_cortex-a9+vfpv3-d16_musl_eabi/libgpg-error-1.45/ipkg-install/usr/lib/pkgconfig/gpg-error.pc:

# /build/build/staging_dir/host/bin/pkg-config --cflags  -- gpgme
Package gpg-error was not found in the pkg-config search path.
Perhaps you should add the directory containing `gpg-error.pc'
to the PKG_CONFIG_PATH environment variable
Package 'gpg-error', required by 'gpgme', not found
Package 'gpg-error', required by 'libassuan', not found

@petrkr
Copy link
Author

petrkr commented Jan 31, 2023

Updating to 3.4.7 helps, but I am still having issues:

Can you please share that ipkg/opkg ? I will try to check this other stuff... But I hope it's not kernel config related

@paper42
Copy link
Contributor

paper42 commented Jan 31, 2023

Updating to 3.4.7 helps, but I am still having issues:

Can you please share that ipkg/opkg ? I will try to check this other stuff... But I hope it's not kernel config related

podman_3.4.7-1_arm_cortex-a9_vfpv3-d16.zip GitHub didn't allow me to upload an .ipk file, so just change the extension from zip to ipk.

@petrkr
Copy link
Author

petrkr commented Jan 31, 2023 •
edited
Loading

Well. this is at-least interesting error message in syslog

Jan 31 14:42:03 turris modprobe: no module folders for kernel version 5.15.88 found

but what kind of kernel module or folder it looking for ?

EDIT:
also syslog

Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/tmp/conmon-term.4GQEZ1} 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/proc/self/fd/12/attach} 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: terminal_ctrl_fd: 12 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: winsz read side: 15, winsz write side: 15 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: about to accept from console_socket_fd: 9 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: about to recvfd from connfd: 11 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: console = {.name = '(null)'; .fd = 0} 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <nwarn>: Failed to get console terminal settings

@petrkr
Copy link
Author

petrkr commented Jan 31, 2023

well, kernel module was just because OpenWRT updated to new kernel without reboot. So ignore that message.

@petrkr
Copy link
Author

petrkr commented Jan 31, 2023

Just wonder if new version of "conmon" can fix it

[v2.0.32](https://github.com/containers/conmon/releases/tag/v2.0.32)
Bug Fixes
Avoid mainfd_std{in,out} sharing the same file descriptor.

in OpenWRT 21.02 is 2.0.30

@oskarirauta
Copy link
Contributor

oskarirauta commented Jan 31, 2023 •
edited
Loading

Latest conmon is 2.1.5 - available at snapshots.

@petrkr
Copy link
Author

petrkr commented Jan 31, 2023

Latest conmon is 2.1.5 - available at snapshots.

Again, seems I can not use snapshot version in 21.02 OpenWRT

root@turris:~# conmon --version
Error relocating /usr/bin/conmon: __clock_gettime64: symbol not found
Error relocating /usr/bin/conmon: __localtime64_r: symbol not found
Error relocating /usr/bin/conmon: __dlsym_time64: symbol not found

@paper42
Copy link
Contributor

paper42 commented Feb 1, 2023

conmon 2.1.5 and libseccomp 2.5.2 which was required for the build for OpenWrt 21.02:
conmon_2.1.5-1_arm_cortex-a9_vfpv3-d16.zip
libseccomp_2.5.2-1_arm_cortex-a9_vfpv3-d16.zip

I am still getting the Error: OCI runtime error: prctl: Invalid argument error

@petrkr
Copy link
Author

petrkr commented Feb 1, 2023

There could be missing something in kernel. I check that debug message and it can not get terminal info, but which terminal? Container's or system (turris one).

I also tried all three runc, crun, uxc... actually uxc ends with "I/O Error" but since they missing lot of debug prints, I really do not know which part of 2000 lines C file ends with return -1

@oskarirauta
Copy link
Contributor

remember to put uxc to different group, if I remember correctly, it doesn't support json like those 2 others- but nevertheless, uxc won't work anyway, I tested that sometime ago as well..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@oskarirauta @petrkr @dangowrt @paper42