Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podman run returns error "unexpected end of JSON input" #11837

Closed
VeitSanner opened this issue Oct 2, 2021 · 31 comments
Closed

podman run returns error "unexpected end of JSON input" #11837

VeitSanner opened this issue Oct 2, 2021 · 31 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@VeitSanner
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When I try to run a container using podman v3.4.0 on macOS 11.6 x86_64, I get the following issue:

$ podman run hello-world 
Resolved "hello-world" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/hello-world:latest...
Getting image source signatures
Copying blob sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54
Copying blob sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54
Copying config sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
Writing manifest to image destination
Storing signatures
Error: error preparing container 4c55ffed4a30a40b87a20cffc862661c8755c81d766b6cac45e0ee680d64f2fe for attach: error configuring network namespace for container 4c55ffed4a30a40b87a20cffc862661c8755c81d766b6cac45e0ee680d64f2fe: error adding pod busy_darwin_busy_darwin to CNI network "podman": unexpected end of JSON input

Running a container fails for podman remote as well as when I ssh into the machine and execute the command with the same error.

Describe the results you received:

Running a container fails with the error message:

Error: error preparing container 4c55ffed4a30a40b87a20cffc862661c8755c81d766b6cac45e0ee680d64f2fe for attach: error configuring network namespace for container 4c55ffed4a30a40b87a20cffc862661c8755c81d766b6cac45e0ee680d64f2fe: error adding pod busy_darwin_busy_darwin to CNI network "podman": unexpected end of JSON input

Describe the results you expected:
The container starts.

Additional information you deem important (e.g. issue happens only occasionally):

Looking at the symptom this looks similar to #11452 and #11235, but I already checked and I do not have a ~/.docker directory anymore.

$ podman network inspect podman

Result

[
    {
        "cniVersion": "0.4.0",
        "name": "podman",
        "plugins": [
            {
                "bridge": "cni-podman0",
                "hairpinMode": true,
                "ipMasq": true,
                "ipam": {
                    "ranges": [
                        [
                            {
                                "gateway": "10.88.0.1",
                                "subnet": "10.88.0.0/16"
                            }
                        ]
                    ],
                    "routes": [
                        {
                            "dst": "0.0.0.0/0"
                        }
                    ],
                    "type": "host-local"
                },
                "isGateway": true,
                "type": "bridge"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "podman-machine"
            },
            {
                "capabilities": {
                    "portMappings": true
                },
                "type": "portmap"
            },
            {
                "type": "firewall"
            },
            {
                "type": "tuning"
            }
        ]
    }
]

Output of podman version:

Client:
Version:      3.4.0
API Version:  3.4.0
Go Version:   go1.17.1
Built:        Thu Sep 30 20:44:31 2021
OS/Arch:      darwin/amd64

Server:
Version:      3.3.1
API Version:  3.3.1
Go Version:   go1.16.6
Built:        Mon Aug 30 22:46:36 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.16-200.fc34.x86_64
  linkmode: dynamic
  logDriver: ""
  memFree: 3644575744
  memTotal: 4104728576
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 20m 17.49s
plugins:
  log: null
  network: null
  volume: null
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 0
    stopped: 11
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Mon Aug 30 20:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ brew list podman
/usr/local/Cellar/podman/3.4.0/bin/podman
/usr/local/Cellar/podman/3.4.0/bin/podman-remote
/usr/local/Cellar/podman/3.4.0/etc/bash_completion.d/podman
/usr/local/Cellar/podman/3.4.0/libexec/gvproxy
/usr/local/Cellar/podman/3.4.0/share/fish/vendor_completions.d/podman.fish
/usr/local/Cellar/podman/3.4.0/share/man/ (160 files)
/usr/local/Cellar/podman/3.4.0/share/zsh/site-functions/_podman

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 2, 2021
@mgorsk1
Copy link

mgorsk1 commented Oct 2, 2021

+1, experiencing the same issue on fresh install:

➜  ~ podman info
host:
  arch: amd64
  buildahVersion: 1.22.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: localhost
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.13.16-200.fc34.x86_64
  linkmode: dynamic
  logDriver: ""
  memFree: 1335873536
  memTotal: 2061852672
  ociRuntime:
    name: crun
    package: crun-1.0-1.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.0
      commit: 139dc6971e2f1d931af520188763e984d6cdfbf8
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc34.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 0
  swapTotal: 0
  uptime: 5m 32.04s
plugins:
  log: null
  network: null
  volume: null
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 0
    stopped: 6
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.3.1
  Built: 1630356396
  BuiltTime: Mon Aug 30 20:46:36 2021
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.3.1
➜  ~ podman run hello-world
Error: error preparing container 7413f0d7def149c20c2beb5d35b0593db342b7a1b97fd5c3e1e56646f0f52319 for attach: error configuring network namespace for container 7413f0d7def149c20c2beb5d35b0593db342b7a1b97fd5c3e1e56646f0f52319: error adding pod relaxed_curie_relaxed_curie to CNI network "podman": unexpected end of JSON input

in my case the issue is on MacOs Big Sur. Downgrading to 3.3.1 helped.

@afbjorklund
Copy link
Contributor

afbjorklund commented Oct 2, 2021

It's a bug in podman-plugins (on the server), that bundles podman-machine-cni

@afbjorklund

This comment has been minimized.

@Luap99
Copy link
Member

Luap99 commented Oct 2, 2021

Duplicate of #11413
Should be fixed once podman 3.4 lands in CoreOS, as workaround you have to forward at least one port, e.g. -p 8080.

@Luap99 Luap99 closed this as completed Oct 2, 2021
@VeitSanner
Copy link
Author

@Luap99 I tried to run with podman run -P hello-world but no joy.

Error: error preparing container 89fed3a2d595dc28c4c5ce1af577b438a19457bcf0ddd9e1b19f9e658074d7aa for attach: error configuring network namespace for container 89fed3a2d595dc28c4c5ce1af577b438a19457bcf0ddd9e1b19f9e658074d7aa: error adding pod jovial_raman_jovial_raman to CNI network "podman": unexpected end of JSON input

@Luap99
Copy link
Member

Luap99 commented Oct 4, 2021

-P only publishes ports which are defined in the image, the hello-world one has none.
use -p 8080 or some other random port number

@afbjorklund
Copy link
Contributor

afbjorklund commented Oct 4, 2021

I suppose there is no way of getting a fixed podman-plugins rpm out ? 3.3.1-3

  • podman-3:3.4.0-0.1.rc1
    RH Container Bot • 17 days ago

  • update dnsname and machine-cni to latest upstream tags
    Lokesh Mandvekar • 19 days ago

https://src.fedoraproject.org/rpms/podman/c/6972e15200c72420008cb71f4f8d1a483ee2aa9e?branch=rawhide

containers/podman-machine-cni@0749884

  • podman-3:3.3.1-2
    Lokesh Mandvekar • a month ago

  • podman-3:3.3.1-1
    RH Container Bot • a month ago

So that one can patch the CoreOS installation so that it works again... (CNI)

@bretonics
Copy link

@Luap99 any idea when podman 3.4 will land in CoreOS? Current experiencing this issue on a fresh podman install

@afbjorklund
Copy link
Contributor

afbjorklund commented Oct 5, 2021

Fedora CoreOS updates every two weeks (14 days): https://getfedora.org/en/coreos?stream=testing - see #11680

@willcohen
Copy link

willcohen commented Oct 5, 2021

It looks like 3.4 is still working its way into both Fedora 34 and 35, which I assume needs to happen before FCOS picks it up:

https://bodhi.fedoraproject.org/updates/?search=podman

@willcohen
Copy link

podman 3.4 landed in coreos next-devel a few hours ago, so it should be in the upcoming build for the next stream: https://builds.coreos.fedoraproject.org/browser?stream=next-devel&arch=x86_64

@Ruski837
Copy link

Any update on when this fix will be available for download from homebrew?

@afbjorklund
Copy link
Contributor

I don't think brew is involved, podman machine will auto-update CoreOS

@kgibm
Copy link

kgibm commented Oct 12, 2021

Do we need to podman machine rm and then podman machine init to get the update, or do we just podman machine start and it will auto-update (if so, how to check it has been updated)?

@afbjorklund
Copy link
Contributor

afbjorklund commented Oct 12, 2021

It will update every two weeks, you can check in the os-release (with ssh)

podman 3.3.1
Fedora CoreOS 35.20211003.1.0
https://getfedora.org/en/coreos?stream=next

podman 3.4.0
Fedora CoreOS 34.20211004.2.0
https://getfedora.org/en/coreos?stream=testing

This is supposed to be documented somewhere, and link to Fedora CoreOS

And yes, the older podman release picks the newer operating system release...


It seems like it is not possible to select the "next-devel" nor "testing-devel" streams. 😔

But maybe this could be a workaround ? podman machine init --image-path stable

https://getfedora.org/en/coreos?stream=stable

EDIT: Sadly, no. It has the same broken podman-plugins-3:3.3.1-1.fc34.x86_64 RPM

@kgibm
Copy link

kgibm commented Oct 12, 2021

But maybe this could be a workaround ? podman machine init --image-path stable

That still uses a build from 2021-09-19:

% podman machine init --image-path stable --cpus 4 --memory 10112 --disk-size 50
Downloading VM image: fedora-coreos-34.20210919.3.0-qemu.x86_64.qcow2.xz [==============================>---------------------------] 327.0MiB / 604.7MiB

Using --image-path testing seems to be better with a build from 2021-10-04:

% podman machine init --image-path testing --cpus 4 --memory 10112 --disk-size 50
Downloading VM image: fedora-coreos-34.20211004.2.0-qemu.x86_64.qcow2.xz [===>------------------------------------------------------] 39.2MiB / 605.3MiB

@kgibm
Copy link

kgibm commented Oct 12, 2021

However, testing and next both seem to still have podman 3.3.1: https://getfedora.org/en/coreos?stream=testing

coreos

@afbjorklund
Copy link
Contributor

afbjorklund commented Oct 12, 2021

Where "newer" is supposed to be "better" ? (In this case, "older" would have been preferred).

I was playing with using Fedora Cloud for podman instead of Fedora CoreOS, while it is broken.

See https://github.com/afbjorklund/fedora-lima

Same QEMU, Same RPMS, different distribution

@willcohen
Copy link

willcohen commented Oct 12, 2021

A manual image path can be provided:

Taking the latest testing-devel qemu build from: https://builds.coreos.fedoraproject.org/browser?stream=testing-devel&arch=x86_64

podman machine init --image-path https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/34.20211011.20.0/x86_64/fedora-coreos-34.20211011.20.0-qemu.x86_64.qcow2.xz

From there I can run podman machine start, and podman machine ssh:

(base) ➜  ~ podman machine ssh
Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Warning: Permanently added '[localhost]:63659' (ECDSA) to the list of known hosts.
Fedora CoreOS 34.20211011.20.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/c/server/coreos/

[core@localhost ~]$ podman --version
podman version 3.4.0
[core@localhost ~]$ 

@afbjorklund
Copy link
Contributor

afbjorklund commented Oct 12, 2021

That seems like the best workaround so far, should probably be featured on the home page

@willcohen
Copy link

Additionally, I can confirm that running the testing-devel version of coreos with podman 3.4 via the temporary workaround above resolves the "unexpected end of JSON input" errors I've been encountering!

@bavardage
Copy link

just a +1 that I encountered the same issue "error configuring network namespace for container ..." and the workaround fixed it

@jgowdy
Copy link

jgowdy commented Oct 16, 2021

If you're on M1 podman machine init --image-path https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/34.20211015.20.0/aarch64/fedora-coreos-34.20211015.20.0-qemu.aarch64.qcow2.xz

@VeitSanner
Copy link
Author

podman 3.4.0 is now available in the coreOS image. The workarounds mentioned above are no longer required.

@Rotonen
Copy link

Rotonen commented Oct 23, 2021

As Podman defaults to testing[1], podman machine init --image-path stable does not work yet at this point in time.

[1] https://docs.podman.io/en/latest/markdown/podman-machine-init.1.html#image-path

@afbjorklund
Copy link
Contributor

afbjorklund commented Oct 23, 2021

Maybe the "image path" (i.e. CoreOS stream) should show in the machine list ? (especially since it changed from 3.3 to 3.4)

@rhatdan
Copy link
Member

rhatdan commented Oct 25, 2021

Please open an issue to show this.

@afbjorklund
Copy link
Contributor

Please open an issue to show this.

Reported as:

@jorhett
Copy link

jorhett commented Oct 29, 2021

I am 100% in sympathy that this is a tricky problem, but I want to point out that these kinds of issues make it very hard to convince developers to use podman instead of docker. For all docker's faults, they do the exact same thing on Mac (run a VM machine) but they never asked the user to determine which machine image to use.

If a new version of podman requires an updated machine image, then it should detect the problem and if not fix it, provide instructions to the user much better than this error message.

@rhatdan
Copy link
Member

rhatdan commented Oct 29, 2021

You are describing close source versus open source. And you may be right, but I think Podman and Fedora CoreOS will improve and still allow flexibility for users to us other distributions. At one time Solaris worked better the Linux as well...

@afbjorklund
Copy link
Contributor

I think some (including myself) was looking for a "slower" track, but it seems like that this would be covered by using Red Hat Enterprise Linux rather than Fedora ? The machine is handled by CoreOS and not by Podman, so I guess that's where to fix.

i.e. Podman doesn't link to a particular version of the server, but uses the latest tested and available from the other project. Which means that any bug fix is first landed upstream, packaged in Fedora (RPM) and then included next fortnight (CoreOS)

Any user wanting to get bugs fixed faster than this monthly cadence, would probably have to run their own VM instead ?

Possibly there could be access to pre-releases or nightly builds, but eventually someone needs to support it as a "product"


There is talk about providing a Podman Desktop, so better to address those concerns there instead of in this bug report:

One way to get fixes faster is to run Fedora Cloud in the VM instead of Fedora CoreOS, as shown in various guides.

Then you can administer and update the podman server yourself, through the package manager or even by building it ?

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

No branches or pull requests