Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The podman remote API service doesn't report a failing image build #12666

Closed
thommarko opened this issue Dec 21, 2021 · 9 comments
Closed

The podman remote API service doesn't report a failing image build #12666

thommarko opened this issue Dec 21, 2021 · 9 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue

Comments

@thommarko
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

In out setup we cannot use local podman installations, so we use the remote podman api with the fabric8 docker-maven-plugin in our maven builds.

Our Problem is in case of a failing image build the podman remote API just puts out a debug message (in its own log output) but does't fail the maven build.

The maven output just shows the following.
...
cif DOCKER> [nexus.server:5011/cif:1.0.0-SNAPSHOT]: Built image null
...
------------------------------------------------------------------------
BUILD SUCCESS
------------------------------------------------------------------------
Total time: 33.711 s
Finished at: 2021-12-17T10:52:46+01:00
------------------------------------------------------------------------

"Build image null" instead of a correct "Built image sha256:xxxxx" is the only hint that something went wrong.

Because we do not want to increase the image version for every build during development its getting even worse, because there may be an "old" version (of course with the same name and tag) of the image in the remote podman cache. And this leads to a "Built image sha256:xxxxx" output and a push of the "old" image even if the latest image build failed.

The only chance the verify a build is greping the podman api service log for possibly occouring errors.

Steps to reproduce the issue:

  1. Setup a remote podman API - "podman system service tcp:hostname:2375 --log-level=debug --time=3600"

  2. Setup a maven project with the fabic8 Plugin
    Dockerfile:

FROM ubuntu:latest
COPY not_existing_resource to_nowhere

pom.xml

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>demo</groupId>
  <artifactId>docker-demo</artifactId>
  <version>0.1.0</version>
  <packaging>pom</packaging>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  </properties>

  <build>
    <plugins>
      <plugin>
        <groupId>io.fabric8</groupId>
        <artifactId>docker-maven-plugin</artifactId>
        <version>0.38.0</version>
        <configuration>
          <images>
            <image>
              <alias>data</alias>
              <name>demo/${project.artifactId}:latest</name>
            </image>
          </images>
        </configuration>
        <executions>
          <execution>
            <id>build-image</id>
            <phase>package</phase>
            <goals>
              <goal>build</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

</project>

Place the two files in the same folder and call "mvn clean install -Ddocker.host=http://hostname:2375"

  1. Produce an error by adding something like "COPY not_existing_resource to_nowhere" into the Dockerfile
    Should result into a debug level error in podmans service output ...
time="2021-12-21T08:49:31+01:00" level=debug msg="error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin container=oci] Command:copy Args:[not_existing_resource to_nowhere] Flags:[] Attrs:map[] Message:COPY not_existing_resource to_nowhere Original:COPY not_existing_resource to_nowhere}: checking on sources under \"/var/tmp/libpod_builder845903974/build\": copier: stat: \"/not_existing_resource\": no such file or directory"

... and an BUILD SUCCESS on the maven side.

Describe the results you received:

Maven/fabric8 do not notice a failing image build.

Describe the results you expected:

A failing image builds should lead to a failing maven build.

Additional information you deem important (e.g. issue happens only occasionally):

We use a rootless podman setup. Even for the reomte API.

Output of podman version:

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.15.7
Built:        Thu Jul 29 17:02:43 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.26-3.module+el8.4.0+11311+9da8acfb.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.26, commit: a35bb9ea67d5a83c7da53202f2fcd505c036d29c'
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "8.3"
  eventLogger: file
  hostname: cdfead01.coci.de.comdirect.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 302
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65535
    uidmap:
    - container_id: 0
      host_id: 8236
      size: 1
    - container_id: 1
      host_id: 200000
      size: 65535
  kernel: 4.18.0-240.22.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 752984064
  memTotal: 3919654912
  ociRuntime:
    name: runc
    package: runc-1.0.0-73.rc93.module+el8.4.0+11311+9da8acfb.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.13
      libseccomp: 2.4.3
  os: linux
  remoteSocket:
    path: /run/user/username/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.4.0+11822+6cc1e7d7.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 1713106944
  swapTotal: 2147479552
  uptime: 1190h 35m 40.65s (Approximately 49.58 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/username/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.4.0-3.module+el8.4.0+11311+9da8acfb.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.4
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/username/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/username/containers
  volumePath: /home/username/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1627570963
  BuiltTime: Thu Jul 29 17:02:43 2021
  GitCommit: ""
  GoVersion: go1.15.7
  OsArch: linux/amd64
  Version: 3.2.3

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.3-0.11.module+el8.4.0+12050+ef972f71.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

OpenStack VMs based on RHEL8
est version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)**

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

OpenStack VMs based on RHEL8

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Dec 21, 2021
@flouthoc
Copy link
Collaborator

I'll check this thanks.

@flouthoc
Copy link
Collaborator

@thommarko This is fixed in upstream here: #12405

Could you please try with latest main as your podman server.

@mheon
Copy link
Member

mheon commented Dec 22, 2021

Closing on the assumption that this was #12405 - reopen if it turns out to be a different issue

@mheon mheon closed this as completed Dec 22, 2021
@thommarko
Copy link
Author

/reopen
At first, happy new year to everybody ;-)

@flouthoc i'm sorry but #12405 did not fix the issue.
I have testet with latest v4.0.0-dev and v3.4.3 and still got the same behavior.

Maven output

╭─    ~/install/tests/data-jolokia-demo-master ················································································ ✔  16:07:53  ─╮
╰─ mvn clean install -Ddocker.host=http://ubuntu:2375                                                                                                 ─╯
[INFO] Scanning for projects...
[INFO] 
[INFO] --------------------------< demo:docker-demo >--------------------------
[INFO] Building docker-demo 0.1.0
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ docker-demo ---
[INFO] Deleting /home/thma/install/tests/data-jolokia-demo-master/target
[INFO] 
[INFO] --- docker-maven-plugin:0.38.0:build (build-image) @ docker-demo ---
[INFO] Building tar: /home/thma/install/tests/data-jolokia-demo-master/target/docker/demo/docker-demo/latest/tmp/docker-build.tar
[INFO] DOCKER> [demo/docker-demo:latest] "data": Created docker-build.tar in 18 milliseconds
[INFO] DOCKER> [demo/docker-demo:latest] "data": Built image null
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ docker-demo ---
[INFO] Installing /home/thma/install/tests/data-jolokia-demo-master/pom.xml to /home/thma/.m2/repository/demo/docker-demo/0.1.0/docker-demo-0.1.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Podman version and service output:

╭─    ~ ··································································································· ✔  16:08:32  ─╮
╰─ podman --version                                                                                                               ─╯
podman version 3.4.4

╭─    ~ ··································································································· ✔  16:10:19  ─╮
╰─ podman system service tcp:ubuntu:2375 --log-level=debug --time=360                                                             ─╯
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called service.PersistentPreRunE(podman system service tcp:ubuntu:2375 --log-level=debug --time=360) 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/thma/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/thma/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/thma/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/thma/.local/share/containers/storage/volumes 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: opaque flag erroneously copied up, consider update to kernel 4.8 or later to fix 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/thma/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 49             
INFO[0000] using API endpoint: 'tcp:ubuntu:2375'        
DEBU[0000] registered SIGHUP watcher for config         
DEBU[0000] CORS Headers were not set                    
INFO[0000] API service listening on "127.0.1.1:2375"    
DEBU[0000] waiting for SIGHUP to reload configuration   
DEBU[0012] IdleTracker:new 0m+0h/0t connection(s)        X-Reference-Id=0xc000128018
DEBU[0012] IdleTracker:active 0m+0h/1t connection(s)     X-Reference-Id=0xc000128018
DEBU[0013] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0013] Loading registries configuration "/etc/containers/registries.conf.d/shortnames.conf" 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "GET /version HTTP/1.1" 200 892 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/1t connection(s)       X-Reference-Id=0xc000128018
DEBU[0013] IdleTracker:closed 1m+0h/1t connection(s)     X-Reference-Id=0xc000128018
DEBU[0013] IdleTracker:new 0m+0h/1t connection(s)        X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:active 0m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0013] Looking up image "ubuntu:latest" in local containers storage 
DEBU[0013] Trying "ubuntu:latest" ...                   
DEBU[0013] Trying "docker.io/library/ubuntu:latest" ... 
DEBU[0013] parsed reference into "[overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] Found image "ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage 
DEBU[0013] Found image "ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage ([overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1) 
DEBU[0013] Inspecting image ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "GET /v1.40/images/ubuntu%3Alatest/json HTTP/1.1" 200 1664 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/2t connection(s)       X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:active 1m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0013] Looking up image "demo/docker-demo:latest" in local containers storage 
DEBU[0013] Trying "demo/docker-demo:latest" ...         
DEBU[0013] Trying "localhost/demo/docker-demo:latest" ... 
DEBU[0013] Trying "docker.io/demo/docker-demo:latest" ... 
INFO[0013] Request Failed(Not Found): failed to find image demo/docker-demo:latest: demo/docker-demo:latest: No such image 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "GET /v1.40/images/demo%2Fdocker-demo%3Alatest/json HTTP/1.1" 404 209 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/2t connection(s)       X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:active 1m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0013] base for stage 0: "ubuntu:latest"            
DEBU[0013] FROM "ubuntu:latest"                         
DEBU[0013] Pulling image ubuntu:latest (policy: missing) 
DEBU[0013] Looking up image "ubuntu:latest" in local containers storage 
DEBU[0013] Trying "ubuntu:latest" ...                   
DEBU[0013] Trying "docker.io/library/ubuntu:latest" ... 
DEBU[0013] parsed reference into "[overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] Found image "ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage 
DEBU[0013] Found image "ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage ([overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1) 
DEBU[0013] Looking up image "docker.io/library/ubuntu:latest" in local containers storage 
DEBU[0013] Trying "docker.io/library/ubuntu:latest" ... 
DEBU[0013] parsed reference into "[overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] Found image "docker.io/library/ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage 
DEBU[0013] Found image "docker.io/library/ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage ([overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1) 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] [graphdriver] trying provided driver "overlay" 
DEBU[0013] Cached value indicated that overlay is supported 
DEBU[0013] Cached value indicated that metacopy is not being used 
DEBU[0013] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0013] Cached value indicated that volatile is being used 
DEBU[0013] overlay: mount_data=,lowerdir=/home/thma/.local/share/containers/storage/overlay/l/DSYMWVJYQMB4PD3GUPZHEGHCP3,upperdir=/home/thma/.local/share/containers/storage/overlay/2d43fad859d595c75a0a2bcce6c3c5c41dc836548cfbf75e3928183404e870b7/diff,workdir=/home/thma/.local/share/containers/storage/overlay/2d43fad859d595c75a0a2bcce6c3c5c41dc836548cfbf75e3928183404e870b7/work,userxattr,volatile 
DEBU[0013] Container ID: e6e36dab61657b9b1439043769e3a38ac0c5c77fedb8881e4445c040f2fa8e05 
DEBU[0013] Parsed Step: {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:copy Args:[not_existing_resource to_nowhere] Flags:[] Attrs:map[] Message:COPY not_existing_resource to_nowhere Original:COPY not_existing_resource to_nowhere} 
DEBU[0013] parsed reference into "[overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] COPY []string(nil), imagebuilder.Copy{FromFS:false, From:"", Src:[]string{"not_existing_resource"}, Dest:"/to_nowhere", Download:false, Chown:"", Chmod:""} 
DEBU[0013] error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:copy Args:[not_existing_resource to_nowhere] Flags:[] Attrs:map[] Message:COPY not_existing_resource to_nowhere Original:COPY not_existing_resource to_nowhere}: checking on sources under "/var/tmp/libpod_builder2750765900/build": copier: stat: "/not_existing_resource": no such file or directory 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "POST /v1.40/build?dockerfile=Dockerfile&forcerm=1&nocache=0&squash=0&t=demo%2Fdocker-demo%3Alatest HTTP/1.1" 200 561 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/2t connection(s)       X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:active 1m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0013] Looking up image "demo/docker-demo:latest" in local containers storage 
DEBU[0013] Trying "demo/docker-demo:latest" ...         
DEBU[0013] Trying "localhost/demo/docker-demo:latest" ... 
DEBU[0013] Trying "docker.io/demo/docker-demo:latest" ... 
INFO[0013] Request Failed(Not Found): failed to find image demo/docker-demo:latest: demo/docker-demo:latest: No such image 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "GET /v1.40/images/demo%2Fdocker-demo%3Alatest/json HTTP/1.1" 404 209 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/2t connection(s)       X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:closed 1m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0373] API service shutting down, idle for 6m0s     
DEBU[0373] API service shutdown by /usr/lib/go-1.17/src/sync/once.go:68, 0/2 connection(s) 
DEBU[0373] Called service.PersistentPostRunE(podman system service tcp:ubuntu:2375 --log-level=debug --time=360) 

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 3, 2022

@thommarko: Reopened this issue.

In response to this:

/reopen
At first, happy new year to everybody ;-)

@flouthoc i'm sorry but #12405 did not fix the issue.
I have testet with latest v4.0.0-dev and v3.4.3 and still got the same behavior.

Maven output

╭─    ~/install/tests/data-jolokia-demo-master ················································································ ✔  16:07:53  ─╮
╰─ mvn clean install -Ddocker.host=http://ubuntu:2375                                                                                                 ─╯
[INFO] Scanning for projects...
[INFO] 
[INFO] --------------------------< demo:docker-demo >--------------------------
[INFO] Building docker-demo 0.1.0
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ docker-demo ---
[INFO] Deleting /home/thma/install/tests/data-jolokia-demo-master/target
[INFO] 
[INFO] --- docker-maven-plugin:0.38.0:build (build-image) @ docker-demo ---
[INFO] Building tar: /home/thma/install/tests/data-jolokia-demo-master/target/docker/demo/docker-demo/latest/tmp/docker-build.tar
[INFO] DOCKER> [demo/docker-demo:latest] "data": Created docker-build.tar in 18 milliseconds
[INFO] DOCKER> [demo/docker-demo:latest] "data": Built image null
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ docker-demo ---
[INFO] Installing /home/thma/install/tests/data-jolokia-demo-master/pom.xml to /home/thma/.m2/repository/demo/docker-demo/0.1.0/docker-demo-0.1.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Podman version and service output:

╭─    ~ ··································································································· ✔  16:08:32  ─╮
╰─ podman --version                                                                                                               ─╯
podman version 3.4.4

╭─    ~ ··································································································· ✔  16:10:19  ─╮
╰─ podman system service tcp:ubuntu:2375 --log-level=debug --time=360                                                             ─╯
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called service.PersistentPreRunE(podman system service tcp:ubuntu:2375 --log-level=debug --time=360) 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/thma/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/thma/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/thma/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/thma/.local/share/containers/storage/volumes 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: opaque flag erroneously copied up, consider update to kernel 4.8 or later to fix 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /home/thma/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 49             
INFO[0000] using API endpoint: 'tcp:ubuntu:2375'        
DEBU[0000] registered SIGHUP watcher for config         
DEBU[0000] CORS Headers were not set                    
INFO[0000] API service listening on "127.0.1.1:2375"    
DEBU[0000] waiting for SIGHUP to reload configuration   
DEBU[0012] IdleTracker:new 0m+0h/0t connection(s)        X-Reference-Id=0xc000128018
DEBU[0012] IdleTracker:active 0m+0h/1t connection(s)     X-Reference-Id=0xc000128018
DEBU[0013] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0013] Loading registries configuration "/etc/containers/registries.conf.d/shortnames.conf" 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "GET /version HTTP/1.1" 200 892 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/1t connection(s)       X-Reference-Id=0xc000128018
DEBU[0013] IdleTracker:closed 1m+0h/1t connection(s)     X-Reference-Id=0xc000128018
DEBU[0013] IdleTracker:new 0m+0h/1t connection(s)        X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:active 0m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0013] Looking up image "ubuntu:latest" in local containers storage 
DEBU[0013] Trying "ubuntu:latest" ...                   
DEBU[0013] Trying "docker.io/library/ubuntu:latest" ... 
DEBU[0013] parsed reference into "[overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] Found image "ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage 
DEBU[0013] Found image "ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage ([overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1) 
DEBU[0013] Inspecting image ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "GET /v1.40/images/ubuntu%3Alatest/json HTTP/1.1" 200 1664 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/2t connection(s)       X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:active 1m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0013] Looking up image "demo/docker-demo:latest" in local containers storage 
DEBU[0013] Trying "demo/docker-demo:latest" ...         
DEBU[0013] Trying "localhost/demo/docker-demo:latest" ... 
DEBU[0013] Trying "docker.io/demo/docker-demo:latest" ... 
INFO[0013] Request Failed(Not Found): failed to find image demo/docker-demo:latest: demo/docker-demo:latest: No such image 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "GET /v1.40/images/demo%2Fdocker-demo%3Alatest/json HTTP/1.1" 404 209 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/2t connection(s)       X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:active 1m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0013] base for stage 0: "ubuntu:latest"            
DEBU[0013] FROM "ubuntu:latest"                         
DEBU[0013] Pulling image ubuntu:latest (policy: missing) 
DEBU[0013] Looking up image "ubuntu:latest" in local containers storage 
DEBU[0013] Trying "ubuntu:latest" ...                   
DEBU[0013] Trying "docker.io/library/ubuntu:latest" ... 
DEBU[0013] parsed reference into "[overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] Found image "ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage 
DEBU[0013] Found image "ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage ([overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1) 
DEBU[0013] Looking up image "docker.io/library/ubuntu:latest" in local containers storage 
DEBU[0013] Trying "docker.io/library/ubuntu:latest" ... 
DEBU[0013] parsed reference into "[overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] Found image "docker.io/library/ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage 
DEBU[0013] Found image "docker.io/library/ubuntu:latest" as "docker.io/library/ubuntu:latest" in local containers storage ([overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1) 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] [graphdriver] trying provided driver "overlay" 
DEBU[0013] Cached value indicated that overlay is supported 
DEBU[0013] Cached value indicated that metacopy is not being used 
DEBU[0013] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0013] Cached value indicated that volatile is being used 
DEBU[0013] overlay: mount_data=,lowerdir=/home/thma/.local/share/containers/storage/overlay/l/DSYMWVJYQMB4PD3GUPZHEGHCP3,upperdir=/home/thma/.local/share/containers/storage/overlay/2d43fad859d595c75a0a2bcce6c3c5c41dc836548cfbf75e3928183404e870b7/diff,workdir=/home/thma/.local/share/containers/storage/overlay/2d43fad859d595c75a0a2bcce6c3c5c41dc836548cfbf75e3928183404e870b7/work,userxattr,volatile 
DEBU[0013] Container ID: e6e36dab61657b9b1439043769e3a38ac0c5c77fedb8881e4445c040f2fa8e05 
DEBU[0013] Parsed Step: {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:copy Args:[not_existing_resource to_nowhere] Flags:[] Attrs:map[] Message:COPY not_existing_resource to_nowhere Original:COPY not_existing_resource to_nowhere} 
DEBU[0013] parsed reference into "[overlay@/home/thma/.local/share/containers/storage+/run/user/1000/containers]@ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] exporting opaque data as blob "sha256:ba6acccedd2923aee4c2acc6a23780b14ed4b8a5fa4e14e252a23b846df9b6c1" 
DEBU[0013] COPY []string(nil), imagebuilder.Copy{FromFS:false, From:"", Src:[]string{"not_existing_resource"}, Dest:"/to_nowhere", Download:false, Chown:"", Chmod:""} 
DEBU[0013] error building at step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:copy Args:[not_existing_resource to_nowhere] Flags:[] Attrs:map[] Message:COPY not_existing_resource to_nowhere Original:COPY not_existing_resource to_nowhere}: checking on sources under "/var/tmp/libpod_builder2750765900/build": copier: stat: "/not_existing_resource": no such file or directory 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "POST /v1.40/build?dockerfile=Dockerfile&forcerm=1&nocache=0&squash=0&t=demo%2Fdocker-demo%3Alatest HTTP/1.1" 200 561 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/2t connection(s)       X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:active 1m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0013] Looking up image "demo/docker-demo:latest" in local containers storage 
DEBU[0013] Trying "demo/docker-demo:latest" ...         
DEBU[0013] Trying "localhost/demo/docker-demo:latest" ... 
DEBU[0013] Trying "docker.io/demo/docker-demo:latest" ... 
INFO[0013] Request Failed(Not Found): failed to find image demo/docker-demo:latest: demo/docker-demo:latest: No such image 
127.0.0.1 - - [03/Jan/2022:16:10:41 +0100] "GET /v1.40/images/demo%2Fdocker-demo%3Alatest/json HTTP/1.1" 404 209 "" "Apache-HttpClient/4.5.13 (Java/11.0.13)"
DEBU[0013] IdleTracker:idle 1m+0h/2t connection(s)       X-Reference-Id=0xc000128220
DEBU[0013] IdleTracker:closed 1m+0h/2t connection(s)     X-Reference-Id=0xc000128220
DEBU[0373] API service shutting down, idle for 6m0s     
DEBU[0373] API service shutdown by /usr/lib/go-1.17/src/sync/once.go:68, 0/2 connection(s) 
DEBU[0373] Called service.PersistentPostRunE(podman system service tcp:ubuntu:2375 --log-level=debug --time=360) 

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot reopened this Jan 3, 2022
@github-actions
Copy link

github-actions bot commented Feb 3, 2022

A friendly reminder that this issue had no activity for 30 days.

@flouthoc
Copy link
Collaborator

flouthoc commented Feb 3, 2022

@thommarko I think your podman server is still running older version. Could you make sure both your client and server are on v4.0.0* from github.

@github-actions
Copy link

github-actions bot commented Mar 6, 2022

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Mar 7, 2022

Since we got no response for information, I think the problem is fixed. Closing, Reopen if I am mistaken.

@rhatdan rhatdan closed this as completed Mar 7, 2022
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue
Projects
None yet
Development

No branches or pull requests

4 participants