Openshift can't mount push-secret causing deploy to time out

[jedla97]

Describe the bug

When deploying to Openshift using local openshift image registry pod can't mount <app-name>-push-secret and timed out. Internal repository should not require the secret or at least if we don't specify one this secret should not be even created.

This bug was introduced by #34241.

Some of the test in quarkus-test-framwork and quarkus-test-suite is failing because of this. If the secret is manually created the test passing (this was tested by @mocenas).

Expected behavior

Not mounting <app-name>-push-secret when pushing to local repository.

Actual behavior

Mounting the <app-name>-push-secret and causing the timed out because it not existing so it can't mount it.

How to Reproduce?

Created simple reproducer.

1. oc login
2. oc new-project
3. git clone https://github.com/jedla97/quarkus-openshift-reproducer.git
4. cd quarkus-openshift-reproducer
5. mvn clean package -Dquarkus.kubernetes.deploy=true

When either wait around 5-6 min to let it fail or see oc get events and there is FailedMount

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

main, 3.3

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.3

Additional information

No response

[quarkus-bot]

/cc @Sgitario (kubernetes), @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)

[michalvavrik]

hey @iocanel this is probably related to #34241, can you have a look please?

10:11:49,066 INFO  mvn: [ERROR] The timeout period of 300000ms has been exceeded while executing POST /apis/build.openshift.io/v1/namespaces/ts-gbebxzwnfr/buildconfigs/app/instantiatebinary?name=app&namespace=ts-gbebxzwnfr for server api.ocp4-13.dynamic.quarkus:6443
10:11:49,067 INFO  mvn: [ERROR] Failed to upload archive file for the build: app
10:11:49,068 INFO  mvn: [ERROR] Please check cluster events via `oc get events` to see what could have possibly gone wrong
10:11:50,053 INFO  mvn: [WARNING] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] An exception: 'Can't instantiate binary build, due to error reading/writing stream. Can be caused if the output stream was closed by the server.See if something's wrong in recent events in Cluster = Scheduled app-1-build.1779f7164a652d23 Successfully assigned ts-gbebxzwnfr/app-1-build to ocp4-13-nrx44-worker-0-d4s72
10:11:50,053 INFO  mvn: FailedMount app-1-build.1779f752c56bcfba Unable to attach or mount volumes: unmounted volumes=[app-push-secret-push], unattached volumes=[app-push-secret-push build-ca-bundles container-storage-run kube-api-access-qrlg5 build-blob-cache node-pullsecrets buildcachedir build-system-configs build-proxy-ca-bundles builder-dockercfg-k547h-pull container-storage-root buildworkdir]: timed out waiting for the condition
10:11:50,054 INFO  mvn: FailedMount app-1-build.1779f71657de76da MountVolume.SetUp failed for volume "app-push-secret-push" : secret "app-push-secret" not found
10:11:50,054 INFO  mvn: FailedMount app-1-build.1779f732ef3d1b83 Unable to attach or mount volumes: unmounted volumes=[app-push-secret-push], unattached volumes=[container-storage-run buildworkdir app-push-secret-push build-ca-bundles build-proxy-ca-bundles build-blob-cache node-pullsecrets buildcachedir builder-dockercfg-k547h-pull build-system-configs kube-api-access-qrlg5 container-storage-root]: timed out waiting for the condition
10:11:50,055 INFO  mvn:  ' occurred while instantiating the build, however the build has been started.

We specify registry like this:

%ServerlessExtensionDockerBuildStrategyOpenShiftHttpMinimumIT.quarkus.container-image.registry=image-registry.openshift-image-registry.svc:5000

and generated secret push looks like

---
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  annotations:
    app.openshift.io/vcs-url: <<unknown>>
    app.quarkus.io/commit-id: a80a87e5124b78d3cf864fdf228e12c9c31c366c
    app.quarkus.io/build-timestamp: 2023-08-10 - 08:06:10 +0000
  labels:
    app.kubernetes.io/name: app
    app.kubernetes.io/version: 1.0.0-SNAPSHOT
    app.kubernetes.io/managed-by: quarkus
    app.openshift.io/runtime: quarkus
  name: app
spec:
  output:
    pushSecret:
      name: app-push-secret
    to:
      kind: DockerImage
      name: image-registry.openshift-image-registry.svc:5000/ts-gbebxzwnfr/app:1.0.0-SNAPSHOT
  source:
    binary: {}
  strategy:
    sourceStrategy:
      from:
        kind: ImageStreamTag
        name: openjdk-17:latest
---

[rsvoboda]

So this is a regression present in 3.3 and 3.4, @iocanel @geoand any plans to look into this so it gets fixed for 3.5?

[iocanel]

Thought I fixed that last week. Let me check again

[jedla97]

FYI the same issue #35647 and was resolved by #35695 . I verified it with quarkus-qe tests and enabled them so we check it again for some regresion.

As it's fix I closing this issue.