Backstage pod is crashing due to : Failed to pull image "backstage:1.0.0": rpc error: code = failed to pull and unpack image "docker.io/library/backstage:1.0.0"

[cmoulliard]

Issue

The Backstage pod created by ./idpbuilder create --buildName localdev is crashing due to the following docker error:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  21m                   default-scheduler  Successfully assigned backstage/backstage-7bb974cbff-ffzj9 to localdev-worker
  Normal   Pulling    20m (x4 over 21m)     kubelet            Pulling image "backstage:1.0.0"
  Warning  Failed     20m (x4 over 21m)     kubelet            Failed to pull image "backstage:1.0.0": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/backstage:1.0.0"
: failed to resolve reference "docker.io/library/backstage:1.0.0": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     20m (x4 over 21m)     kubelet            Error: ErrImagePull
  Warning  Failed     19m (x6 over 21m)     kubelet            Error: ImagePullBackOff
  Normal   BackOff    6m36s (x64 over 21m)  kubelet            Back-off pulling image "backstage:1.0.0"

AFAIK, the official backtage image is now hosted on ghcr - https://github.com/backstage/backstage/pkgs/container/backstage. See what the helm chart references as image here: https://github.com/backstage/charts/blob/main/charts/backstage/values.yaml#L93-L99

  image:

    # -- Backstage image registry
    registry: ghcr.io

    # -- Backstage image repository
    repository: backstage/backstage

    # -- Backstage image tag (immutable tags are recommended)
    tag: latest

Warning: If we install backstage using ghcr.io repo, then we will get another error as a secret is missing

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  15m                 default-scheduler  Successfully assigned backstage/backstage-5c7f6948db-bqhnv to localdev-worker
  Normal   Pulling    15m                 kubelet            Pulling image "ghcr.io/backstage/backstage:latest"
  Normal   Pulled     14m                 kubelet            Successfully pulled image "ghcr.io/backstage/backstage:latest" in 28.307310138s (28.307373638s including waiting)
  Warning  Failed     12m (x12 over 14m)  kubelet            Error: secret "backstage-secrets" not found
  Normal   Pulled     5m (x46 over 14m)   kubelet            Container image "ghcr.io/backstage/backstage:latest" already present on machine

[jessesanford]

The image should actually be: https://gallery.ecr.aws/cnoe-io/backstage which is our CNOE branded backstage image. We should update the argo app to use this image

[cmoulliard]

The image should actually be: https://gallery.ecr.aws/cnoe-io/backstage which is our CNOE branded backstage image. We should update the argo app to use this image

Why having another backstage image vs using the official ? Is it because you package top of backstage some CNOE plugins ? If this is the case, what are then the plugins you are packaging AND which app.config should we use ?

[jessesanford]

@nabuskey can you respond to this?

[nabuskey]

Backstage customization requires code changes and it's really more of a framework than a ready-to-use solution. So we did have to publish our own image to include plugins and integrations. You don't have to use it and go with whatever image you'd like to use. As for the configuration file, it should be baked into the image.

We really need to make this more configurable via #5

[cmoulliard]

Can you please fix the existing issues as by the following secret do not exist ?

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-10-12T07:16:38Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-10-12T07:16:38Z"
    message: 'containers with unready status: [backstage]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-10-12T07:16:38Z"
    message: 'containers with unready status: [backstage]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-10-12T07:16:38Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: public.ecr.aws/cnoe-io/backstage:v0.0.2
    imageID: ""
    lastState: {}
    name: backstage
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: secret "backstage-secrets" not found
        reason: CreateContainerConfigError

and pod will crash using the image: public.ecr.aws/cnoe-io/backstage:v0.0.2

Can you also document and add the link about where you maintain the project + plugins to build such image ?