Container builds are very slow after adding new platforms

[swiatekm]

Component(s)

collector, target allocator, opamp bridge

Describe the issue you're reporting

After starting to build container images for ppc64le and s390x, our container image builds became ~4x slower, going from 30 minutes to nearly 2 hours on main. See operator, for example: https://github.com/open-telemetry/opentelemetry-operator/actions/workflows/publish-images.yaml, but this also affects target allocator and Opamp bridge.

This is most likely caused by worse emulation performance in QEMU for these architectures, as we build our binaries within the container build process.

We should instead cross-compile the images and copy them into the image during the build. This adds an overhead of managing the Go toolchain in our CI, but our dependencies are fairly manageable, so this shouldn't be too big of a problem.

[jennydaman]

Hello,

May I suggest a fix to a tiny temporary issue? The release-related GitHub Actions should be dependent on one another. Since the release 0.88.0 was made before its image was published, I'm getting an error

Failed to pull image "ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.88.0": rpc error: code = NotFound

[swiatekm]

Hello,

May I suggest a fix to a tiny temporary issue? The release-related GitHub Actions should be dependent on one another. Since the release 0.88.0 was made before its image was published, I'm getting an error

Failed to pull image "ghcr.io/open-telemetry/opentelemetry-operator/opentelemetry-operator:0.88.0": rpc error: code = NotFound

I created a separate issue to track this: #2304. Unfortunately, this is a lot more work than just making the builds faster, and I expect the latter to be merged before 0.89.0 anyway.

[jaronoff97]

This should be done now :) on to #2304