Fix intermittent failures coming from buildkit parallel builds (#37403)

The PROD cachie build often fails (especially in v2-8-test) when it
tries to rebuild the PROD cache in parallel on ARM. There is some weird
inter-buildx problem with it and some people experience it sometimes
as documented in moby/buildkit#2367

Instead of finding the root cause, we change this specific job in
CI to run sequentially. This changes the time it will take to update
cache - but not as much as 4x because the builds do not parallelise
very well anyway. Likely instead of 8m we will get maybe total 15m,
and since this is just cache update after everything else has
completed, it does not matter too much if it runs a little longer.

(cherry picked from commit 7a1c600)

.github/workflows/ci.yml

@@ -2519,14 +2519,14 @@ jobs:
       - name: "Copy dist packages to docker-context files"
         run: cp -v --no-preserve=mode,ownership ./dist/*.whl ./docker-context-files
       - name: "Push PROD cache ${{ matrix.python-version }} ${{ matrix.platform }}"
-        run: >
-          breeze prod-image build
-          --builder airflow_cache
-          --install-packages-from-context
-          --run-in-parallel
-          --airflow-constraints-mode constraints-source-providers
-          --prepare-buildx-cache
-          --platform ${{ matrix.platform }}
+        run: |
+          # Do not run parallel builds here as they often fail due to github token expiry issues similar to
+          # those described in https://github.com/moby/buildkit/issues/2367
+          for python in ${{needs.build-info.outputs.python-versions-list-as-string}}; do
+            breeze prod-image build --builder airflow_cache --install-packages-from-context \
+            --airflow-constraints-mode constraints-source-providers --prepare-buildx-cache \
+            --platform ${{ matrix.platform }} --python ${python}
+          done
         env:
           COMMIT_SHA: ${{ github.sha }}
       - name: "Push PROD latest image ${{ matrix.platform }}"