build: Parallelize the CI image builds (continued)

[mistercrunch]

new approach

originally started from #25564, but got tangled in intricate changes around release tag management that conflicted with the approach used originally, so I started over by

1. refactoring scripts/docker_build_push.sh, that originally generated a bunch of docker build commands to be executed sequentially, here I move to this bash script running a single command given the parameters, so it can be reused to run multiple build commands in parallel while orchestrating in GitHub actions. I tried to DRY what was in there cautiously
2. changed GitHub actions to generate a matrix of commands for each "target" (dev, lean, lean39, lean310, ...) and "platform" (linux/amd64, linux/arm64), similar to build: Parallelize the CI image builds #25564

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (f8f5c54) 69.48% compared to head (49a6f9b) 69.49%.
Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #26698   +/-   ##
=======================================
  Coverage   69.48%   69.49%           
=======================================
  Files        1894     1894           
  Lines       74151    74151           
  Branches     8243     8243           
=======================================
+ Hits        51527    51528    +1     
+ Misses      20555    20554    -1     
  Partials     2069     2069           

Flag	Coverage Δ
hive	53.80% <ø> (ø)
javascript	56.86% <ø> (ø)
mysql	77.98% <ø> (+<0.01%)	⬆️
postgres	78.08% <ø> (ø)
presto	53.75% <ø> (ø)
python	83.02% <ø> (+<0.01%)	⬆️
sqlite	77.66% <ø> (ø)
unit	56.39% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dpgaspar]

Thanks for recovering this!
just a comment

[dpgaspar]

this is secrets.DOCKERHUB_USER

[dpgaspar]

Just a comment: pull_request trigger will use the PR SHA and run any changes made on any PR. PRs from forks don't contain secrets

[eschutho]

@mistercrunch I updated the docker build at the same time that @sebastianliebscher was writing his initial PR so he didn't have a chance to capture the changes from this PR, but @sfirke and I had updated the docker image latest tag to point to the latest official release image only and not the latest production master image so that subsequent docker deployments would run from the latest official release instead of master. There's a script that you can use from the PR to get the value for latest, or feel free to update the way that runs, too.

[mistercrunch]

Ok, starting over and try a different approach

[mistercrunch]

Request for comments here, given this docker build matrix:

    strategy:
      matrix:
        target: ["dev", "lean", "lean310", "lean39", "websocket", "dockerize"]
        platform: ["linux/amd64", "linux/arm64"]
      fail-fast: false

Do we need every combination for each PR and each SHA in master? Seems a bit crazy. I get that people are on Apple hardware and that's why we need arm as a base platform, but 2 versions of python too? @craig-rueda @rusackas @dpgaspar

[eschutho]

Request for comments here, given this docker build matrix:

    strategy:
      matrix:
        target: ["dev", "lean", "lean310", "lean39", "websocket", "dockerize"]
        platform: ["linux/amd64", "linux/arm64"]
      fail-fast: false

Do we need every combination for each PR and each SHA in master? Seems a bit crazy. I get that people are on Apple hardware and that's why we need arm as a base platform, but 2 versions of python too? @craig-rueda @rusackas @dpgaspar

We can probably remove lean39. I added that version as a request from someone in the community who later found a workaround, and it's probably not needed. We can maybe check with @rusackas on data around who is using it, but I don't think we really need to have docker images based on multiple versions of python.

[eschutho]

LGTM with or without the lean39 image. 👍

[dpgaspar]

Request for comments here, given this docker build matrix:

    strategy:
      matrix:
        target: ["dev", "lean", "lean310", "lean39", "websocket", "dockerize"]
        platform: ["linux/amd64", "linux/arm64"]
      fail-fast: false

Do we need every combination for each PR and each SHA in master? Seems a bit crazy. I get that people are on Apple hardware and that's why we need arm as a base platform, but 2 versions of python too? @craig-rueda @rusackas @dpgaspar

We can probably remove lean39. I added that version as a request from someone in the community who later found a workaround, and it's probably not needed. We can maybe check with @rusackas on data around who is using it, but I don't think we really need to have docker images based on multiple versions of python.

I may be missing some requirements, but totally agree, I actually don't see the point on building every flavour on every PR SHA, also on a PR from a fork, images are just built and not pushed so we are just testing them. My take would be to just leave lean, linux/amd64 and websocket, linux/amd64 on every PR SHA, should be enough for testing purposes then build push all of them when merging to master