chore: Add GitHub workflow to publish Docker image #847
Conversation
.github/workflows/docker-publish.yml
Outdated
| - name: Set up Java | ||
| uses: actions/setup-java@v3 | ||
| with: | ||
| distribution: 'temurin' |
There was a problem hiding this comment.
Is this just for the build environment or does this get included in the image? Why not OpenJDK?
There was a problem hiding this comment.
This is just for the build environment so that we can use Maven to get the Comet version number.
There was a problem hiding this comment.
I removed this line - this was copied from an example
.github/workflows/docker-publish.yml
Outdated
| run: echo "The current Maven version is ${{ env.COMET_VERSION }}" | ||
| - name: Build and push Docker image | ||
| run: | | ||
| docker build -t datafusion-comet:spark-3.4-scala-2.12-${{ env.COMET_VERSION }} -f kube/Dockerfile . |
There was a problem hiding this comment.
are we planning to parametrize spark and scala versions later? the same like its done for comet version
There was a problem hiding this comment.
good point, I will update that
There was a problem hiding this comment.
I think also it might be important here to use docker buildx to create images that are compatible with arm if comet supports it, see https://docs.docker.com/build/ci/github-actions/multi-platform/
There was a problem hiding this comment.
Thanks @edmondop I have made this change
Co-authored-by: Oleks V <[email protected]>
.github/workflows/docker-publish.yml
Outdated
| docker push ghcr.io/apache/datafusion-comet:$DOCKER_TAG | ||
| env: | ||
| DOCKER_USER: ${{ github.actor }} | ||
| DOCKER_PASS: ${{ secrets.GITHUB_TOKEN }} |
|
|
||
| # ntoe the use of a wildcard in the file name so that this works with both snapshot and final release versions | ||
| COPY --from=builder /comet/spark/target/comet-spark-spark${SPARK_VERSION}_$SCALA_VERSION-0.2.0*.jar $SPARK_HOME/jars |
.github/workflows/docker-publish.yml
Outdated
| run: echo "The current Maven version is ${{ env.COMET_VERSION }}" | ||
| - name: Build and push Docker image | ||
| run: | | ||
| docker build -t datafusion-comet:spark-3.4-scala-2.12-${{ env.COMET_VERSION }} -f kube/Dockerfile . |
There was a problem hiding this comment.
I think also it might be important here to use docker buildx to create images that are compatible with arm if comet supports it, see https://docs.docker.com/build/ci/github-actions/multi-platform/
Co-authored-by: Edmondo Porcu <[email protected]>
…t into docker-publish
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #847 +/- ##
============================================
+ Coverage 34.05% 34.19% +0.14%
- Complexity 879 885 +6
============================================
Files 112 112
Lines 42959 42950 -9
Branches 9488 9486 -2
============================================
+ Hits 14629 14688 +59
+ Misses 25330 25268 -62
+ Partials 3000 2994 -6 ☔ View full report in Codecov by Sentry. |
I'll be following up with another PR almost immediately after merging this, so will include in that |
|
| # if only Scala code gets modified | ||
| COPY rust-toolchain.toml /comet/rust-toolchain.toml | ||
| COPY native /comet/native | ||
| RUN cd native && RUSTFLAGS="-Ctarget-cpu=native" cargo build --release |
There was a problem hiding this comment.
Is the -Ctarget-cpu=native a good idea? My guess would have been that github runners might use a range of different generations of cpus and therefore have different features? If that is the case different runs here might for example enable different simd instructions and cause varying performance.
Would it not be better to target some specific set of features and document that? And user what something else they should build from source.
Which issue does this PR close?
Part of #721
Rationale for this change
Make it easier for users to use Comet
What changes are included in this PR?
How are these changes tested?
I tested building the Docker image locally.
I don't think I can test the workflow until this PR is merged 😞