Integration test fails to build docker images

[andygrove]

Describe the bug

ballista-base gets tagged as 0.6.0 but then a subsequent docker build is looking for 0.5.0-SNAPSHOT

Successfully tagged ballista-base:0.6.0
Sending build context to Docker daemon  11.42MB
Step 1/56 : ARG RELEASE_FLAG=--release
Step 2/56 : FROM ballista-base:0.5.0-SNAPSHOT AS base
pull access denied for ballista-base, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

To Reproduce
./dev/integration-tests.sh

[andygrove]

I changed 0.5.0-SNAPSHOT to 0.6.0 but there is another issue:

Step 20/56 : FROM base as cacher
 ---> 374276d706db
Step 21/56 : COPY --from=planner /tmp/ballista/recipe.json recipe.json
 ---> Using cache
 ---> 80fb39713935
Step 22/56 : RUN cargo chef cook $RELEASE_FLAG --recipe-path recipe.json
 ---> Running in 3a6371cc71a5
    Updating crates.io index
error: failed to select a version for the requirement `ballista-core = "^0.6.0"`
candidate versions found which didn't match: 0.0.1
location searched: /tmp/ballista/ballista/rust/core
required by package `ballista v0.0.1 (/tmp/ballista/ballista/rust/client)`
thread 'main' panicked at 'Exited with status code: 101', /usr/local/cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/cargo-chef-0.1.25/src/recipe.rs:142:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
The command '/bin/sh -c cargo chef cook $RELEASE_FLAG --recipe-path recipe.json' returned a non-zero code: 101

[andygrove]

I commented out some of the caching in the docker image just to try and get the tests working locally and ran into yet another issue

error[E0658]: arbitrary expressions in key-value attributes are unstable
  --> ballista/rust/core/src/lib.rs:18:10
   |
18 | #![doc = include_str!("../README.md")]
   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: see issue #78835 <https://github.com/rust-lang/rust/issues/78835> for more information

error: aborting due to previous error

EDIT: upgrading to Rust 1.54 in the ballista-base image resolves this issue

[houqp]

I am able to reproduce this as well, I will take a closer look at it later tonight. One really strange thing is cargo-chef is detecting both ballista-core and ballista-client to be at version 0.0.1. But this version string doesn't exist in the code base, it likely has something to do with docker cache.

[houqp]

one interesting observation from my quick test, removing , version = "0.6.0" in ballista-core = { path = "../core", version = "0.6.0" } gets rid of the error

[houqp]

this is very strange because datafusion dependency pin doesn't have this problem, it's only happening to ballista dependencies.

[houqp]

Pinning cargo chef to version 0.1.23 using cargo install cargo-chef --version 0.1.23 fixes the error. There are a bunch of version masking changes (https://github.com/LukeMathWalker/cargo-chef/commits/main) introduced lately, which I imagine are the source of this regression. We should probably file an upstream issue to further troubleshoot there.