Problem with docker in docker dind

[Stargateur]

I have a project that use rust, we use gitlab, our workflow use dind, long story short, I wanted to test when rustfmt --check fail. So, I add rustfmt component, then I wanted to remove it because it was not strickly needed in the final image, I don't know if it a good practice or not but my problem is not here.

Dockerfile (should reproduce the problem):

FROM rust:1.40-alpine

RUN rustup --verbose component add rustfmt
RUN rustup --verbose component remove rustfmt

I get the following error:

Step 12/14 : RUN rustup --verbose component remove rustfmt
 ---> Running in 133d04a776ac
verbose: read metadata version: '12'
info: removing component 'rustfmt'
verbose: creating temp file: /usr/local/rustup/tmp/awdp5enle5vsit3d_file
verbose: creating temp file: /usr/local/rustup/tmp/oayeo9qtm87lcqix_file
verbose: creating temp file: /usr/local/rustup/tmp/bqwoqrjghalz2lbz_file
verbose: creating temp file: /usr/local/rustup/tmp/dki4ve129_9ygmnu_file
verbose: creating temp file: /usr/local/rustup/tmp/aifttdiq8mtyiar2_file
verbose: creating temp file: /usr/local/rustup/tmp/xwwc_fuu2536wsli_file
verbose: creating temp file: /usr/local/rustup/tmp/gw64jbi0qndr2qnj_file
verbose: creating temp directory: /usr/local/rustup/tmp/g0_8cy88fnnddr5j_dir
verbose: deleted temp directory: /usr/local/rustup/tmp/g0_8cy88fnnddr5j_dir
info: rolling back changes
error: could not rename component file from '/usr/local/rustup/toolchains/1.40.0-x86_64-unknown-linux-musl/share/doc/rustfmt' to '/usr/local/rustup/tmp/g0_8cy88fnnddr5j_dir/bk'
error: caused by: other os error
The command '/bin/sh -c rustup --verbose component remove rustfmt' returned a non-zero code: 1

with (removed some information):

image: docker:19.03.1

variables:
  DOCKER_TLS_CERTDIR: "/certs"

services:
  - docker:19.03.1-dind

before_script:
  - docker info

build:
  stage: build
  only:
    - branches
  script:
    - docker build -t $CI_PROJECT_NAME .
    - docker run $CI_PROJECT_NAME cargo test

  tags:
    - prod

As you can see error: caused by: other os error is not self describing, before create a docker issue I wanted to ask here. Note that I changed all the pipeline process anyway so this bug is not a problem for me.

I think its because dind image is based on alpine that doesn't have rename.

[RobbieClarken]

backtrace

error: could not rename component file from '/app/rustup/home/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rustfmt' to '/app/rustup/home/tmp/493lnq5mn_71xo1y_dir/bk'
error: caused by: other os error
error: backtrace:
error: stack backtrace:
   0: error_chain::backtrace::imp::InternalBacktrace::new
             at /usr/local/cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/error-chain-0.12.1/src/backtrace.rs:56
   1: core::ops::function::FnOnce::call_once
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/ops/function.rs:232
   2: core::option::Option<T>::unwrap_or_else
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/option.rs:422
   3: error_chain::State::new
             at /usr/local/cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/error-chain-0.12.1/src/lib.rs:679
   4: <core::result::Result<T,E> as rustup::errors::ResultExt<T>>::chain_err::{{closure}}
             at <::error_chain::error_chain::impl_error_chain_processed macros>:201
   5: core::result::Result<T,E>::map_err
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/result.rs:597
   6: <core::result::Result<T,E> as rustup::errors::ResultExt<T>>::chain_err
             at <::error_chain::error_chain::impl_error_chain_processed macros>:198
   7: rustup::utils::utils::rename
             at src/utils/utils.rs:589
   8: rustup::utils::utils::rename_dir
             at src/utils/utils.rs:110
   9: rustup::dist::component::transaction::ChangedItem::remove_dir
             at src/dist/component/transaction.rs:313
  10: rustup::dist::component::transaction::Transaction::remove_dir
             at src/dist/component/transaction.rs:111
  11: rustup::dist::component::components::Component::uninstall
             at src/dist/component/components.rs:311
  12: rustup::dist::manifestation::Manifestation::uninstall_component
             at src/dist/manifestation.rs:329
  13: rustup::dist::manifestation::Manifestation::update
             at src/dist/manifestation.rs:218
  14: rustup::toolchain::Toolchain::remove_component
             at src/toolchain.rs:779
  15: rustup_init::rustup_mode::component_remove
             at src/cli/rustup_mode.rs:1182
  16: rustup_init::rustup_mode::main
             at src/cli/rustup_mode.rs:99
  17: rustup_init::run_rustup_inner
             at src/cli/main.rs:81
  18: rustup_init::run_rustup
             at src/cli/main.rs:52
  19: rustup_init::main
             at src/cli/main.rs:42
  20: std::rt::lang_start::{{closure}}
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/rt.rs:67
  21: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
      std::panicking::try::do_call
             at src/libstd/panicking.rs:292
  22: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  23: std::panicking::try
             at src/libstd/panicking.rs:270
      std::panic::catch_unwind
             at src/libstd/panic.rs:394
      std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  24: std::rt::lang_start
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/rt.rs:67
  25: main
  26: __libc_start_main
  27: _start

The same error occurs with debian base images so it's not an alpine thing. I could replicate the problem with:

FROM rust:1.41.0-slim-buster
RUN echo 'fn main() { std::fs::rename("/tmp/old", "/tmp/new").unwrap(); }' > demo.rs \
    && rustc demo.rs
RUN mkdir /tmp/old
RUN /demo

strace revealed std::fs::rename makes a system call to renameat2 which throws an EXDEV error: "oldpath and newpath are not on the same mounted filesystem". It looks like this syscall cannot move directories across overlayfs layers: moby/moby#25409.

One option would be to try std::fs::rename and if it fails fallback to recursively copying the directory and then deleting the original. This is what the Unix mv command does, as does the Python shutil.move function.

[kinnison]

rustup assumes that the .rustup directory is within a single filesystem so that it can use rename() rather than having to copy/delete when doing its transactional component updates. It'd be very bad for rustup to have to fall back. Instead ensure that you always set up a toolchain in a single RUN line in your Dockerfile. I'm not a docker expert so I don't know the absolute best way, or if there's a way to collapse the layers so that rustup can do its job properly.

[Stargateur]

@kinnison I think I see, that lame we don't have a better error message from std.