Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Value too large for defined data type when cargo build on armv7 with qemu #8719

Open
jdrouet opened this issue Sep 19, 2020 · 15 comments
Open
Labels
A-git Area: anything dealing with git C-bug Category: bug S-triage Status: This issue is waiting on initial triage.

Comments

@jdrouet
Copy link

jdrouet commented Sep 19, 2020

Problem

Trying to build an app for armv7 in a docker container with qemu.
The current result is explained here with the steps to reproduce it : https://github.com/jdrouet/docker-bug-value-too-long

Steps

  1. install docker
  2. install docker-buildx
  3. follow the steps here

Possible Solution(s)
No idea.

Notes
The guys from docker claim it's coming from cargo.

Output of cargo version:

The one from the docker image rust:1-slim-buster

@jdrouet jdrouet added the C-bug Category: bug label Sep 19, 2020
@ehuss
Copy link
Contributor

ehuss commented Oct 7, 2020

Thanks for the report. I think this is a duplicate of #7451. There is some sort of strange interaction of Linux Large File Support with a 64-bit host and a 32-bit docker image running qemu. There's some more discussion on that issue, but in particular see https://bugs.launchpad.net/qemu/+bug/1805913. I'm not sure there is anything Cargo can do about this. I suspect the underlying problem is that libgit2 uses size-agnostic functions like readdir instead of explicitly using readdir64. It may be possible to build libgit2 with -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 (or getconf LFS_CFLAGS), but I have no idea if that will work or what the consequences would be (it seems like it could have portability issues). This may also just be an issue with qemu.

@diabloxenon
Copy link

diabloxenon commented Jun 5, 2021

Hello all,

I have a similar use case i.e. (Qemu userspace emulation for armv7 on x86_64 arch). After chrooting inside the image, i tried building my project repository using that and this was the output.

[root@mnt ~]# cargo build --target=armv7-unknown-linux-gnueabihf
    Updating crates.io index
warning: spurious network error (2 tries remaining): could not read directory '/root/.cargo/registry/index/github.aaakk.us.kg-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
warning: spurious network error (1 tries remaining): could not read directory '/root/.cargo/registry/index/github.aaakk.us.kg-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  could not read directory '/root/.cargo/registry/index/github.aaakk.us.kg-1285ae84e5963aae/.git//refs': Value too large for defined data type; class=Os (2)
[root@mnt ~]# 

However, when fetching it manually using curl https://github.com/rust-lang/crates.io-index from chroot, this worked exactly as it should have. Because of that, I also tested this on qemu aarch64 image and there it worked perfectly.

As suggested earlier by @ehuss, it seems the issue is still persistent to this date with qemu emulation of armv7. The cargo binaries are definitely not the culprit here, and more likely libgit2 seems to be causing this issue.

Internally, there is some incompatiblity going on, even after following the suggestion for building the library with parameters provided by getconf LFS_CFLAGS, the issue remains the same.

Lastly, for this quick and simple test case, I would prefer to use command cargo install exa inside an emulated armv7 image already installled with Rust, for any future testing regarding to its compatiblity.

EDIT: I also proposed a simple solution for anyone that is currenly facing this problem, and until its resolution. #9545

@tillsteinbach
Copy link

If someone is coming here and also needs a workaround for a docker build:
Put this in the beginning of the Dockerfile:
# syntax = docker/dockerfile:experimental
Then do something like this:
RUN --security=insecure mkdir -p /root/.cargo && chmod 777 /root/.cargo && mount -t tmpfs none /root/.cargo && some_command_that_uses_cargo

nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
- That avoids bug in cargo/qemu/ext4, see rust-lang/cargo#8719.
- Increase swap space to give tmpfs bigger size
nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
- That avoids bug in cargo/qemu/ext4, see rust-lang/cargo#8719.
- Increase swap space to give tmpfs bigger size
nijel added a commit to nijel/docker that referenced this issue Nov 23, 2021
- That avoids bug in cargo/qemu/ext4, see rust-lang/cargo#8719.
- Increase swap space to give tmpfs bigger size
nijel added a commit to WeblateOrg/docker that referenced this issue Nov 24, 2021
- That avoids bug in cargo/qemu/ext4, see rust-lang/cargo#8719.
- Increase swap space to give tmpfs bigger size
@nijel
Copy link

nijel commented Nov 24, 2021

If you can't make the above Dockerfile-based workaround, what works as well is using tmpfs for the whole /var/lib/docker on the build host. For GitHub actions, I've done that here: WeblateOrg/docker#1274

mikenye added a commit to sdr-enthusiasts/browser-screenshot-service that referenced this issue Feb 25, 2022
ultram4rine added a commit to ultram4rine/alerter that referenced this issue Mar 12, 2022
hongxu-jia added a commit to Wind-River/meta-lat that referenced this issue Mar 16, 2022
While use qemu-arm to run 32-bit shlibsign on 64-bit host[1], there is a
potential Linux Large File error
...
$ qemu-arm -r 3.2.0 -L path_to/rootfs path_to/rootfs/usr/bin/shlibsign \
  -i path_to/rootfs/usr/lib/libfreebl3.so
path_to/rootfs/usr/lib/libfreebl3.so: Value too large for defined data
type
...

To workaround QEMU issue [2], use host shlibsign(native or nativesdk)
directly

[1] rust-lang/cargo#8719
[2] https://lkml.org/lkml/2018/12/28/461

Signed-off-by: Hongxu Jia <[email protected]>
@tropxy
Copy link

tropxy commented Sep 21, 2022

If you are using buildkit, there is a much cleaner workaround than using the --security=insecureoption. Just use RUN --mount like this:

RUN --mount=type=tmpfs,target=/root/.cargo some_command_that_uses_cargo

For me this worked, but I had to install rust within the same RUN layer:

RUN --mount=type=tmpfs,target=/root/.cargo curl https://sh.rustup.rs -sSf | bash -s -- -y && pip install cryptography==38.0.1

finally, to activate Buildkit, I had to run docker build as:

DOCKER_BUILDKIT=1 docker build .

@philw07
Copy link

philw07 commented Oct 23, 2022

If you are using buildkit, there is a much cleaner workaround than using the --security=insecureoption. Just use RUN --mount like this:

RUN --mount=type=tmpfs,target=/root/.cargo some_command_that_uses_cargo

I had to adapt this slightly to use a different directory for cargo to get it working with the rust base image to build a rust project.

RUN --mount=type=tmpfs,target=/.cargo CARGO_HOME=/.cargo cargo build --release

ci7lus added a commit to vivid-lapin/docker-mitmproxy that referenced this issue Nov 14, 2022
y-almannaee added a commit to y-almannaee/tecberry that referenced this issue Nov 18, 2022
Ruakij added a commit to Ruakij/DockerBUILDs that referenced this issue Nov 23, 2022
FIXME: Disable armv7, because of cargo bug: rust-lang/cargo#8719
@flo-at
Copy link

flo-at commented Jan 10, 2023

Doesn't the tmpfs workaround hurt caching a lot? Cargo has to download and build everything every time if CARGO_HOME is not stored on the filesystem.

theofficialgman added a commit to Botspot/pi-apps that referenced this issue Jan 11, 2023
due to rust-lang/cargo#8719 which itself is an issue with libgit2

DDNet will still be tested on ARM64 (the bullseye ARM64 image)
Taknok added a commit to Taknok/docker-strongswan that referenced this issue Feb 13, 2023
QEMU has issue for mapping data of inode on a 64 arch to a 32 arch
https://gitlab.com/qemu-project/qemu/-/issues/263
rust-lang/cargo#8719
sagikazarmark added a commit to media-streaming-mesh/msm-rtsp-stub that referenced this issue Apr 6, 2023
jennydaman pushed a commit to FNNDSC/pl-abs that referenced this issue Apr 11, 2023
@omarandlorraine
Copy link

Notes
The guys from docker claim it's coming from cargo.

It's a problem with the virtualization layer. The error is not coming from cargo or docker, but from QEMU running 32-bit platforms on 64-bit hosts (I don't mean it's a bug in QEMU though). An alternative workaround could be to run a 32-bit QEMU process if you have a multilib set up. This would mean using the host kernel's 32-bit API, which does not suffer from the conversion problem.

@flo-at
Copy link

flo-at commented Apr 20, 2023

Since cargo (stable) now supports the sparse registry protocol, the easiest workaround is to set the envinronment variable CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse. This avoids the problematic call to libgit2 altogether.
You have to make sure the git binary is installed though, but that shouldn't be a problem.

madiele added a commit to madiele/vod2pod-rss that referenced this issue Sep 17, 2023
cbs228 added a commit to cbs228/sameold that referenced this issue Jan 20, 2024
Our arm64 alpine job was taking too long, and it is likely that
the host VM was under memory pressure. Ensure the VM has enough
swap space to complete the build, and use an improved workaround
from madiele/vod2pod-rss.
cbs228 added a commit to cbs228/sameold that referenced this issue Jan 20, 2024
Our arm64 alpine job was taking too long, and it is likely that
the host VM was under memory pressure. Ensure the VM has enough
swap space to complete the build, and use an improved workaround
from madiele/vod2pod-rss.
@MichaIng
Copy link

MichaIng commented Feb 6, 2024

@flo-at
CARGO_REGISTRIES_CRATES_IO_PROTOCOL=sparse works very well, many thanks. However, basically the same error happens on rustup self uninstall and when temporary directories are attempted to be removed during the process:

info: syncing channel updates for '1.73.0-armv7-unknown-linux-gnueabihf'
info: latest update on 2023-10-05, rust version 1.73.0 (cc66ad468 2023-10-03)
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: downloading component 'rustfmt'
info: installing component 'cargo'
warning: could not delete temp directory: /root/.rustup/tmp/dq_vgjwah9211lz1_dir
info: installing component 'clippy'
warning: could not delete temp directory: /root/.rustup/tmp/_42_3qtwdsd2gjit_dir
info: installing component 'rust-std'
warning: could not delete temp directory: /root/.rustup/tmp/hg7ojeltqnog7xdr_dir
info: installing component 'rustc'
warning: could not delete temp directory: /root/.rustup/tmp/a10n8s4a7catp38h_dir
info: installing component 'rustfmt'
warning: could not delete temp directory: /root/.rustup/tmp/8fbaomir76h2yb8j_dir
    Updating crates.io index
 Downloading crates ...
...
# rustup self uninstall -y
info: removing rustup home
error: could not remove 'rustup_home' directory: '/root/.rustup': Value too large for defined data type (os error 75)

Are there similar options for cargo and rustup past the installation? Otherwise to me the tmpfs workaround seems cleaner, to avoid possible surprises.

@madiele
Copy link

madiele commented Apr 26, 2024

For anyone having this issue, in the end I went to the path of making a Dockerfile with multi-stage builds that will cross-compile for all the architectures I need on the architecture of the pipeline agent and scrapped using qemu for the cargo build, I only use it to run apt install and other small stuff at the end.

If you project allows for this route, you can follow my Dockerfile as guidance

https://github.com/madiele/vod2pod-rss/blob/cac3d27c415ed8341fa8da738e4604b7427c04ae/Dockerfile

I went form 2+ hours build times to 5-10 minutes build times, it was worth the effort.

@chairwa
Copy link

chairwa commented Jun 30, 2024

docker run --privileged -itd --network host --mount type=tmpfs,target=/root/.cargo arm32v7/ubuntu
this way,
/root/.cargo/bin/cargo
bash: ./cargo: Permission denied

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-git Area: anything dealing with git C-bug Category: bug S-triage Status: This issue is waiting on initial triage.
Projects
None yet
Development

No branches or pull requests