CI is failing due to "/usr/bin/ld: final link failed: No space left on device"

[alamb]

Describe the bug

The CI on main is failing due to the runners running out of disk space

To Reproduce

Example: https://github.com/apache/arrow-datafusion/actions/runs/5277391664/jobs/9545549038

"-Wl,-Bdynamic" "-llzma" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/__w/arrow-datafusion/arrow-datafusion/target/debug/deps/roundtrip_physical_plan-8dff8073998bbd8f" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-nodefaultlibs"
  = note: /usr/bin/ld: final link failed: No space left on device
          collect2: error: ld returned 1 exit status
          

error: could not compile `datafusion-substrait` (test "roundtrip_physical_plan") due to previous error
Error: Process completed with exit code 101.

Expected behavior

Should pass

Additional context

No response

[alamb]

If someone else could look at this that would be great, otherwise I plan to do so in a few hours

[alamb]

Based off of this experiment:

Before any builds start the builder has 11G/35GB if its total disk, and has 33MB in the datafusion checkout:

11G	/
33M	/__w/arrow-datafusion/arrow-datafusion

After the build ends there is 35GB on the disk, all of it in the datafusion checkout:

35G	/
24G	/__w/arrow-datafusion/arrow-datafusion

Of that, almost all of it is in target:

4.0K	/__w/arrow-datafusion/arrow-datafusion/CHANGELOG.md
4.0K	/__w/arrow-datafusion/arrow-datafusion/CODE_OF_CONDUCT.md
4.0K	/__w/arrow-datafusion/arrow-datafusion/CONTRIBUTING.md
108K	/__w/arrow-datafusion/arrow-datafusion/Cargo.lock
4.0K	/__w/arrow-datafusion/arrow-datafusion/Cargo.toml
12K	/__w/arrow-datafusion/arrow-datafusion/LICENSE.txt
4.0K	/__w/arrow-datafusion/arrow-datafusion/NOTICE.txt
4.0K	/__w/arrow-datafusion/arrow-datafusion/README.md
224K	/__w/arrow-datafusion/arrow-datafusion/benchmarks
24K	/__w/arrow-datafusion/arrow-datafusion/ci
44K	/__w/arrow-datafusion/arrow-datafusion/conbench
13M	/__w/arrow-datafusion/arrow-datafusion/datafusion
196K	/__w/arrow-datafusion/arrow-datafusion/datafusion-cli
160K	/__w/arrow-datafusion/arrow-datafusion/datafusion-examples
792K	/__w/arrow-datafusion/arrow-datafusion/dev
444K	/__w/arrow-datafusion/arrow-datafusion/docs
4.0K	/__w/arrow-datafusion/arrow-datafusion/header
2.7M	/__w/arrow-datafusion/arrow-datafusion/parquet-testing
4.0K	/__w/arrow-datafusion/arrow-datafusion/pre-commit.sh
8.0K	/__w/arrow-datafusion/arrow-datafusion/python
4.0K	/__w/arrow-datafusion/arrow-datafusion/rustfmt.toml
24G	/__w/arrow-datafusion/arrow-datafusion/target.   <-- here is the problem
28K	/__w/arrow-datafusion/arrow-datafusion/test-utils
8.8M	/__w/arrow-datafusion/arrow-datafusion/testing

[alamb]

I will next sort out how to reduce the number and size of binaries

At least locally, for me, incremental compilation intermediate files seem to consume a large amount of space:

(arrow_dev) alamb@MacBook-Pro-8:~/Software/arrow-datafusion$ du -s -h target/debug/*
463M	target/debug/build
1.6G	target/debug/deps
780K	target/debug/examples
4.2G	target/debug/incremental

I will experiment with disabling that:
https://doc.rust-lang.org/cargo/reference/profiles.html?highlight=incremental#incremental

[alamb]

incremental seems to take 2GB of space on the builder -- will try disabling it and see what happens

https://github.com/apache/arrow-datafusion/actions/runs/5281745040/jobs/9555736433?pr=6678

Run du -s -h $GITHUB_WORKSPACE/target/debug/* || true
429M	/__w/arrow-datafusion/arrow-datafusion/target/debug/build
21G	/__w/arrow-datafusion/arrow-datafusion/target/debug/deps
4.0K	/__w/arrow-datafusion/arrow-datafusion/target/debug/examples
2.3G	/__w/arrow-datafusion/arrow-datafusion/target/debug/incremental

[alamb]

LOL well I have some fixes that I think will give us some breathing room for a while:

• Disable incremental compilation on CI #6688 (Save 2GB)
• Minor: combine dataframe and dataframe_functions tests to save build time and builder disk space #6682 (save 125MB)
• Minor: combine statistics, filter_pushdown and custom_sources provider tests together to reduce CI disk space #6683 (save 250MB)
• Minor: consolidate repartition test into sql_integration to save builder space and build time #6685 (save 125MB)
• Minor: combine fuzz tests into a single binary to save builder space and build time #6684 (save 1GB)
• Minor: consolidate datafusion_substrait tests into substrait_integration to save builder space and build time #6685 #6686 (save 250MB)

For a total of 22% ( 3.75GB out of 22GB currently made by this job)

[alamb]

After applying all our the updates from yesterday, I measured disk usage again: https://github.com/apache/arrow-datafusion/actions/runs/5291555219/jobs/9577253624?pr=6678

Before the work the disk had 35GB at the end of the test run

35G	/

After all the changes it has 8GB to spare

27G	/

🎉