-
-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
hadoop: add aarch64 support #158613
hadoop: add aarch64 support #158613
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
diff looks good to me.
How expensive are the tests? the x86_64-linux machine has pretty slow cores
I used an I'm running the tests again on |
cc @grahamc about resource usage |
let | ||
platformUrlSuffix = | ||
if stdenv.isx86_64 && (stdenv.isLinux || stdenv.isDarwin) | ||
then "" | ||
else | ||
if stdenv.isAarch64 && (stdenv.isLinux || stdenv.isDarwin) | ||
then "-aarch64" | ||
else throw "Hadoop does not currently support ${stdenv.system}."; | ||
in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
generally we don't bother with doing platform radix, as that's captured in the .drv
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[13:26:04] jon@nixos ~/projects/nixpkgs (staging-next)
$ nix-instantiate -A hadoop
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/smsf7vdmscgj1fqdvc7b14fw186pn7kn-hadoop-3.3.1.drv
[13:47:28] jon@nixos ~/projects/nixpkgs (staging-next)
$ nix show-derivation /nix/store/smsf7vdmscgj1fqdvc7b14fw186pn7kn-hadoop-3.3.1.drv | grep system
"system": "x86_64-linux",
"system": "x86_64-linux",
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does a24835e handle it better?
x86_64-linux = "1b3v16ihysqaxw8za1r5jlnphy8dwhivdx2d0z64309w57ihlxxd"; | ||
x86_64-darwin = "1b3v16ihysqaxw8za1r5jlnphy8dwhivdx2d0z64309w57ihlxxd"; | ||
aarch64-linux = "sha256-v1Om2pk0wsgKBghRD2wgTSHJoKd3jkm1wPKAeDcKlgI="; | ||
aarch64-darwin = "sha256-v1Om2pk0wsgKBghRD2wgTSHJoKd3jkm1wPKAeDcKlgI="; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
there should be some instruction on how to generate the shas
E.g. nix-prefetch-url ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whoops! It should be fixed in 0aaa8f7.
Uh oh, I'm getting a failed build for Info follows below:
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 72
On-line CPU(s) list: 0-71
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
CPU family: 6
Model: 85
Thread(s) per core: 2
Core(s) per socket: 18
Socket(s): 2
Stepping: 4
CPU max MHz: 3500.0000
CPU min MHz: 1200.0000
BogoMIPS: 6000.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_
tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpu
id_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq r
dseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke
md_clear flush_l1d
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 1.1 MiB (36 instances)
L1i: 1.1 MiB (36 instances)
L2: 36 MiB (36 instances)
L3: 49.5 MiB (2 instances)
NUMA:
NUMA node(s): 2
NUMA node0 CPU(s): 0-17,36-53
NUMA node1 CPU(s): 18-35,54-71
Vulnerabilities:
Itlb multihit: KVM: Mitigation: VMX disabled
L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Meltdown: Mitigation; PTI
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Srbds: Not affected
Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
$ lsmem
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000307fffffff 194G online yes 0-96
Memory block size: 2G
Total online memory: 194G
Total offline memory: 0B
$ time nix build github:connorbaker/nixpkgs#nixosTests.hadoop-hdfs --out-link hadoop-hdfs
real 2m35.571s
user 2m38.870s
sys 1m39.416s
$ time nix build github:connorbaker/nixpkgs#nixosTests.hadoop-yarn --out-link hadoop-yarn
real 0m27.148s
user 0m39.548s
sys 0m20.513s
$ time nix build github:connorbaker/nixpkgs#nixosTests.hadoop --out-link hadoop
error: builder for '/nix/store/kxzchmmkwh64ain16k7jsqwgxdmpg256-vm-test-run-unnamed.drv' failed with exit code 1;
last 10 log lines:
> kill vlan (pid 16)
> kill vlan (pid 15)
> kill vlan (pid 14)
> kill vlan (pid 13)
> kill vlan (pid 12)
> kill vlan (pid 11)
> kill vlan (pid 10)
> kill vlan (pid 9)
> kill vlan (pid 8)
> kill vlan (pid 7)
For full logs, run 'nix log /nix/store/kxzchmmkwh64ain16k7jsqwgxdmpg256-vm-test-run-unnamed.drv'.
real 18m2.483s
user 20m10.371s
sys 7m27.104s Logs for the failed tests are here: https://gist.github.com/ConnorBaker/ec83223ed31256d9ba637d40f0ee4296 I'm seeing a lot of java.io.IOException: Cannot run program "/run/wrappers/yarn-nodemanager/bin/container-executor": error=2, No such file or directory� repeated in the logs. EDIT: Kind of unrelated, but is there a way to get |
On the same It works, so I think I may have messed up by setting $ curl -L --silent https://github.com/NixOS/nixpkgs/archive/refs/heads/master.tar.gz | tar -xz
$ cd nixpkgs-master/
$ time nix build -f ./nixos/tests/hadoop/hdfs.nix --out-link hadoop-hdfs
real 1m44.698s
user 2m23.901s
sys 0m55.611s
$ time nix build -f ./nixos/tests/hadoop/yarn.nix --out-link hadoop-yarn
real 0m28.016s
user 0m41.740s
sys 0m19.863s
$ time nix build -f ./nixos/tests/hadoop/hadoop.nix --out-link hadoop
real 5m2.715s
user 7m42.968s
sys 3m52.261s |
|
||
let |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
probably going to need to make an assertion
attribute 'i686-linux' missing, at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/mr-est/ofborg-evaluator-0/pkgs/applications/networking/cluster/hadoop/default.nix:31:18
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What would that assert look like? Do you have any pointers/references to things I could look at?
As of the latest, I was able to build the tests successfully on Here it is on $ time nix build github:/connorbaker/nixpkgs/master#nixosTests.{hadoop,hadoop-yarn,hadoop-hdfs} --store ~/ramdisk
real 5m49.831s
user 8m3.129s
sys 4m53.577s and on
While debugging, I found that some of the problems I was having were a result of specifying After modifying |
This commit also changes the names of the tests for Hadoop so they use dashes instead of dots, and makes the default `hadoop` test what would have been `hadoop-all` after the rename. This change should mean that we're able to run `nix build github:nixos/nixpkgs/master#nixosTests.hadoop` which I was unable to do prior to this change.
Volth and @illustris, would you also be willing to review? Volth, I can't find your GitHub user -- if you're still maintaining Hadoop, can you update |
Diff looks good. Also verified that NixOS tests succeed on aarch64 and x86_64. Thanks! |
…to simplify expression
Sure! See ConnorBaker@799d743 for the latest. (I had to wrap So here's the latest build on an $ nix build github:connorbaker/nixpkgs#hadoop --print-build-logs
hadoop> unpacking sources
hadoop> unpacking source archive /nix/store/57vfasjl07by1fb4krlh9sn4hmz4w5cs-hadoop-3.3.1.tar.gz
hadoop> source root is hadoop-3.3.1
hadoop> setting SOURCE_DATE_EPOCH to timestamp 1623737898 of file hadoop-3.3.1/share/doc/hadoop/project-reports.html
hadoop> patching sources
hadoop> configuring
hadoop> no configure script, doing nothing
hadoop> building
hadoop> no Makefile, doing nothing
hadoop> running tests
hadoop> no Makefile or custom checkPhase, doing nothing
hadoop> installing
hadoop> post-installation fixup
hadoop> shrinking RPATHs of ELF executables and libraries in /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libnativetask.so.1.0.0
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libhdfs.so.0.0.0
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libhadoop.so.1.0.0
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libhdfspp.so.0.1.0
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/examples/wordcount-nopipe
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/examples/wordcount-part
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/examples/wordcount-simple
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/examples/pipes-sort
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/test-container-executor
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/container-executor
hadoop> shrinking /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/oom-listener
hadoop> strip is /nix/store/v5a649mvjkgwcxsjiz6ngw03xhnh7isp-binutils-wrapper-2.35.2/bin/strip
hadoop> stripping (with command strip and flags -S) in /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/bin
hadoop> patching script interpreter paths in /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/stop-estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/start-estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/sls/bin/rumen2sls.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/sls/bin/slsrun.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-workload/bin/parse-start-timestamp.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-workload/bin/start-workload.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-blockgen/bin/generate-block-lists.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-infra/bin/upload-fsimage.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-infra/bin/parse-metrics.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-infra/bin/create-slim-hadoop-tar.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-infra/bin/start-dynamometer-cluster.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-all.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-yarn.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-dfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-dfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/hadoop-daemon.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/mr-jobhistory-daemon.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-all.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/kms.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/yarn-daemons.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-secure-dns.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/distribute-exclude.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/workers.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-yarn.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/refresh-namenodes.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-secure-dns.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-balancer.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-balancer.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/yarn-daemon.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/httpfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/hadoop-daemons.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/hdfs-config.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/hadoop-config.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-streaming.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-distcp.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-archive-logs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-gridmix.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-mapreduce.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-azure.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-s3guard.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-kms.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-azure-datalake.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-httpfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-openstack.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-yarn.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-aws.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-kafka.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-rumen.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-archives.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-extras.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-hdfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-aliyun.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/mapred-config.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/hadoop-functions.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-streaming.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-distcp.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-archive-logs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-gridmix.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-sls.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-resourceestimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-dynamometer-blockgen.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-dynamometer-workload.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-aws.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-rumen.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-archives.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-extras.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-dynamometer-infra.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/hadoop-layout.sh.example: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/mapred: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/yarn: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/hadoop: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/hdfs: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash"
hadoop> checking for references to /build/ in /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1...
hadoop> automatically fixing dependencies for ELF files
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libhdfspp.so.0.1.0
hadoop> libprotobuf.so.18 -> found: /nix/store/n7wm7y0jlppd8hrmbn1gd7f9ji7qrkpg-protobuf-3.7.1/lib/libprotobuf.so.18
hadoop> libcrypto.so.1.1 -> found: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib/libcrypto.so.1.1
hadoop> libsasl2.so.2 -> found: /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libsasl2.so.2
hadoop> libstdc++.so.6 -> found: /nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib/libstdc++.so.6
hadoop> setting RPATH to: /nix/store/n7wm7y0jlppd8hrmbn1gd7f9ji7qrkpg-protobuf-3.7.1/lib:/nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib:/nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native:/nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libhadoop.so.1.0.0
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libhdfs.so.0.0.0
hadoop> libjvm.so -> found: /nix/store/kylb9qkyg5njrkln7fpx5lpb4qidnjvg-openjdk-headless-11.0.12+7/lib/openjdk/lib/server/libjvm.so
hadoop> setting RPATH to: /nix/store/kylb9qkyg5njrkln7fpx5lpb4qidnjvg-openjdk-headless-11.0.12+7/lib/openjdk/lib/server
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libnativetask.so.1.0.0
hadoop> libz.so.1 -> found: /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libz.so.1
hadoop> libsnappy.so.1 -> found: /nix/store/4i9d1kj13qw0fm1f05zcga715ld3cfp9-snappy-1.1.9/lib/libsnappy.so.1
hadoop> libjvm.so -> found: /nix/store/kylb9qkyg5njrkln7fpx5lpb4qidnjvg-openjdk-headless-11.0.12+7/lib/openjdk/lib/server/libjvm.so
hadoop> libstdc++.so.6 -> found: /nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib/libstdc++.so.6
hadoop> setting RPATH to: /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native:/nix/store/4i9d1kj13qw0fm1f05zcga715ld3cfp9-snappy-1.1.9/lib:/nix/store/kylb9qkyg5njrkln7fpx5lpb4qidnjvg-openjdk-headless-11.0.12+7/lib/openjdk/lib/server:/nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/examples/pipes-sort
hadoop> libcrypto.so.1.1 -> found: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib/libcrypto.so.1.1
hadoop> libstdc++.so.6 -> found: /nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib/libstdc++.so.6
hadoop> setting RPATH to: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib:/nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/examples/wordcount-simple
hadoop> libcrypto.so.1.1 -> found: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib/libcrypto.so.1.1
hadoop> libstdc++.so.6 -> found: /nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib/libstdc++.so.6
hadoop> setting RPATH to: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib:/nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/examples/wordcount-part
hadoop> libcrypto.so.1.1 -> found: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib/libcrypto.so.1.1
hadoop> libstdc++.so.6 -> found: /nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib/libstdc++.so.6
hadoop> setting RPATH to: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib:/nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/examples/wordcount-nopipe
hadoop> libcrypto.so.1.1 -> found: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib/libcrypto.so.1.1
hadoop> libstdc++.so.6 -> found: /nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib/libstdc++.so.6
hadoop> setting RPATH to: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib:/nix/store/ndnqiz3nnifj1blhg9q626xlmkqq1nmh-gcc-10.3.0-lib/lib
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/oom-listener
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/container-executor
hadoop> libcrypto.so.1.1 -> found: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib/libcrypto.so.1.1
hadoop> setting RPATH to: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib
hadoop> searching for dependencies of /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/bin/test-container-executor
hadoop> libcrypto.so.1.1 -> found: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib/libcrypto.so.1.1
hadoop> setting RPATH to: /nix/store/2aprvflf41z1dfma87mzd3md51xm2g71-openssl-1.1.1m/lib And here's the same build on the latest on $ nix build .#hadoop --print-build-logs
hadoop> unpacking sources
hadoop> unpacking source archive /nix/store/7i751wf2lyx1rfkm2207rb51vbckzch5-hadoop-3.3.1-aarch64.tar.gz
hadoop> source root is hadoop-3.3.1
hadoop> setting SOURCE_DATE_EPOCH to timestamp 1623758040 of file hadoop-3.3.1/share/doc/hadoop/project-reports.html
hadoop> patching sources
hadoop> updateAutotoolsGnuConfigScriptsPhase
hadoop> configuring
hadoop> no configure script, doing nothing
hadoop> building
hadoop> no Makefile, doing nothing
hadoop> running tests
hadoop> no Makefile or custom checkPhase, doing nothing
hadoop> installing
hadoop> post-installation fixup
hadoop> strip is /nix/store/km02igh4pshp20d0wn89rf5jjfxcm8v5-clang-wrapper-11.1.0/bin/strip
hadoop> stripping (with command strip and flags -S) in /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/bin
hadoop> patching script interpreter paths in /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/bin/mapred: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/bin/yarn: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/bin/hadoop: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/bin/hdfs: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-extras.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-distcp.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-archive-logs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-sls.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-resourceestimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-dynamometer-infra.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-rumen.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-dynamometer-blockgen.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-streaming.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-aws.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-dynamometer-workload.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-archives.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/tools/hadoop-gridmix.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/hadoop-config.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/hadoop-layout.sh.example: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/hadoop-functions.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-extras.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-s3guard.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-distcp.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-kms.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-openstack.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-archive-logs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-mapreduce.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-httpfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-azure-datalake.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-azure.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-rumen.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-yarn.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-streaming.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-aws.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-kafka.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-aliyun.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-hdfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-archives.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/shellprofile.d/hadoop-gridmix.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/hdfs-config.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/libexec/mapred-config.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/yarn-daemon.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/yarn-daemons.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/distribute-exclude.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-yarn.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-all.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/mr-jobhistory-daemon.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/kms.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/workers.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-dfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-secure-dns.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-balancer.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-secure-dns.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-dfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/httpfs.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-yarn.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/start-balancer.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/hadoop-daemons.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/hadoop-daemon.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/refresh-namenodes.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/sbin/stop-all.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/sls/bin/rumen2sls.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/sls/bin/slsrun.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-infra/bin/upload-fsimage.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-infra/bin/create-slim-hadoop-tar.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-infra/bin/parse-metrics.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-infra/bin/start-dynamometer-cluster.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-blockgen/bin/generate-block-lists.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-workload/bin/start-workload.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-workload/bin/parse-start-timestamp.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/stop-estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash"
hadoop> /nix/store/g1bkm13bdn7a2im98ijwkgx7vbwfsgl3-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/start-estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p12/bin/bash" On $ ./result/bin/hadoop checknative -a
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
2022-02-24 03:57:41,976 INFO [main] bzip2.Bzip2Factory (Bzip2Factory.java:isNativeBzip2Loaded(61)) - Successfully loaded & initialized native-bzip2 library system-native
2022-02-24 03:57:41,978 INFO [main] zlib.ZlibFactory (ZlibFactory.java:loadNativeZLib(59)) - Successfully loaded & initialized native-zlib library
2022-02-24 03:57:41,981 WARN [main] erasurecode.ErasureCodeNative (ErasureCodeNative.java:<clinit>(49)) - Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)
2022-02-24 03:57:41,981 WARN [main] erasurecode.ErasureCodeNative (ErasureCodeNative.java:<clinit>(55)) - ISA-L support is not available in your platform... using builtin-java codec where applicable
2022-02-24 03:57:42,002 INFO [main] nativeio.NativeIO (NativeIO.java:isPmdkAvailable(192)) - The native code was built without PMDK support.
Native library checking:
hadoop: true /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libhadoop.so.1.0.0
zlib: true /nix/store/bqjh8dc6m5plswxrmmn9x4cpxrrw94lv-zlib-1.2.11/lib/libz.so.1
zstd : true /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libzstd.so.1
bzip2: true /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libbz2.so.1
openssl: true /nix/store/i9pg1y2ki390xp6ypcpglzqk92slr6bd-hadoop-3.3.1/lib/hadoop-3.3.1/lib/native/libcrypto.so
ISA-L: false Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)
PMDK: false The native code was built without PMDK support.
2022-02-24 03:57:42,007 INFO [main] util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: ExitException while on $ ./result/bin/hadoop checknative -a
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
2022-02-23 23:03:11,804 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop: false
zlib: false
zstd : false
bzip2: false
openssl: false
ISA-L: false
PMDK: false
2022-02-23 23:03:11,924 INFO [main] util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: ExitException I've talked (a little bit) previously with people about how to do the same patching on macOS but I can't find my notes on it at the moment. If you happen to know what commands would be needed to help Hadoop find the libraries, let me know! |
From the logs, it looks like autoPatchelfHook isn't running at all on darwin.
You'll just have to drop the stdenv.isLinux here. |
I removed it as suggested and it errored (I thought Here are the last few lines of the log (the rest of which is identical to the log before the change): /nix/store/qx36hf946n1bcdb60nw8iambf1500r28-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-blockgen/bin/generate-block-lists.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40
/nix/store/qx36hf946n1bcdb60nw8iambf1500r28-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-workload/bin/start-workload.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9
/nix/store/qx36hf946n1bcdb60nw8iambf1500r28-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/dynamometer/dynamometer-workload/bin/parse-start-timestamp.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c4
/nix/store/qx36hf946n1bcdb60nw8iambf1500r28-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/stop-estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5
/nix/store/qx36hf946n1bcdb60nw8iambf1500r28-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-5.1-p1
/nix/store/qx36hf946n1bcdb60nw8iambf1500r28-hadoop-3.3.1/lib/hadoop-3.3.1/share/hadoop/tools/resourceestimator/bin/start-estimator.sh: interpreter directive changed from "#!/usr/bin/env bash" to "/nix/store/m0nsx4ai768kjp9c40vhkpj9pxlii46m-bash-
automatically fixing dependencies for ELF files
/nix/store/a4q92isrpiy004njsk6x66rsahw62am3-auto-patchelf-hook/nix-support/setup-hook: line 333: -l: command not found I tried again after also removing the optional in
to see if that helped (it would make sense to use $ nix log /nix/store/s4jpnbvmmfk4kgk7rl2292kis1n2gd35-hadoop-3.3.1.drv
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/7i751wf2lyx1rfkm2207rb51vbckzch5-hadoop-3.3.1-aarch64.tar.gz
source root is hadoop-3.3.1
setting SOURCE_DATE_EPOCH to timestamp 1623758040 of file hadoop-3.3.1/share/doc/hadoop/project-reports.html
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
no Makefile, doing nothing
@nix { "action": "setPhase", "phase": "checkPhase" }
running tests
no Makefile or custom checkPhase, doing nothing
@nix { "action": "setPhase", "phase": "installPhase" }
installing
/nix/store/0qlq4s175mzlx9g0aqcdk95j92g1z7mf-stdenv-darwin/setup: line 1378: patchelf: command not found Any ideas? |
ah, my bad... Yes, patchelf doesn't work on darwin. I'm not sure if hadoop with native libs can even work in darwin with the way we're packaging now.
The native lib files bundled with hadoop are ELF files. Apache doesn't release builds with |
I actually looked into building Hadoop from source, but all the Java tooling threw me... maybe I'll try again? As long as the builds are cached, end-users shouldn't have to compile it themselves, right? And we can always offer the binary distribution alongside it. |
That's how it was done before #141143 Building from source is not a good idea because
|
Yikes, no kidding! So without finding a way to link the libraries on macOS (which I absolutely lack the knowledge to do) I guess this is as good as it'll get. In terms of the scope of this pull request, do you think it's good? Are there other things you'd like to see done, or ways I can clean it up further? I'm still learning about most of nix's builtins and the nixpkg's lib, so any advice would be appreciated! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The PR looks good. I just have a few minor non-blocking suggestions.
Great feedback! I had to keep the $ nix log /nix/store/3xmyasz6bxsfq0608krxv7pbs9kbx4by-hadoop-3.3.1.drv
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/7i751wf2lyx1rfkm2207rb51vbckzch5-hadoop-3.3.1-aarch64.tar.gz
source root is hadoop-3.3.1
setting SOURCE_DATE_EPOCH to timestamp 1623758040 of file hadoop-3.3.1/share/doc/hadoop/project-reports.html
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
no configure script, doing nothing
@nix { "action": "setPhase", "phase": "buildPhase" }
building
no Makefile, doing nothing
@nix { "action": "setPhase", "phase": "checkPhase" }
running tests
no Makefile or custom checkPhase, doing nothing
@nix { "action": "setPhase", "phase": "installPhase" }
installing
/nix/store/0qlq4s175mzlx9g0aqcdk95j92g1z7mf-stdenv-darwin/setup: line 1378: patchelf: command not found |
This commit also changes the names of the tests for Hadoop so they use dashes instead of dots,
and makes the default
hadoop
test what would have beenhadoop-all
after the rename.This change should mean that we're able to run
nix build github:nixos/nixpkgs/master#nixosTests.hadoop
which I was unable to do prior to this change.
I've built this change on
aarch64-darwin
,aarch64-linux
, andx86_64-linux
. I've run the NixOS tests for Hadoop onaarch64-linux
, andx86_64-linux
. Onaarch64-darwin
I just ranhadoop version
andhadoop checknative
.I made a flake to spin up an AWS EC2 instance on a bare-metal instance (required because NixOS tests need KVM), create a ramdisk (to avoid provisioning a larger amount of disk space), and build and test the change there: https://github.com/ConnorBaker/nix-hadoop-aarch64-test.
I've seen some posts discussing getting the native libraries working for macOS, but I wasn't sure how to do that. I'm not familiar with
patchelf
orinstall_name_tool
or how linking works, so any help would be appreciated!Motivation for this change
Spark requires Hadoop, so adding Hadoop's aarch64 distribution allows us to use Spark on that platform.
Things done
sandbox = true
set innix.conf
? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)nixos/doc/manual/md-to-db.sh
to update generated release notes