Installation problems on Ubuntu 18.04.5 #117
Comments
mbojan
changed the title
mbojan
changed the title
|
Yikes. Are you using the default gcc compiler? I'm not seeing this on my ubuntu 18.04 docker images. |
|
I think so: |
|
Are you running on any unusual hardware? Anything else that may give me a hint on what is different between your environment and my testing boxes? |
|
It is HP 6450b laptop, nothing unusual. How can I debug this further? |
|
So in the new version we are using a custom build of libgit2 instead of the system one. But that doesn't seem to work in your case. A workaround is to use this: Sys.setenv(USE_SYSTEM_LIBGIT2=1)
install.packages("gert")But I still want to understand why our binaries don't work on your system. |
Thanks for the workaround.
Sure. Let me know what do you need. |
|
I've added some debug statements, could you maybe run this to see where it crashes: remotes::install_github("r-lib/gert@debug") |
|
I did the above installation with |
|
Thanks. I don't think the make process is the culprit; it seems your build succeeds, but it crashes when trying to read the global config options on startup. |
|
Same issue on Ubuntu 20.04 and Debian 10, both about 8 year old X86 boxes. Tried installing r-cran-git2r, libgit2-dev, neither helped. The above Sys.setenv() did the trick. gcc (Debian 8.3.0-6) 8.3.0, and gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0. |
|
|
Here is my output: |
|
That's so strange, you all seems to be crashing in git_config_open_default which simply reads your global git config files. I can't reproduce this on any of my machines, there must be something related to the local setup. Do you folks have a config file in cc @ethomson |
|
FWIW here is my (successful) output on an Ubuntu 18 AMI |
nothing set for |
|
Thanks a lot for the quick workaround earlier, BTW. I was really banging my head against the wall. |
|
Well this really shouldn't be happening in the first place. Are you using a desktop linux, or server? Also you say 8 year old X86 boxes, these are 64-bit, right? |
|
I got a similar error in the setup specified below. The workaround works well. EDIT: corrected error message & add further info |
|
Thanks. It seems your CPU seems pretty recent, which makes me doubt if this is hardware related... Does |
|
Ubuntu desktop, Debian (bare net-install) |
|
I run into similar issues with old Linux distros. After debugging, I found that it's due to the static version of Lines 24 to 30 in 6585a65 which results in: ** using staged installation
> curl::curl_download("http://r-lib.github.io/gert/get-libgit2-linux.sh","get-libgit2-linux.sh")
> curl::curl_download('http://r-lib.github.io/gert/libgit2-1.1.0.x86_64_linux.tar.gz','bundle.tar.gz')
Using static libgit2-1.1.0 for Linux x86_64The solution, for me, was to use the system version of
sudo add-apt-repository ppa:cran/libgit2
sudo apt-get update
sudo apt-get install libgit2-dev
export USE_SYSTEM_LIBGIT2=1
install.packages("gert")OR remotes::install_github("r-lib/gert") |
@jeroen My bad! I've just seen your comment. The binaries were working for me before upgrading the kernel and other system libraries. So, it's a combination of CPU and compiler optimizations (CPU-specific instructions) as well as the kernel version and system dependencies that causes I think the system version of |
Works on some CPUs but not on othersI also get the Hopefully, this info helps narrow in on which CPU instruction we're talking about. CPUs where it worksIntel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz[c4-n16]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
Stepping: 4
CPU MHz: 3369.561
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4200.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 22528K
NUMA node0 CPU(s): 0-15,32-47
NUMA node1 CPU(s): 16-31,48-63
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_ppin intel_pt ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear spec_ctrl intel_stibp flush_l1dIntel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz[c4-n6]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 112
On-line CPU(s) list: 0-111
Thread(s) per core: 2
Core(s) per socket: 28
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6238R CPU @ 2.20GHz
Stepping: 7
CPU MHz: 999.963
CPU max MHz: 4000.0000
CPU min MHz: 1000.0000
BogoMIPS: 4400.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 39424K
NUMA node0 CPU(s): 0-27,56-83
NUMA node1 CPU(s): 28-55,84-111
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_ppin intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilitiesCPUs where it failsAMD Opteron(tm) Processor 6176[c4-dev2]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Thread(s) per core: 1
Core(s) per socket: 12
Socket(s): 4
NUMA node(s): 8
Vendor ID: AuthenticAMD
CPU family: 16
Model: 9
Model name: AMD Opteron(tm) Processor 6176
Stepping: 1
CPU MHz: 800.000
CPU max MHz: 2300.0000
CPU min MHz: 800.0000
BogoMIPS: 4600.27
Virtualization: AMD-V
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K
L3 cache: 5118K
NUMA node0 CPU(s): 0-5
NUMA node1 CPU(s): 6-11
NUMA node2 CPU(s): 12-17
NUMA node3 CPU(s): 18-23
NUMA node4 CPU(s): 24-29
NUMA node5 CPU(s): 30-35
NUMA node6 CPU(s): 36-41
NUMA node7 CPU(s): 42-47
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt nodeid_msr hw_pstate retpoline_amd ibp_disable vmmcall npt lbrv svm_lock nrip_save pausefilterSix-Core AMD Opteron(tm) Processor 2431[c4-dev3]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 1
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: AuthenticAMD
CPU family: 16
Model: 8
Model name: Six-Core AMD Opteron(tm) Processor 2431
Stepping: 0
CPU MHz: 2412.234
BogoMIPS: 4824.46
Virtualization: AMD-V
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K
L3 cache: 6144K
NUMA node0 CPU(s): 0-5
NUMA node1 CPU(s): 6-11
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc art rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt hw_pstate retpoline_amd ibp_disable vmmcall npt lbrv svm_lock nrip_save pausefilterAMD Opteron(tm) Processor 6282 SE[c4-n12]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 4
NUMA node(s): 8
Vendor ID: AuthenticAMD
CPU family: 21
Model: 1
Model name: AMD Opteron(tm) Processor 6282 SE
Stepping: 2
CPU MHz: 1400.000
CPU max MHz: 2600.0000
CPU min MHz: 1400.0000
BogoMIPS: 5199.94
Virtualization: AMD-V
L1d cache: 16K
L1i cache: 64K
L2 cache: 2048K
L3 cache: 6144K
NUMA node0 CPU(s): 0-7
NUMA node1 CPU(s): 8-15
NUMA node2 CPU(s): 16-23
NUMA node3 CPU(s): 24-31
NUMA node4 CPU(s): 32-39
NUMA node5 CPU(s): 40-47
NUMA node6 CPU(s): 48-55
NUMA node7 CPU(s): 56-63
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt fma4 nodeid_msr topoext perfctr_core perfctr_nb cpb hw_pstate retpoline_amd ssbd ibpb vmmcall arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthresholdIntel(R) Xeon(R) CPU X5690 @ 3.47GHzc4-n14 ~]$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 24
On-line CPU(s) list: 0-23
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 44
Model name: Intel(R) Xeon(R) CPU X5690 @ 3.47GHz
Stepping: 2
CPU MHz: 1600.000
CPU max MHz: 3468.0000
CPU min MHz: 1600.0000
BogoMIPS: 6933.67
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K
NUMA node0 CPU(s): 0-5,12-17
NUMA node1 CPU(s): 6-11,18-23
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt aes lahf_lm epb ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat spec_ctrl intel_stibp flush_l1dIntel(R) Xeon(R) CPU E5430 @ 2.66GHz[dev1] $ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 23
Model name: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
Stepping: 10
CPU MHz: 2003.000
CPU max MHz: 2670.0000
CPU min MHz: 2003.0000
BogoMIPS: 5320.42
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 6144K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf eagerfpu pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 xsave lahf_lm tpr_shadow vnmi flexpriority dtherm$ R --vanilla
R version 4.0.3 (2020-10-10) -- "Bunny-Wunnies Freak Out"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> install.packages("gert", repos="https://cloud.r-project.org")
Installing package into ‘/c4/home/henrik/R/x86_64-pc-linux-gnu-library/4.0-CBI’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/gert_1.1.0.tar.gz'
Content type 'application/x-gzip' length 67224 bytes (65 KB)
==================================================
downloaded 65 KB
* installing *source* package ‘gert’ ...
** package ‘gert’ successfully unpacked and MD5 sums checked
** using staged installation
> curl::curl_download("http://r-lib.github.io/gert/get-libgit2-linux.sh","get-libgit2-linux.sh")
>
>
> curl::curl_download('http://r-lib.github.io/gert/libgit2-1.1.0.x86_64_linux.tar.gz','bundle.tar.gz')
>
>
Using static libgit2-1.1.0 for Linux x86_64
Using PKG_CFLAGS=-DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include
Using PKG_LIBS=-L/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/lib -lgit2 -lrt -lpthread -lssh2 -lssl -lcrypto -ldl -lpcre -lz
Configuration OK!
** libs
rm -f gert.so branch.o clone.o commit.o config.o conflicts.o files.o init.o merge.o rebase.o stash.o submodules.o tag.o utils.o version.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c branch.c -o branch.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c clone.c -o clone.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c commit.c -o commit.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c config.c -o config.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c conflicts.c -o conflicts.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c files.c -o files.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c init.c -o init.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c merge.c -o merge.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c rebase.c -o rebase.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c stash.c -o stash.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c submodules.c -o submodules.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c tag.c -o tag.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c utils.c -o utils.o
gcc -std=gnu99 -I"/software/c4/cbi/software/R-4.0.3/lib64/R/include" -DNDEBUG -DSTATIC_LIBGIT2 -I/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/include -DR_NO_REMAP -DSTRICT_R_HEADERS -I/usr/local/include -fvisibility=hidden -fpic -g -O2 -c version.c -o version.o
gcc -std=gnu99 -shared -L/software/c4/cbi/software/R-4.0.3/lib64/R/lib -L/usr/local/lib64 -o gert.so branch.o clone.o commit.o config.o conflicts.o files.o init.o merge.o rebase.o stash.o submodules.o tag.o utils.o version.o -L/scratch/henrik/RtmpZJmUfH/R.INSTALL19cff5b29b5/gert/libgit2/lib -lgit2 -lrt -lpthread -lssh2 -lssl -lcrypto -ldl -lpcre -lz -L/software/c4/cbi/software/R-4.0.3/lib64/R/lib -lR
installing to /c4/home/henrik/R/x86_64-pc-linux-gnu-library/4.0-CBI/00LOCK-gert/00new/gert/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
sh: line 1: 6815 Illegal instruction R_TESTS= '/software/c4/cbi/software/R-4.0.3/lib64/R/bin/R' --no-save --no-restore --no-echo 2>&1 < '/scratch/henrik/RtmpZJmUfH/file19cf41d51c2b'
*** caught illegal operation ***
address 0x2b3a8b234ccd, cause 'illegal operand'
Traceback:
1: git_config_global()
2: doTryCatch(return(expr), name, parentenv, handler)
3: tryCatchOne(expr, names, parentenv, handlers[[1L]])
4: tryCatchList(expr, classes, parentenv, handlers)
5: tryCatch(expr, error = function(e) { call <- conditionCall(e) if (!is.null(call)) { if (identical(call[[1L]], quote(doTryCatch))) call <- sys.call(-4L) dcall <- deparse(call)[1L] prefix <- paste("Error in", dcall, ": ") LONG <- 75L sm <- strsplit(conditionMessage(e), "\n")[[1L]] w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w") if (is.na(w)) w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L], type = "b") if (w > LONG) prefix <- paste0(prefix, "\n ") } else prefix <- "Error : " msg <- paste0(prefix, conditionMessage(e), "\n") .Internal(seterrmessage(msg[1L])) if (!silent && isTRUE(getOption("show.error.messages"))) { cat(msg, file = outFile) .Internal(printDeferredWarnings()) } invisible(structure(msg, class = "try-error", condition = e))})
6: try({ settings <- git_config_global() name <- subset(settings, name == "user.name")$value email <- subset(settings, name == "user.email")$value if (length(name) || length(email)) { packageStartupInform("Default user: %s <%s>", as_string(name), as_string(email)) } else { packageStartupInform("No default user configured") }})
7: fun(libname, pkgname)
8: doTryCatch(return(expr), name, parentenv, handler)
9: tryCatchOne(expr, names, parentenv, handlers[[1L]])
10: tryCatchList(expr, classes, parentenv, handlers)
11: tryCatch(fun(libname, pkgname), error = identity)
12: runHook(".onAttach", ns, dirname(nspath), nsname)
13: attachNamespace(ns, pos = pos, deps, exclude, include.only)
14: doTryCatch(return(expr), name, parentenv, handler)
15: tryCatchOne(expr, names, parentenv, handlers[[1L]])
16: tryCatchList(expr, classes, parentenv, handlers)
17: tryCatch({ attr(package, "LibPath") <- which.lib.loc ns <- loadNamespace(package, lib.loc) env <- attachNamespace(ns, pos = pos, deps, exclude, include.only)}, error = function(e) { P <- if (!is.null(cc <- conditionCall(e))) paste(" in", deparse(cc)[1L]) else "" msg <- gettextf("package or namespace load failed for %s%s:\n %s", sQuote(package), P, conditionMessage(e)) if (logical.return) message(paste("Error:", msg), domain = NA) else stop(msg, call. = FALSE, domain = NA)})
18: library(pkg_name, lib.loc = lib, character.only = TRUE, logical.return = TRUE)
19: withCallingHandlers(expr, packageStartupMessage = function(c) tryInvokeRestart("muffleMessage"))
20: suppressPackageStartupMessages(library(pkg_name, lib.loc = lib, character.only = TRUE, logical.return = TRUE))
21: doTryCatch(return(expr), name, parentenv, handler)
22: tryCatchOne(expr, names, parentenv, handlers[[1L]])
23: tryCatchList(expr, classes, parentenv, handlers)
24: tryCatch(expr, error = function(e) { call <- conditionCall(e) if (!is.null(call)) { if (identical(call[[1L]], quote(doTryCatch))) call <- sys.call(-4L) dcall <- deparse(call)[1L] prefix <- paste("Error in", dcall, ": ") LONG <- 75L sm <- strsplit(conditionMessage(e), "\n")[[1L]] w <- 14L + nchar(dcall, type = "w") + nchar(sm[1L], type = "w") if (is.na(w)) w <- 14L + nchar(dcall, type = "b") + nchar(sm[1L], type = "b") if (w > LONG) prefix <- paste0(prefix, "\n ") } else prefix <- "Error : " msg <- paste0(prefix, conditionMessage(e), "\n") .Internal(seterrmessage(msg[1L])) if (!silent && isTRUE(getOption("show.error.messages"))) { cat(msg, file = outFile) .Internal(printDeferredWarnings()) } invisible(structure(msg, class = "try-error", condition = e))})
25: try(suppressPackageStartupMessages(library(pkg_name, lib.loc = lib, character.only = TRUE, logical.return = TRUE)))
26: tools:::.test_load_package("gert", "/c4/home/henrik/R/x86_64-pc-linux-gnu-library/4.0-CBI/00LOCK-gert/00new")
An irrecoverable exception occurred. R is aborting now ...
ERROR: loading failed
* removing ‘/c4/home/henrik/R/x86_64-pc-linux-gnu-library/4.0-CBI/gert’
* restoring previous ‘/c4/home/henrik/R/x86_64-pc-linux-gnu-library/4.0-CBI/gert’
The downloaded source packages are in
‘/scratch/henrik/Rtmps4SVaO/downloaded_packages’
Warning message:
In install.packages("gert", repos = "https://cloud.r-project.org") :
installation of package ‘gert’ had non-zero exit status
> Session info> sessionInfo()
R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS: /software/c4/cbi/software/R-4.0.3/lib64/R/lib/libRblas.so
LAPACK: /software/c4/cbi/software/R-4.0.3/lib64/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C LC_TIME=C
[4] LC_COLLATE=en_US.UTF-8 LC_MONETARY=C LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=C LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] compiler_4.0.3 tools_4.0.3 and $ gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.EDIT: Added another failing CPU (Intel(R) Xeon(R) CPU E5430 @ 2.66GHz). |
TroubleshootingBTW, for troubleshooting purposes one can force install gert 1.1.0 using: > install.packages("gert", repos = "https://cloud.r-project.org", INSTALL_opts="--no-test-load")With this, I can load the package but as we already know, it fails with [c4-n14]$ Rscript --vanilla -e "loadNamespace('gert')" -e "message('*******')" -e ''
<environment: namespace:gert>
*******
*** caught illegal operation ***
address 0x2ae97ccbfccd, cause 'illegal operand'
Traceback:
1: gert::git_config_global()
An irrecoverable exception occurred. R is aborting now ...
Illegal instructionStraceHere's the end of the [c4-n14]$ strace Rscript --quiet --vanilla -e 'gert::git_config_global()'
...
read(5, "", 4096) = 0
close(5) = 0
munmap(0x2ad428e9b000, 4096) = 0
access("/c4/home/henrik/.gitconfig", F_OK) = 0
stat("/c4/home/henrik/.gitconfig", {st_mode=S_IFREG|0644, st_size=116, ...}) = 0
access("/c4/home/henrik/.gitconfig", F_OK) = 0
access("/c4/home/henrik/.gitconfig", R_OK) = 0
stat("/c4/home/henrik/.gitconfig", {st_mode=S_IFREG|0644, st_size=116, ...}) = 0
stat("/c4/home/henrik/.gitconfig", {st_mode=S_IFREG|0644, st_size=116, ...}) = 0
open("/c4/home/henrik/.gitconfig", O_RDONLY|O_CLOEXEC) = 5
read(5, "[user]\n\temail = hb@aroma-project"..., 116) = 116
close(5) = 0
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPN, si_addr=0x2ad4385a6ccd} ---
write(2, "\n *** caught illegal operation *"..., 35
*** caught illegal operation ***
) = 35
write(2, "address 0x2ad4385a6ccd, cause 'i"..., 48address 0x2ad4385a6ccd, cause 'illegal operand'
) = 48
write(2, "\nTraceback:\n", 12
Traceback:
) = 12
write(2, " 1: ", 4 1: ) = 4
write(2, "gert::git_config_global()", 25gert::git_config_global()) = 25
write(2, "\n", 1
) = 1
write(2, "An irrecoverable exception occur"..., 59An irrecoverable exception occurred. R is aborting now ...
) = 59
rt_sigaction(SIGINT, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x2ad429dd9400}, {sa_handler=0x2ad42921a510, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x2ad429dd9400}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x2ad429dd9400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [ILL], 8) = 0
clone(child_stack=NULL, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x16da450) = 17840
wait4(17840, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 17840
rt_sigaction(SIGINT, {sa_handler=0x2ad42921a510, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x2ad429dd9400}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x2ad429dd9400}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [ILL], NULL, 8) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=17840, si_uid=581, si_status=0, si_utime=0, si_stime=0} ---
rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[ILL], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x2ad429dd9400}, {sa_handler=0x2ad42921a570, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_SIGINFO, sa_restorer=0x2ad429b96630}, 8) = 0
tgkill(17822, 17822, SIGILL) = 0
rt_sigreturn({mask=[]}) = 2562383102
--- SIGILL {si_signo=SIGILL, si_code=SI_TKILL, si_pid=17822, si_uid=581} ---
+++ killed by SIGILL +++
Illegal instructionWithout ~/.gitconfig[c4-n14 ~]$ mv ~/.gitconfig ~/.gitconfig.HIDE
[c4-n14 ~]$ Rscript --quiet --vanilla -e 'gert::git_config_global()'
[1] name value level
<0 rows> (or 0-length row.names)With empty ~/.gitconfig[c4-n14]$ touch ~/.gitconfig
[c4-n14]$ cat ~/.gitconfig
[c4-n14]$ ls -l ~/.gitconfig
-rw-r--r-- 1 henrik cbc 0 Jan 26 18:37 /c4/home/henrik/.gitconfig
[c4-n14]$ Rscript --quiet --vanilla -e 'gert::git_config_global()'
*** caught illegal operation ***
address 0x2b2b19c4accd, cause 'illegal operand'
Traceback:
1: gert::git_config_global()
An irrecoverable exception occurred. R is aborting now ...
Illegal instructionThe same with ...
open("/c4/home/henrik/.gitconfig", O_RDONLY|O_CLOEXEC) = 5
close(5) = 0
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPN, si_addr=0x2b4c71ca0ccd} ---
write(2, "\n *** caught illegal operation *"..., 35
*** caught illegal operation ***
) = 35
write(2, "address 0x2b4c71ca0ccd, cause 'i"..., 48address 0x2b4c71ca0ccd, cause 'illegal operand'
) = 48
write(2, "\nTraceback:\n", 12
Traceback:
) = 12
write(2, " 1: ", 4 1: ) = 4
write(2, "gert::git_config_global()", 25gert::git_config_global()) = 25
write(2, "\n", 1
) = 1
write(2, "An irrecoverable exception occur"..., 59An irrecoverable exception occurred. R is aborting now ...
) = 59
rt_sigaction(SIGINT, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x2b4c634d3400}, {sa_handler=0x2b4c62914510, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x2b4c634d3400}, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x2b4c634d3400}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [ILL], 8) = 0
clone(child_stack=NULL, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x1dbe450) = 17933
wait4(17933, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 17933
rt_sigaction(SIGINT, {sa_handler=0x2b4c62914510, sa_mask=[INT], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x2b4c634d3400}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x2b4c634d3400}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [ILL], NULL, 8) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=17933, si_uid=581, si_status=0, si_utime=0, si_stime=0} ---
rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[ILL], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x2b4c634d3400}, {sa_handler=0x2b4c62914570, sa_mask=[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_SIGINFO, sa_restorer=0x2b4c63290630}, 8) = 0
tgkill(17914, 17914, SIGILL) = 0
rt_sigreturn({mask=[]}) = 2562383102
--- SIGILL {si_signo=SIGILL, si_code=SI_TKILL, si_pid=17914, si_uid=581} ---
+++ killed by SIGILL +++
Illegal instructionHopefully this adds some more clues. |
|
Looking at
|
|
@HenrikBengtsson's older CPU reports |
|
@kevinushey I think that CPU-specific instructions are enabled by the compiler with optimization flags, which will cause similar issues when compiling on newer CPUs. Intel compilers explicitly print the used instructions if not supported by the machine at runtime. For example, using That's why I think if a static version should be shipped, it will have to use safe optimizations only or be built for old CPUs. |
HenrikBengtsson
added a commit
to HenrikBengtsson/CBI-software
that referenced
this issue
Jan 27, 2021
…th shipped static libgit2 library, cf. r-lib/gert#117
Personally, I wouldn't bother with the system libgit2 - at least not without a very explicit opt-in. System libgit2 will fall into one of two categories:
But they're also built for a wide variety of architectures / processors, and are optimized for a general amd64. Which means that they're no faster than what you could produce on your own (with the same targets and optimization settings). So the upside of using the system libgit2 is pretty limited.
👍 It could certainly also be vendored (which is what the rugged ruby bindings do) and built optimized for each target. This will bring certainly longer compile times but might be an interesting option. But, personally, I would ship a static library that is optimized for amd64 but has no special other optimizations. Where can I see the build for this static library? This feels like a place that libgit2 could improve - given that so many of our users come via language bindings like these, we should either default-to or have an easy opt-in to the right optimizations. |
|
Thanks all for the input. I had built the binaries using linuxbrew, which maybe buils with |
|
Aha - I think that Clang and GCC started adding more variants of x86-64 optimization recently. I think that dropping that back to In particular, avx2 is enabled by |
|
Oh sorry, I misread what you said - I thought that you said that you were building with Anyway, I'm fascinated to find out what's happening. 🍿 |
|
Hi all, I have rebuilt and redeployed the static library. Could you try again to see if the problem has disappeared? |
|
No. I was not aware |
|
@jan-glx are you still getting the crash? I just tested this with Intel's Software Development Emulator on sandybridge and mtm-core2 and that seems to work. I don't have any old hardware myself to debug this 😢 |
|
Seem to work for me, @jeroen : Install log$ MAKE='make -j3' Rscript -e 'devtools::install_github("r-lib/gert@debug")' Using github PAT from envvar GITHUB_PAT Downloading GitHub repo r-lib/gert@debug |
|
@mbojan and can you please try if the regular install works as well? install.packages("gert")And can you also test if connecting to SSH/HTTPS remotes works for you? library(gert)
git_clone("https://github.com/jeroen/curl")
# Only works if you have a registered ssh key with github
git_clone("[email protected]:jeroen/jsonlite")And if you really want to help me, can you test if gert passed all checks on your machine? curl -OL https://cran.r-project.org/src/contrib/gert_1.1.0.tar.gz
R CMD check gert_1.1.0.tar.gz |
|
My error is still there, but it wasn't about the avx instruction ( Anyway, (despite being a regular windows user 😅 ) I feel uncomfortable executing unsigned binaries downloaded from some (despite official) github repo on that machine. Wouldn't it be better to get explicit consent (e.g. by changing the default) and to put the static compilation into CI at least. I mean I trust you, but I doubt my sysadmins like that... |
|
@jan-glx your linker error is probably related to your use of LTO. Some people were complaining they found gert annoying to install because it requires them to install libgit2 which is outdated on many distros or users simply don't have permission to install libraries. So this is the compromise: if libgit2 0.28 or newer is available (the latest versions of Debian, Ubuntu, Fedora, etc), we use this. On old Linux x86_64 systems we try the static build, unless you opt-out by setting I think this will make the installation seamless for almost everyone (given that the binaries work 😅 ). Everyone always wants the default to match his personal preference and situation, and that's just not possible.
I think we're fine, as you may be aware the libgit2 lead maintainer is above you in this same topic. There are enough mirrors of the libgit2 code already but we can always add another one, if that would be required. |
I guess LTO refers to link time optimization? I don't know anything about that and didn't change anything there (tried unsetting the FLAGS in my Makevars but that did not change anything). Should I try anything else or open a separate issue?
I don't see anyone complaining about having to set an environment variable (or confirming a dialogue) to allow download and use of some binary lib. (yes, did see ethomson (and quite a few other familiar handles) above, sorry for the noise everyone) |
|
Success - I can verify that the new static library is compatible with all the nodes, including the ones that previously failed. Test detailsTo avoid avoid to re-install all dependencies on each test host, I prepoluate a temporary site package library: cd ~/test-gert
mkdir R-libs
USE_SYSTEM_LIBGIT2= R_LIBS_USER="~/test-gert/R-libs" Rscript --vanilla -e "install.packages('gert', repos='https://cloud.r-project.org')"
Installing package into ‘/c4/home/henrik/test-gert/R-libs’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/gert_1.1.0.tar.gz'
Content type 'application/x-gzip' length 67224 bytes (65 KB)
==================================================
downloaded 65 KB
* installing *source* package ‘gert’ ...
** package ‘gert’ successfully unpacked and MD5 sums checked
** using staged installation
> curl::curl_download("http://r-lib.github.io/gert/get-libgit2-linux.sh","get-libgit2-linux.sh")
>
>
Using static libgit2-1.1.0 for Linux x86_64
Using PKG_CFLAGS=-DSTATIC_LIBGIT2 -I/scratch/henrik/Rtmp63BAwf/R.INSTALL5a825fe58c55/gert/libgit2/include
Using PKG_LIBS=-L/scratch/henrik/Rtmp63BAwf/R.INSTALL5a825fe58c55/gert/libgit2/lib -lgit2 -lrt -lpthread -lssh2 -lssl -lcrypto -ldl -lpcre -lz
...
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (gert)
The downloaded source packages are in
‘/scratch/henrik/RtmpL8PxRp/downloaded_packages’Then, I test load this centrally installed 'gert' on each test host: cd
R_LIBS_USER="~/test-gert/R-libs" Rscript --vanilla -e "find.package(\"gert\")" -e "library(gert)"Practically, I use clush to test across multiple hosts concurrently, e.g. $ clush -b -w 'c4-dev[1-3],c4-n[1-16]' 'R_LIBS_USER="~/test-gert/R-libs" Rscript --vanilla -e "find.package(\"gert\")" -e "library(gert)" 2>&1'
---------------
c4-dev[1-3],c4-n[1-16] (19)
---------------
[1] "/c4/home/henrik/test-gert/R-libs/gert"
Linking to libgit2 v1.1.0, ssh support: YES
Global config: /c4/home/henrik/.gitconfig
Default user: Henrik <[email protected]> |
|
Thanks for the feedback everyone. I am closing this issue to prevent the discussion from getting conflated with other matters. If you run into any additional problems, please open a new issue. A short post-mortem of what happened: As described in the NEWS file on CRAN: on x86_64 Linux systems where libgit2 is too old or unavailable, we automatically try to download a precompiled static version of libgit2. This is mainly for users on CentOS-7 and Ubuntu 16/18 which complained that the system version of libgit2 is old, or they don't have permission to install system libraries, or they don't want to bother. Note that the current versions of Ubuntu, Fedora or Debian, etc (i.e. all cran plaforms) include a recent enough system version of libgit2 which will be used. The static library is just a fallback to make gert work seamlessly legacy linux systems. The bug: I had used linuxbrew to build the static libraries, which is a great platform to do exactly this. However, I made a small mistake: linuxbrew by default compiles with The solution: I have rebuilt and deployed new static libraries, and now |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I'm getting:
while
The text was updated successfully, but these errors were encountered: