Skip to content

Commit

Permalink
[SPARK-49986][INFRA] Restore scipy installation in dockerfile
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
Restore `scipy` installation in dockerfile

### Why are the changes needed?
https://docs.scipy.org/doc/scipy-1.13.1/building/index.html#system-level-dependencies

> If you want to use the system Python and pip, you will need:
 C, C++, and Fortran compilers (typically gcc, g++, and gfortran).
 ...

`scipy` actually depends on `gfortran`, but `apt-get remove --purge -y 'gfortran-11'` broke this dependency.

### Does this PR introduce _any_ user-facing change?
no

### How was this patch tested?
manually check with the first commit 5be0dfa:
move `apt-get remove --purge -y 'gfortran-11'` ahead of `scipy` installation, then the installation fails with
```
#18 394.3 Collecting scipy
#18 394.4   Downloading scipy-1.13.1.tar.gz (57.2 MB)
#18 395.2      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.2/57.2 MB 76.7 MB/s eta 0:00:00
#18 401.3   Installing build dependencies: started
#18 410.5   Installing build dependencies: finished with status 'done'
#18 410.5   Getting requirements to build wheel: started
#18 410.7   Getting requirements to build wheel: finished with status 'done'
#18 410.7   Installing backend dependencies: started
#18 411.8   Installing backend dependencies: finished with status 'done'
#18 411.8   Preparing metadata (pyproject.toml): started
#18 414.9   Preparing metadata (pyproject.toml): finished with status 'error'
#18 414.9   error: subprocess-exited-with-error
#18 414.9
#18 414.9   × Preparing metadata (pyproject.toml) did not run successfully.
#18 414.9   │ exit code: 1
#18 414.9   ╰─> [42 lines of output]
#18 414.9       + meson setup /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d/.mesonpy-xqfvs4ek -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d/.mesonpy-xqfvs4ek/meson-python-native-file.ini
#18 414.9       The Meson build system
#18 414.9       Version: 1.5.2
#18 414.9       Source dir: /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d
#18 414.9       Build dir: /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984415533ae9079d/.mesonpy-xqfvs4ek
#18 414.9       Build type: native build
#18 414.9       Project name: scipy
#18 414.9       Project version: 1.13.1
#18 414.9       C compiler for the host machine: cc (gcc 11.4.0 "cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
#18 414.9       C linker for the host machine: cc ld.bfd 2.38
#18 414.9       C++ compiler for the host machine: c++ (gcc 11.4.0 "c++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
#18 414.9       C++ linker for the host machine: c++ ld.bfd 2.38
#18 414.9       Cython compiler for the host machine: cython (cython 3.0.11)
#18 414.9       Host machine cpu family: x86_64
#18 414.9       Host machine cpu: x86_64
#18 414.9       Program python found: YES (/usr/local/bin/pypy3)
#18 414.9       Run-time dependency python found: YES 3.9
#18 414.9       Program cython found: YES (/tmp/pip-build-env-v_vnvt3h/overlay/bin/cython)
#18 414.9       Compiler for C supports arguments -Wno-unused-but-set-variable: YES
#18 414.9       Compiler for C supports arguments -Wno-unused-function: YES
#18 414.9       Compiler for C supports arguments -Wno-conversion: YES
#18 414.9       Compiler for C supports arguments -Wno-misleading-indentation: YES
#18 414.9       Library m found: YES
#18 414.9
#18 414.9       ../meson.build:78:0: ERROR: Unknown compiler(s): [['gfortran'], ['flang'], ['nvfortran'], ['pgfortran'], ['ifort'], ['ifx'], ['g95']]
#18 414.9       The following exception(s) were encountered:
#18 414.9       Running `gfortran --version` gave "[Errno 2] No such file or directory: 'gfortran'"
#18 414.9       Running `gfortran -V` gave "[Errno 2] No such file or directory: 'gfortran'"
#18 414.9       Running `flang --version` gave "[Errno 2] No such file or directory: 'flang'"
#18 414.9       Running `flang -V` gave "[Errno 2] No such file or directory: 'flang'"
#18 414.9       Running `nvfortran --version` gave "[Errno 2] No such file or directory: 'nvfortran'"
#18 414.9       Running `nvfortran -V` gave "[Errno 2] No such file or directory: 'nvfortran'"
#18 414.9       Running `pgfortran --version` gave "[Errno 2] No such file or directory: 'pgfortran'"
#18 414.9       Running `pgfortran -V` gave "[Errno 2] No such file or directory: 'pgfortran'"
#18 414.9       Running `ifort --version` gave "[Errno 2] No such file or directory: 'ifort'"
#18 414.9       Running `ifort -V` gave "[Errno 2] No such file or directory: 'ifort'"
#18 414.9       Running `ifx --version` gave "[Errno 2] No such file or directory: 'ifx'"
#18 414.9       Running `ifx -V` gave "[Errno 2] No such file or directory: 'ifx'"
#18 414.9       Running `g95 --version` gave "[Errno 2] No such file or directory: 'g95'"
#18 414.9       Running `g95 -V` gave "[Errno 2] No such file or directory: 'g95'"
#18 414.9
#18 414.9       A full log can be found at /tmp/pip-install-y77ar9d0/scipy_1e543e0816ed4b26984[4155](https://github.com/zhengruifeng/spark/actions/runs/11357130578/job/31589506939#step:7:4161)33ae9079d/.mesonpy-xqfvs4ek/meson-logs/meson-log.txt
#18 414.9       [end of output]
```

see https://github.com/zhengruifeng/spark/actions/runs/11357130578/job/31589506939

### Was this patch authored or co-authored using generative AI tooling?
no

Closes #48489 from zhengruifeng/infra_scipy.

Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
  • Loading branch information
zhengruifeng authored and HyukjinKwon committed Oct 17, 2024
1 parent e92bf37 commit 224d3ba
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion dev/infra/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,6 @@ RUN python3.13 -m pip install lxml numpy>=2.1 && \
python3.13 -m pip cache purge

# Remove unused installation packages to free up disk space
RUN apt-get remove --purge -y 'gfortran-11' 'humanity-icon-theme' 'nodejs-doc' || true
RUN apt-get remove --purge -y 'humanity-icon-theme' 'nodejs-doc'
RUN apt-get autoremove --purge -y
RUN apt-get clean

0 comments on commit 224d3ba

Please sign in to comment.