Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Non-deterministic behavior and segmentation fault when environment cannot be solved #286

Closed
vsoch opened this issue Mar 28, 2023 · 10 comments
Closed

Non-deterministic behavior and segmentation fault when environment cannot be solved #286

vsoch opened this issue Mar 28, 2023 · 10 comments

Comments

@vsoch
Copy link

vsoch commented Mar 28, 2023

I'm posting here because I cannot determine if this is related to the container setup or micromamba, but please feel free to transfer the issue!

I wanted to try out the container base today, and specifically have this Dockerfile:

ARG tag=latest
FROM ghcr.io/mamba-org/micromamba:${tag}

RUN micromamba install -c conda-forge python git && \
    git clone --depth 1 https://github.com/MLMI2-CSSI/foundry && \
    cd foundry && \
    pip install . && \
    micromamba install -c conda-forge matplotlib \
      scikit-learn \
      scikit-image \
      tensorflow \
      keras-unet \
      opencv-python \
      seaborn \
      tables \
      pandas && \
    pip install \
      pymatgen \
      matminer \

I'm working on it interactively, and for the second micromamba install command I just get a segmentation fault:

(base) mambauser@2fa61d316c50:/tmp/foundry$  micromamba install -c conda-forge matplotlib \
      scikit-learn \
      scikit-image \
      tensorflow \
      keras-unet \
      opencv-python \
      seaborn \
      tables \
      pandas

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Pinned packages:
  - python 3.11.*

Segmentation fault (core dumped)

I don't see anything about this in the FAQ - is there any way to debug this or am I doing something wrong? Adding --debug did not reveal any additional information. Thanks!
@wholtz
Copy link
Member

wholtz commented Mar 28, 2023

Hi @vsoch - thanks for the report and the interest in micromamba.

I don't get a seg fault when I follow your steps. It would be great if you had a Dockerfile that directly produced the seg fault. But I did run into some other issues. Should opencv-python be py-opencv? Conda forge also doesn't have a tables package -- should that be pytables?

I was able to get the following to build:

FROM ghcr.io/mamba-org/micromamba:1.4.1

ARG MAMBA_DOCKERFILE_ACTIVATE=1
RUN micromamba install -y -n base -c conda-forge python=3.10 git && \
    git clone --depth 1 https://github.com/MLMI2-CSSI/foundry && \
    cd foundry && \
    pip install . && \
    micromamba install -y -n base -c conda-forge matplotlib \
      scikit-learn \
      scikit-image \
      tensorflow \
      keras-unet \
      py-opencv \
      seaborn \
      pandas && \
    pip install \
      pymatgen \
      matminer \

@vsoch
Copy link
Author

vsoch commented Mar 28, 2023

I didn't get that far because the basic install didn't work! Let me try this again and make sure I give you the right commands I was running.

@vsoch
Copy link
Author

vsoch commented Mar 28, 2023

okay here is what segfaults:

ARG tag=latest
FROM ghcr.io/mamba-org/micromamba:${tag}

ARG MAMBA_DOCKERFILE_ACTIVATE=1
RUN micromamba install -c conda-forge python git && \
    git clone --depth 1 https://github.com/MLMI2-CSSI/foundry && \
    cd foundry && \
    pip install . && \
    micromamba install -c conda-forge matplotlib \
      scikit-learn \
      scikit-image \
      tensorflow \
      keras-unet \
      py-opencv \
      seaborn \
      pytables \
      pandas && \
    pip install \
      pymatgen \
      matminer
Build Output 
$ docker build  -f Dockerfile.mamba -t test .                 
[+] Building 75.8s (5/5) FINISHED                                                                                                       
 => [internal] load .dockerignore                                                                                                  0.1s 
 => => transferring context: 2B                                                                                                    0.0s 
 => [internal] load build definition from Dockerfile.mamba                                                                         0.1s 
 => => transferring dockerfile: 548B                                                                                               0.0s
 => [internal] load metadata for ghcr.io/mamba-org/micromamba:latest                                                               0.0s
 => CACHED [1/2] FROM ghcr.io/mamba-org/micromamba:latest                                                                          0.0s
 => ERROR [2/2] RUN micromamba install -c conda-forge python git &&     git clone --depth 1 https://github.com/MLMI2-CSSI/foundr  75.6s
------                                                                                                                                  
 > [2/2] RUN micromamba install -c conda-forge python git &&     git clone --depth 1 https://github.com/MLMI2-CSSI/foundry &&     cd foundry &&     pip install . &&     micromamba install -c conda-forge matplotlib       scikit-learn       scikit-image       tensorflow       keras-unet       py-opencv       seaborn       pytables       pandas &&     pip install       pymatgen       matminer:               
#0 0.313                                                                                                                                
#0 0.313                                            __                                                                                  
#0 0.313           __  ______ ___  ____ _____ ___  / /_  ____ _
#0 0.313          / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
#0 0.313         / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
#0 0.313        / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
#0 0.313       /_/
#0 0.313 
#0 19.89 
#0 19.89 Transaction
#0 19.89 
#0 19.89   Prefix: /opt/conda
#0 19.89 
#0 19.89   Updating specs:
#0 19.89 
#0 19.89    - python
#0 19.89    - git
#0 19.89 
#0 19.89 
#0 19.90   Package                  Version  Build               Channel                   Size
#0 19.90 ────────────────────────────────────────────────────────────────────────────────────────
#0 19.90   Install:
#0 19.90 ────────────────────────────────────────────────────────────────────────────────────────
#0 19.90 
#0 19.90   + _libgcc_mutex              0.1  conda_forge         conda-forge/linux-64       3kB
#0 19.90   + _openmp_mutex              4.5  2_gnu               conda-forge/linux-64      24kB
#0 19.90   + bzip2                    1.0.8  h7f98852_4          conda-forge/linux-64     496kB
#0 19.90   + c-ares                  1.18.1  h7f98852_0          conda-forge/linux-64     115kB
#0 19.90   + ca-certificates      2022.12.7  ha878542_0          conda-forge/linux-64     146kB
#0 19.90   + curl                     8.0.1  h588be90_0          conda-forge/linux-64      90kB
#0 19.90   + expat                    2.5.0  h27087fc_0          conda-forge/linux-64     194kB
#0 19.90   + gettext                 0.21.1  h27087fc_0          conda-forge/linux-64       4MB
#0 19.90   + git                     2.40.0  pl5321h3ca6407_1    conda-forge/linux-64       9MB
#0 19.90   + keyutils                 1.6.1  h166bdaf_0          conda-forge/linux-64     118kB
#0 19.90   + krb5                    1.20.1  h81ceb04_0          conda-forge/linux-64       1MB
#0 19.90   + ld_impl_linux-64          2.40  h41732ed_0          conda-forge/linux-64     705kB
#0 19.90   + libcurl                  8.0.1  h588be90_0          conda-forge/linux-64     361kB
#0 19.90   + libedit           3.1.20191231  he28a2e2_2          conda-forge/linux-64     124kB
#0 19.90   + libev                     4.33  h516909a_1          conda-forge/linux-64     106kB
#0 19.90   + libffi                   3.4.2  h7f98852_5          conda-forge/linux-64      58kB
#0 19.90   + libgcc-ng               12.2.0  h65d4601_19         conda-forge/linux-64     954kB
#0 19.90   + libgomp                 12.2.0  h65d4601_19         conda-forge/linux-64     466kB
#0 19.90   + libiconv                  1.17  h166bdaf_0          conda-forge/linux-64       1MB
#0 19.90   + libnghttp2              1.52.0  h61bc06f_0          conda-forge/linux-64     622kB
#0 19.90   + libnsl                   2.0.0  h7f98852_0          conda-forge/linux-64      31kB
#0 19.90   + libsqlite               3.40.0  h753d276_0          conda-forge/linux-64     810kB
#0 19.90   + libssh2                 1.10.0  hf14f497_3          conda-forge/linux-64     239kB
#0 19.90   + libstdcxx-ng            12.2.0  h46fd767_19         conda-forge/linux-64       4MB
#0 19.90   + libuuid                 2.32.1  h7f98852_1000       conda-forge/linux-64      28kB
#0 19.90   + libzlib                 1.2.13  h166bdaf_4          conda-forge/linux-64      66kB
#0 19.90   + ncurses                    6.3  h27087fc_1          conda-forge/linux-64       1MB
#0 19.90   + openssl                  3.1.0  h0b41bf4_0          conda-forge/linux-64       3MB
#0 19.90   + pcre2                    10.40  hc3806b6_0          conda-forge/linux-64       2MB
#0 19.90   + perl                    5.32.1  2_h7f98852_perl5    conda-forge/linux-64      15MB
#0 19.90   + pip                     23.0.1  pyhd8ed1ab_0        conda-forge/noarch         1MB
#0 19.90   + python                  3.11.0  he550d4f_1_cpython  conda-forge/linux-64      31MB
#0 19.90   + readline                   8.2  h8228510_1          conda-forge/linux-64     281kB
#0 19.90   + setuptools              67.6.1  pyhd8ed1ab_0        conda-forge/noarch       580kB
#0 19.90   + tk                      8.6.12  h27826a3_0          conda-forge/linux-64       3MB
#0 19.90   + tzdata                   2023b  h71feb2d_0          conda-forge/noarch       118kB
#0 19.90   + wheel                   0.40.0  pyhd8ed1ab_0        conda-forge/noarch        56kB
#0 19.90   + xz                       5.2.6  h166bdaf_0          conda-forge/linux-64     418kB
#0 19.90   + zstd                     1.5.2  h3eb15da_6          conda-forge/linux-64     420kB
#0 19.90 
#0 19.90   Summary:
#0 19.90 
#0 19.90   Install: 39 packages
#0 19.90 
#0 19.90   Total download: 86MB
#0 19.90 
#0 19.90 ────────────────────────────────────────────────────────────────────────────────────────
#0 19.90 
#0 19.90 
#0 19.90 Confirm changes: [Y/n] 
#0 19.90 Transaction starting
#0 30.43 Linking ld_impl_linux-64-2.40-h41732ed_0
#0 30.43 Linking _libgcc_mutex-0.1-conda_forge
#0 30.43 Linking ca-certificates-2022.12.7-ha878542_0
#0 30.43 Linking libstdcxx-ng-12.2.0-h46fd767_19
#0 30.43 Linking libgomp-12.2.0-h65d4601_19
#0 30.44 Linking _openmp_mutex-4.5-2_gnu
#0 30.44 Linking libgcc-ng-12.2.0-h65d4601_19
#0 30.45 Linking libuuid-2.32.1-h7f98852_1000
#0 30.45 Linking libev-4.33-h516909a_1
#0 30.45 Linking gettext-0.21.1-h27087fc_0
#0 30.59 Linking c-ares-1.18.1-h7f98852_0
#0 30.60 Linking keyutils-1.6.1-h166bdaf_0
#0 30.61 Linking ncurses-6.3-h27087fc_1
#0 32.12 Linking libffi-3.4.2-h7f98852_5
#0 32.13 Linking bzip2-1.0.8-h7f98852_4
#0 32.13 Linking openssl-3.1.0-h0b41bf4_0
#0 32.16 Linking libiconv-1.17-h166bdaf_0
#0 32.16 Linking expat-2.5.0-h27087fc_0
#0 32.17 Linking xz-5.2.6-h166bdaf_0
#0 32.18 Linking libzlib-1.2.13-h166bdaf_4
#0 32.18 Linking libnsl-2.0.0-h7f98852_0
#0 32.18 Linking libedit-3.1.20191231-he28a2e2_2
#0 32.18 Linking readline-8.2-h8228510_1
#0 32.18 Linking zstd-1.5.2-h3eb15da_6
#0 32.19 Linking libssh2-1.10.0-hf14f497_3
#0 32.20 Linking libnghttp2-1.52.0-h61bc06f_0
#0 32.20 Linking pcre2-10.40-hc3806b6_0
#0 32.21 Linking tk-8.6.12-h27826a3_0
#0 32.25 Linking libsqlite-3.40.0-h753d276_0
#0 32.25 Linking perl-5.32.1-2_h7f98852_perl5
#0 32.51 Linking krb5-1.20.1-h81ceb04_0
#0 32.53 Linking libcurl-8.0.1-h588be90_0
#0 32.53 Linking curl-8.0.1-h588be90_0
#0 32.54 Linking git-2.40.0-pl5321h3ca6407_1
#0 32.77 Linking tzdata-2023b-h71feb2d_0
#0 32.82 Linking python-3.11.0-he550d4f_1_cpython
#0 33.14 Linking wheel-0.40.0-pyhd8ed1ab_0
#0 33.26 Linking setuptools-67.6.1-pyhd8ed1ab_0
#0 33.27 Linking pip-23.0.1-pyhd8ed1ab_0
#0 33.70 Transaction finished
#0 33.73 Cloning into 'foundry'...
#0 40.31 Processing /tmp/foundry
#0 40.32   Preparing metadata (setup.py): started
#0 40.45   Preparing metadata (setup.py): finished with status 'done'
#0 40.61 Collecting mdf_forge>=0.8.0
#0 40.67   Downloading mdf_forge-0.8.0-py2.py3-none-any.whl (17 kB)
#0 40.81 Collecting globus-sdk<4,>=3
#0 40.82   Downloading globus_sdk-3.18.0-py3-none-any.whl (216 kB)
#0 40.85      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 216.1/216.1 kB 8.2 MB/s eta 0:00:00
#0 40.88 Collecting dlhub_sdk>=1.0.0
#0 40.89   Downloading dlhub_sdk-1.1.1-py2.py3-none-any.whl (51 kB)
#0 40.90      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.1/51.1 kB 8.0 MB/s eta 0:00:00
#0 41.18 Collecting numpy>=1.15.4
#0 41.19   Downloading numpy-1.24.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
#0 42.42      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 14.4 MB/s eta 0:00:00
#0 42.68 Collecting pandas>=0.23.4
#0 42.69   Downloading pandas-1.5.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.0 MB)
#0 43.35      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 19.0 MB/s eta 0:00:00
#0 43.53 Collecting pydantic>=1.4
#0 43.54   Downloading pydantic-1.10.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
#0 43.71      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 17.7 MB/s eta 0:00:00
#0 43.75 Collecting mdf_connect_client>=0.4.0
#0 43.76   Downloading mdf_connect_client-0.4.0-py2.py3-none-any.whl (19 kB)
#0 43.83 Collecting h5py>=2.10.0
#0 43.84   Downloading h5py-3.8.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
#0 44.04      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 23.6 MB/s eta 0:00:00
#0 44.14 Collecting json2table
#0 44.14   Downloading json2table-1.1.5-py2.py3-none-any.whl (8.7 kB)
#0 44.20 Collecting requests>=2.24.0
#0 44.20   Downloading requests-2.28.2-py3-none-any.whl (62 kB)
#0 44.21      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 17.4 MB/s eta 0:00:00
#0 44.24 Collecting mdf-toolbox>=0.5.4
#0 44.25   Downloading mdf_toolbox-0.6.0-py2.py3-none-any.whl (35 kB)
#0 44.30 Collecting jsonschema>=3.2.0
#0 44.30   Downloading jsonschema-4.17.3-py3-none-any.whl (90 kB)
#0 44.31      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.4/90.4 kB 22.1 MB/s eta 0:00:00
#0 44.35 Collecting funcx>=1.0.0
#0 44.35   Downloading funcx-1.0.12-py3-none-any.whl (51 kB)
#0 44.36      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.7/51.7 kB 16.2 MB/s eta 0:00:00
#0 44.41 Collecting pyjwt[crypto]<3.0.0,>=2.0.0
#0 44.42   Downloading PyJWT-2.6.0-py3-none-any.whl (20 kB)
#0 44.67 Collecting cryptography!=3.4.0,>=3.3.1
#0 44.68   Downloading cryptography-40.0.1-cp36-abi3-manylinux_2_28_x86_64.whl (3.7 MB)
#0 44.86      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.7/3.7 MB 21.5 MB/s eta 0:00:00
#0 44.99 Collecting nameparser>=1.0.4
#0 45.00   Downloading nameparser-1.1.2-py2.py3-none-any.whl (24 kB)
#0 45.10 Collecting tqdm>=4.19.4
#0 45.11   Downloading tqdm-4.65.0-py3-none-any.whl (77 kB)
#0 45.11      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 kB 18.1 MB/s eta 0:00:00
#0 45.14 Collecting python-dateutil>=2.8.1
#0 45.15   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
#0 45.17      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 24.0 MB/s eta 0:00:00
#0 45.25 Collecting pytz>=2020.1
#0 45.26   Downloading pytz-2023.2-py2.py3-none-any.whl (502 kB)
#0 45.29      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 502.1/502.1 kB 21.1 MB/s eta 0:00:00
#0 45.33 Collecting typing-extensions>=4.2.0
#0 45.34   Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
#0 45.52 Collecting cffi>=1.12
#0 45.53   Downloading cffi-1.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (462 kB)
#0 45.57      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 462.6/462.6 kB 14.1 MB/s eta 0:00:00
#0 45.69 Collecting websockets==10.3
#0 45.70   Downloading websockets-10.3.tar.gz (84 kB)
#0 45.71      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.1/84.1 kB 18.5 MB/s eta 0:00:00
#0 45.73   Preparing metadata (setup.py): started
#0 45.86   Preparing metadata (setup.py): finished with status 'done'
#0 45.88 Collecting dill==0.3.5.1
#0 45.89   Downloading dill-0.3.5.1-py2.py3-none-any.whl (95 kB)
#0 45.90      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.8/95.8 kB 22.0 MB/s eta 0:00:00
#0 45.93 Collecting packaging>=21.1
#0 45.94   Downloading packaging-23.0-py3-none-any.whl (42 kB)
#0 45.95      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 kB 19.6 MB/s eta 0:00:00
#0 45.97 Collecting pika>=1.2
#0 45.98   Downloading pika-1.3.1-py3-none-any.whl (155 kB)
#0 45.99      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 18.4 MB/s eta 0:00:00
#0 46.09 Collecting funcx-common==0.0.25
#0 46.10   Downloading funcx_common-0.0.25-py3-none-any.whl (29 kB)
#0 46.12 Collecting tblib==1.7.0
#0 46.13   Downloading tblib-1.7.0-py2.py3-none-any.whl (12 kB)
#0 46.15 Collecting texttable>=1.6.7
#0 46.16   Downloading texttable-1.6.7-py2.py3-none-any.whl (10 kB)
#0 46.22 Collecting attrs>=17.4.0
#0 46.23   Downloading attrs-22.2.0-py3-none-any.whl (60 kB)
#0 46.24      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.0/60.0 kB 8.1 MB/s eta 0:00:00
#0 46.33 Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
#0 46.34   Downloading pyrsistent-0.19.3-py3-none-any.whl (57 kB)
#0 46.35      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.5/57.5 kB 13.3 MB/s eta 0:00:00
#0 46.45 Collecting fair-research-login>=0.2.4
#0 46.46   Downloading fair_research_login-0.3.0-py3-none-any.whl (27 kB)
#0 46.49 Collecting globus-nexus-client>=0.4.1
#0 46.49   Downloading globus_nexus_client-0.4.1-py3-none-any.whl (9.2 kB)
#0 46.52 Collecting jsonschema>=3.2.0
#0 46.53   Downloading jsonschema-4.3.0-py3-none-any.whl (71 kB)
#0 46.53      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.6/71.6 kB 21.7 MB/s eta 0:00:00
#0 46.65 Collecting six>=1.5
#0 46.66   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
#0 46.77 Collecting charset-normalizer<4,>=2
#0 46.77   Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (197 kB)
#0 46.78      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 197.3/197.3 kB 33.0 MB/s eta 0:00:00
#0 46.80 Collecting idna<4,>=2.5
#0 46.81   Downloading idna-3.4-py3-none-any.whl (61 kB)
#0 46.82      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 9.6 MB/s eta 0:00:00
#0 46.89 Collecting urllib3<1.27,>=1.21.1
#0 46.90   Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
#0 46.90      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 kB 22.2 MB/s eta 0:00:00
#0 46.94 Collecting certifi>=2017.4.17
#0 46.95   Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
#0 46.96      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 20.5 MB/s eta 0:00:00
#0 47.03 Collecting pycparser
#0 47.04   Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
#0 47.05      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 21.4 MB/s eta 0:00:00
#0 47.13 Building wheels for collected packages: foundry-ml, websockets
#0 47.13   Building wheel for foundry-ml (setup.py): started
#0 47.32   Building wheel for foundry-ml (setup.py): finished with status 'done'
#0 47.32   Created wheel for foundry-ml: filename=foundry_ml-0.5.1-py3-none-any.whl size=19673 sha256=8a919dd4487483bdeaee6af28925237d22b93fd7d2261d56be3bc5f56420a6e8
#0 47.32   Stored in directory: /tmp/pip-ephem-wheel-cache-wp7jpwhg/wheels/7f/aa/f2/9847f2f32bd7464955ae0117c797b70b269edaf20610f3f07e
#0 47.32   Building wheel for websockets (setup.py): started
#0 47.51   Building wheel for websockets (setup.py): finished with status 'done'
#0 47.51   Created wheel for websockets: filename=websockets-10.3-cp311-cp311-linux_x86_64.whl size=94361 sha256=bcb6e84a7da71ea9d89f1ebb5327208372e3e927583057c3cc108f4eb5f10eeb
#0 47.51   Stored in directory: /home/mambauser/.cache/pip/wheels/96/6c/0a/ef3325d86b9a8f81580dd332d9ff195e71065f10e3008927a8
#0 47.52 Successfully built foundry-ml websockets
#0 47.73 Installing collected packages: texttable, pytz, nameparser, json2table, websockets, urllib3, typing-extensions, tqdm, tblib, six, pyrsistent, pyjwt, pycparser, pika, packaging, numpy, idna, dill, charset-normalizer, certifi, attrs, requests, python-dateutil, pydantic, jsonschema, h5py, cffi, pandas, funcx-common, cryptography, globus-sdk, globus-nexus-client, funcx, fair-research-login, mdf-toolbox, mdf_forge, mdf_connect_client, dlhub_sdk, foundry-ml
#0 55.39 Successfully installed attrs-22.2.0 certifi-2022.12.7 cffi-1.15.1 charset-normalizer-3.1.0 cryptography-40.0.1 dill-0.3.5.1 dlhub_sdk-1.1.1 fair-research-login-0.3.0 foundry-ml-0.5.1 funcx-1.0.12 funcx-common-0.0.25 globus-nexus-client-0.4.1 globus-sdk-3.18.0 h5py-3.8.0 idna-3.4 json2table-1.1.5 jsonschema-4.3.0 mdf-toolbox-0.6.0 mdf_connect_client-0.4.0 mdf_forge-0.8.0 nameparser-1.1.2 numpy-1.24.2 packaging-23.0 pandas-1.5.3 pika-1.3.1 pycparser-2.21 pydantic-1.10.7 pyjwt-2.6.0 pyrsistent-0.19.3 python-dateutil-2.8.2 pytz-2023.2 requests-2.28.2 six-1.16.0 tblib-1.7.0 texttable-1.6.7 tqdm-4.65.0 typing-extensions-4.5.0 urllib3-1.26.15 websockets-10.3
#0 55.79 
#0 55.79                                            __
#0 55.79           __  ______ ___  ____ _____ ___  / /_  ____ _
#0 55.79          / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
#0 55.79         / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
#0 55.79        / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
#0 55.79       /_/
#0 55.79 
#0 55.79 conda-forge/linux-64                                        Using cache
#0 55.79 conda-forge/noarch                                          Using cache
#0 56.34 
#0 56.34 Pinned packages:
#0 56.34   - python 3.11.*
#0 56.34 
#0 75.59 bash: line 1:   206 Segmentation fault      (core dumped) micromamba install -c conda-forge matplotlib scikit-learn scikit-image tensorflow keras-unet py-opencv seaborn pytables pandas
------
Dockerfile.mamba:5
--------------------
   4 |     ARG MAMBA_DOCKERFILE_ACTIVATE=1
   5 | >>> RUN micromamba install -c conda-forge python git && \
   6 | >>>     git clone --depth 1 https://github.com/MLMI2-CSSI/foundry && \
   7 | >>>     cd foundry && \
   8 | >>>     pip install . && \
   9 | >>>     micromamba install -c conda-forge matplotlib \
  10 | >>>       scikit-learn \
  11 | >>>       scikit-image \
  12 | >>>       tensorflow \
  13 | >>>       keras-unet \
  14 | >>>       py-opencv \
  15 | >>>       seaborn \
  16 | >>>       pytables \
  17 | >>>       pandas && \
  18 | >>>     pip install \
  19 | >>>       pymatgen \
  20 | >>>       matminer
  21 |     
--------------------
ERROR: failed to solve: process "/usr/local/bin/_dockerfile_shell.sh micromamba install -c conda-forge python git &&     git clone --depth 1 https://github.com/MLMI2-CSSI/foundry &&     cd foundry &&     pip install . &&     micromamba install -c conda-forge matplotlib       scikit-learn       scikit-image       tensorflow       keras-unet       py-opencv       seaborn       pytables       pandas &&     pip install       pymatgen       matminer" did not complete successfully: exit code: 139

I'll try your tag first, then the entire thing.

@vsoch
Copy link
Author

vsoch commented Mar 28, 2023 •
edited
Loading

Okay update:

  • I tried my Dockerfile with your pinned version above, same segfault
  • I tried your Dockerfile (minus the extra line escape)
FROM ghcr.io/mamba-org/micromamba:1.4.1

ARG MAMBA_DOCKERFILE_ACTIVATE=1
RUN micromamba install -y -n base -c conda-forge python=3.10 git && \
    git clone --depth 1 https://github.com/MLMI2-CSSI/foundry && \
    cd foundry && \
    pip install . && \
    micromamba install -y -n base -c conda-forge matplotlib \
      scikit-learn \
      scikit-image \
      tensorflow \
      keras-unet \
      py-opencv \
      seaborn \
      pandas && \
    pip install \
      pymatgen \
      matminer

Your Dockerfile worked (at least it hasn't failed yet... 😆 ) The main difference is the python version - could this be related to python 3.11?

@wholtz
Copy link
Member

wholtz commented Mar 29, 2023

I was able to get the seg faulting Dockerfile minimized down to:

FROM ghcr.io/mamba-org/micromamba:1.4.1

RUN micromamba install -y -n base -c conda-forge python=3.11.0
RUN micromamba install -y -n base -c conda-forge \
      keras-unet=0.1.2 && true

It has to be python 3.11 and it needs to end with && some_command

It is probably related to being unable to solve this:

$ micromamba install -y -n base -c conda-forge python=3.11.0 keras-unet=0.1.2
error    libmamba Could not solve for environment specs
    The following packages are incompatible
    ├─ keras-unet 0.1.2**  is installable and it requires
    │  └─ tensorflow   with the potential options
    │     ├─ tensorflow [0.10.0|0.11.0|...|1.9.0] would require
    │     │  ├─ python [2.7* |>=2.7,<2.8.0a0 ], which can be installed;
    │     │  └─ tensorflow-base [1.13.1 py27_5|1.13.1 py27_7], which requires
    │     │     └─ python >=2.7,<2.8.0a0 , which can be installed;
    │     ├─ tensorflow [0.10.0|0.11.0|...|0.9.0] would require
    │     │  └─ python 3.4* , which can be installed;
    │     ├─ tensorflow [0.10.0|0.11.0|...|1.9.0] would require
    │     │  └─ python [3.5* |>=3.5,<3.6.0a0 ], which can be installed;
    │     ├─ tensorflow [0.12.1|1.0.0|...|1.5.1] would require
    │     │  └─ python 3.6* , which can be installed;
    │     ├─ tensorflow [1.10.0|1.13.1|...|1.9.0] would require
    │     │  ├─ python >=3.6,<3.7.0a0 , which can be installed;
    │     │  └─ tensorflow-base 1.13.2 py36_0, which requires
    │     │     └─ python >=3.6,<3.7.0a0 , which can be installed;
    │     ├─ tensorflow 1.13.1 would require
    │     │  ├─ python >=3.7,<3.8.0a0 , which can be installed;
    │     │  └─ tensorflow-base [1.13.1 py37_4|1.13.1 py37_6|1.13.1 py37_8], which requires
    │     │     └─ python >=3.7,<3.8.0a0 , which can be installed;
    │     ├─ tensorflow [1.14.0|2.10.0|...|2.9.1] would require
    │     │  └─ python_abi 3.7.* *_cp37m, which can be installed;
    │     ├─ tensorflow [1.14.0|2.4.0|2.4.1|2.4.3|2.6.0] would require
    │     │  └─ python_abi 3.6.* *_cp36m, which can be installed;
    │     ├─ tensorflow [2.10.0|2.11.0|...|2.9.1] would require
    │     │  └─ python_abi 3.10.* *_cp310, which can be installed;
    │     ├─ tensorflow [2.10.0|2.11.0|...|2.9.1] would require
    │     │  └─ python_abi 3.8.* *_cp38, which can be installed;
    │     └─ tensorflow [2.10.0|2.11.0|...|2.9.1] would require
    │        └─ python_abi 3.9.* *_cp39, which can be installed;
    └─ python 3.11.0**  is uninstallable because it requires
       └─ python_abi 3.11.* *_cp311, which conflicts with any installable versions previously reported.
critical libmamba Could not solve for environment specs

And then I was able to get the seg fault to occur without using docker:

#!/bin/bash

eval "$(micromamba shell hook --shell=bash)"
micromamba create -y -n issue286
micromamba activate issue286
micromamba install -y -c conda-forge python=3.11.0
micromamba install -y -c conda-forge keras-unet=0.1.2 && true

But oddly enough, the seg fault without docker only occurs ~5% of the time. The non-seg fault error messages from that script are oddly not deterministic either....

That's all I have time for right now. I'd like to find a way to always get the seg fault without using docker.....

@vsoch
Copy link
Author

vsoch commented Mar 29, 2023

I guess we found an interesting bug then! 😆 Sorry I didn't mean to inflict extra work or pain. I am good to use a different version of Python, and if you don't want to work on this we can just leave it open (and see if others run into segfaults in other contexts). It indeed is worrisome that it only happens a small percentage of the time locally. Were there any subtle differences between the local runs - libraries on the system or environment variables?

@wholtz
Copy link
Member

wholtz commented Mar 29, 2023

Nothing was changing, they were back-to-back executions. Now I explicitly put it in a loop and logged the output:

#!/bin/bash

set -eo pipefail

eval "$(micromamba shell hook --shell=bash)"
for i in $(seq 1 100); do
  micromamba create -y -n issue286
  micromamba activate issue286
  micromamba install -y -c conda-forge python=3.11.0
  micromamba install -y -c conda-forge keras-unet=0.1.2 && true
  micromamba env remove -n issue286 -y || true
done

Results in:
2x seg faults
13x "critical libmamba Could not solve for environment specs"
85x "critical libmamba Invalid spec, no package name found: "

I have no idea why I am seeing non-deterministic behavior. I'm going to move this over to the mamba issue tracker. But I'm not expecting this to get resolved until it can be deterministically triggered without involving docker.

@wholtz wholtz changed the title Segmentation fault in docker latest container Non-deterministic behavior and segmentation fault when environment cannot be solved Mar 29, 2023
@maresb
Copy link
Collaborator

maresb commented Mar 29, 2023

Wow, this is a fairly crazy one!

I think with @wholtz's work that this is reproducible enough. I think the Mamba folks will be able to make a debug build and get a stack trace with this info.

@vsoch
Copy link
Author

vsoch commented Mar 29, 2023

Nice! I found a good one! 😆 Thanks for the help and triaging the issue - if it gets out of sync here can you update me (or if the issue tracker is another GitHub issue can you ping me there?) I am interested too.

@wholtz wholtz mentioned this issue Mar 29, 2023
Closed
2 tasks
@wholtz
Copy link
Member

wholtz commented Apr 11, 2023

My testing indicates this was fixed by mamba-org/mamba#2428 - please let me know if anyone believes this needs to be reopened.

@wholtz wholtz closed this as completed Apr 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vsoch @wholtz @maresb