Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Minimal docker support #669

Merged
merged 1 commit into from
Feb 9, 2022
Merged

Minimal docker support #669

merged 1 commit into from
Feb 9, 2022

Conversation

dmp1ce
Copy link
Contributor

@dmp1ce dmp1ce commented Aug 17, 2020

This PR allows install.sh to run with a --docker-install option which installs JoinMarket without virtualenv or sudo. By not installing with virtualenv, it makes running scripts through Docker containers more straight forward. Sudo is also unnecessary because Docker containers run as root by default.

Also included is a minimal Dockerfile which can be used as a starting point for more Docker based JoinMarket services. Documentation is also provided explaining how to use the Dockerfile.

@kristapsk
Copy link
Member

Concept ACK

But does not work as is on a Gentoo Linux, as it requires --user option to pip if outside virtualenv.

ERROR: (Gentoo) Please run pip with the --user option to avoid breaking python-exec

So it's hard for me to test it properly.

@kristapsk
Copy link
Member

And you have typo in title - miminal vs minimal.

@dmp1ce dmp1ce changed the title Miminal docker support Minimal docker support Aug 21, 2020
@dmp1ce
Copy link
Contributor Author

dmp1ce commented Aug 21, 2020 •
edited
Loading

@kristapsk You could test within a Docker image. I tested with Debian, buster-slim image. Just run docker build -t joinmarket-test ./.

This PR isn't intended to install on Gentoo because it might mess with how Gentoo wants to install Python packages. In which case, it makes sense to use the default installation method, without the --docker-install flag. Are you testing with Gentoo installed in a Docker container?

Thank you for taking a look at the PR.

@dmp1ce
Copy link
Contributor Author

dmp1ce commented Aug 21, 2020 •
edited
Loading

The proper way to install on Gentoo is probably to create an ebuild which installs all the dependencies as expected for Gentoo. One would have to maintain it along side the install.sh script though. I made an attempt at something similar for Arch Linux, but it is probably out-of-date already. https://aur.archlinux.org/packages/?O=0&SeB=nd&K=joinmarket&outdated=&SB=n&SO=a&PP=50&do_Search=Go

@dmp1ce dmp1ce force-pushed the miminal-docker-support branch from e8e37c2 to 45296c0 Compare December 20, 2020 23:35
@dmp1ce dmp1ce force-pushed the miminal-docker-support branch from 45296c0 to 4a7a480 Compare November 18, 2021 23:24
@manasgandy
Copy link
Contributor

This PR might be useful for containerization of JoinMarket for Umbrel.

@dmp1ce
Copy link
Contributor Author

dmp1ce commented Dec 10, 2021

I couldn't say. I use this patch to build my own Dockerfile and use it in a Compose configuration I have. I've never used Umbrel.

https://github.com/dmp1ce/joinmarket-DOCKERFILE

@AdamISZ
Copy link
Member

AdamISZ commented Feb 7, 2022

@kristapsk as per above, apparently this is already in active use, is this mergeable, in some version?

@kristapsk
Copy link
Member

@AdamISZ Will look at this. Likely needs to be tested on different platforms as it has changes in install.sh which can potentially break things for non-Docker users too.

@AdamISZ
Copy link
Member

AdamISZ commented Feb 7, 2022

Agreed.

@AdamISZ
Copy link
Member

AdamISZ commented Feb 7, 2022

If it's easy to logically distinguish that using the with_jmvenv flag results in the identical script as before, then we can avoid checking that it still works on all environments it used to work on. I realise that's not 100% trivial because there are quite a few changes to install.sh, but it seems reasonable, at least.

kristapsk
kristapsk requested changes Feb 8, 2022
View reviewed changes
install.sh Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@kristapsk
Copy link
Member

Changes does not seem to break anything for non-Docker users (at least from my tests and code review). But when running docker build -t joinmarket-test ./ got at the end:

  Building wheels for collected packages: cffi
    Running setup.py bdist_wheel for cffi: started
    Running setup.py bdist_wheel for cffi: finished with status 'error'
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-nc50vy00/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-mayk69fy --python-tag cp37:
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    Package libffi was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libffi.pc'
    to the PKG_CONFIG_PATH environment variable
    No package 'libffi' found
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/error.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/api.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/recompiler.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/verifier.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/model.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/cparser.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/__init__.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/lock.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/commontypes.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_embedding.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.7/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/c
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
    c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
     #include <ffi.h>
              ^~~~~~~
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
    ----------------------------------------
    Failed building wheel for cffi
    Running setup.py clean for cffi
  Failed to build cffi
  Installing collected packages: setuptools, tomli, pyparsing, packaging, setuptools-scm, wheel, pycparser, cffi
    Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-nc50vy00/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-57u8vao8/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-n7smdagq --compile:
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      Package libffi was not found in the pkg-config search path.
      Perhaps you should add the directory containing `libffi.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'libffi' found
      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.7
      creating build/lib.linux-x86_64-3.7/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/error.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/api.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/recompiler.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/verifier.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/model.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/cparser.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/__init__.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/lock.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/commontypes.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_embedding.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.7/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-3.7
      creating build/temp.linux-x86_64-3.7/c
      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
      c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
       #include <ffi.h>
                ^~~~~~~
      compilation terminated.
      error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
      ----------------------------------------
  Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-nc50vy00/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-57u8vao8/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-n7smdagq --compile" failed with error code 1 in /tmp/pip-install-nc50vy00/cffi/
  
  ----------------------------------------
Command "/usr/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-n7smdagq --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=45 setuptools_scm>=6.2 wheel cffi>=1.0.1" failed with error code 1 in None
Joinmarket was not installed. Exiting.
The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends ca-certificates=* curl=*   python3-pip=*   && pip3 install 'wheel>=0.35.1'   && ./install.sh --docker-install   && apt-get clean   && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 1

@dmp1ce dmp1ce force-pushed the miminal-docker-support branch from 4a7a480 to 542b826 Compare February 8, 2022 21:36
@dmp1ce
Copy link
Contributor Author

dmp1ce commented Feb 8, 2022

I rebased to master and I didn't get any errors from running docker build -t joinmarket-test ./. I am running docker on Arch Linux. I'm not sure the host OS matters.

I think I got your documentation changes in my latest commit. I couldn't figure out how to git a .patch file from the Github comments so I eyeballed it.

@kristapsk kristapsk self-requested a review February 8, 2022 22:33
@kristapsk
Copy link
Member

Have the same error about ffi.h missing. Host OS should not matter as building happens inside Docker container. I'm currently running Docker version 20.10.9, build c2ea9bc90b under Gentoo Linux.

@dmp1ce
Copy link
Contributor Author

dmp1ce commented Feb 8, 2022

Did you get the latest upstream image? I used docker build --no-cache.

@kristapsk
Copy link
Member

Actual issue is some lines above:

skipping libffi-3.2.1...
skipping libsodium-1.0.18...

For some reason on my machine it thinks libffi and libsodium are already installed, so it ends up missing libffi on JM install (last) phase...

@dmp1ce
Copy link
Contributor Author

dmp1ce commented Feb 8, 2022 •
edited
Loading

Maybe in your build cache. That could be an issue with the Dockerfile setup actually. I should ignore files already built outside of Docker.

Adding the appropriate .dockerignore might fix the issue for you.

@dmp1ce
Copy link
Contributor Author

dmp1ce commented Feb 9, 2022

@kristapsk Can you try the build again with the .dockeringore I added?

I reproduced the error on my machine and fixed it with the .dockerignore.

@dmp1ce
Copy link
Contributor Author

dmp1ce commented Feb 9, 2022 •
edited
Loading

I noticed the Darwin install path was missing the checks I added for Debian install. I'm not sure it is even possible to create a Darwin image but the --docker-install flag should work the same as in Debian now.

For that matter, the --docker-install flag maybe should be called something different like --root-install. I'm not sure.

@kristapsk
Copy link
Member

Now it works also for me. ACK, I'm ready to merge this, but you should squash commits into one.

@dmp1ce dmp1ce force-pushed the miminal-docker-support branch from e620084 to c28bfd5 Compare February 9, 2022 13:48
@kristapsk kristapsk merged commit 930b2a0 into JoinMarket-Org:master Feb 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kristapsk kristapsk Awaiting requested review from kristapsk

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dmp1ce @kristapsk @manasgandy @AdamISZ