Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pre-commit not compatible with hooks that require compiled dependencies (e.g. ansible-lint) #95

Open
m-roberts opened this issue Apr 8, 2021 · 2 comments
Open

pre-commit not compatible with hooks that require compiled dependencies (e.g. ansible-lint) #95

m-roberts opened this issue Apr 8, 2021 · 2 comments

Comments

@m-roberts
Copy link

When trying to use this config:

repos:
  - repo: https://github.com/ansible-community/ansible-lint.git
    rev: v5.0.7
    hooks:
      - id: ansible-lint
        files: \.(yaml|yml)$

I get this:

[INFO] Initializing environment for https://github.com/ansible-community/ansible-lint.git.
[INFO] Initializing environment for https://github.com/ansible-community/ansible-lint.git:.[community,yamllint].
[INFO] Installing environment for https://github.com/ansible-community/ansible-lint.git.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/root/.cache/pre-commit/reponmshav7x/py_env-python3.9/bin/python', '-mpip', 'install', '.', '.[community,yamllint]')
return code: 1
expected return code: 0
stdout:
    Processing /root/.cache/pre-commit/reponmshav7x
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Processing /root/.cache/pre-commit/reponmshav7x
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Collecting packaging
      Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
    Collecting ruamel.yaml<1,>=0.15.37; python_version >= "3.7"
      Using cached ruamel.yaml-0.17.4-py3-none-any.whl (101 kB)
    Collecting pyyaml
      Downloading PyYAML-5.4.1.tar.gz (175 kB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
        Preparing wheel metadata: started
        Preparing wheel metadata: finished with status 'done'
    Collecting tenacity
      Downloading tenacity-7.0.0-py2.py3-none-any.whl (23 kB)
    Collecting enrich>=1.2.6
      Downloading enrich-1.2.6-py3-none-any.whl (8.6 kB)
    Collecting rich>=9.5.1
      Downloading rich-10.1.0-py3-none-any.whl (201 kB)
    Collecting wcmatch>=7.0
      Downloading wcmatch-8.1.2-py3-none-any.whl (40 kB)
    Collecting ansible>=2.10; extra == "community"
      Downloading ansible-3.2.0.tar.gz (31.3 MB)
    Collecting yamllint>=1.25.0; extra == "yamllint"
      Downloading yamllint-1.26.1.tar.gz (126 kB)
    Collecting pyparsing>=2.0.2
      Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
    Processing /root/.cache/pip/wheels/4d/ec/94/94b660639d0a3d73c59770481208fb98c1d8dd38c87d246698/ruamel.yaml.clib-0.2.2-py3-none-any.whl
    Collecting six>=1.9.0
      Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
    Collecting pygments<3.0.0,>=2.6.0
      Downloading Pygments-2.8.1-py3-none-any.whl (983 kB)
    Collecting colorama<0.5.0,>=0.4.0
      Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)
    Collecting commonmark<0.10.0,>=0.9.0
      Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB)
    Collecting typing-extensions<4.0.0,>=3.7.4
      Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
    Collecting bracex>=2.1.1
      Downloading bracex-2.1.1-py3-none-any.whl (10 kB)
    Collecting ansible-base<2.11,>=2.10.7
      Downloading ansible-base-2.10.7.tar.gz (5.7 MB)
    Collecting pathspec>=0.5.3
      Downloading pathspec-0.8.1-py2.py3-none-any.whl (28 kB)
    Collecting jinja2
      Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
    Collecting cryptography
      Downloading cryptography-3.4.7.tar.gz (546 kB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'error'

stderr:
      ERROR: Command errored out with exit status 1:
       command: /root/.cache/pre-commit/reponmshav7x/py_env-python3.9/bin/python /root/.cache/pre-commit/reponmshav7x/py_env-python3.9/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-u7scfaiu/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"'' 'setuptools-rust>=0.11.4'
           cwd: None
      Complete output (124 lines):
      Collecting setuptools>=40.6.0
        Using cached setuptools-54.2.0-py3-none-any.whl (785 kB)
      Collecting wheel
        Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
      Collecting cffi>=1.12
        Downloading cffi-1.14.5.tar.gz (475 kB)
      Collecting setuptools-rust>=0.11.4
        Downloading setuptools_rust-0.12.1-py3-none-any.whl (22 kB)
      Collecting pycparser
        Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
      Collecting toml>=0.9.0
        Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
      Collecting semantic-version>=2.6.0
        Downloading semantic_version-2.8.5-py2.py3-none-any.whl (15 kB)
      Building wheels for collected packages: cffi
        Building wheel for cffi (setup.py): started
        Building wheel for cffi (setup.py): finished with status 'error'
        ERROR: Command errored out with exit status 1:
         command: /root/.cache/pre-commit/reponmshav7x/py_env-python3.9/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_1btoeh2/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_1btoeh2/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-0nu13nh7
             cwd: /tmp/pip-install-_1btoeh2/cffi/
        Complete output (45 lines):

            No working compiler found, or bogus compiler options passed to
            the compiler from Python's standard "distutils" module.  See
            the error messages above.  Likely, the problem is not related
            to CFFI but generic to the setup.py of any Python package that
            tries to compile C code.  (Hints: on OS/X 10.8, for errors about
            -mno-fused-madd see http://stackoverflow.com/questions/22313407/
            Otherwise, see https://wiki.python.org/moin/CompLangPython or
            the IRC channel #python on irc.freenode.net.)

            Trying to continue anyway.  If you are trying to install CFFI from
            a build done in a different context, you can ignore this warning.

        running bdist_wheel
        running build
        running build_py
        creating build
        creating build/lib.linux-x86_64-3.9
        creating build/lib.linux-x86_64-3.9/cffi
        copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
        copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
        running build_ext
        building '_cffi_backend' extension
        creating build/temp.linux-x86_64-3.9
        creating build/temp.linux-x86_64-3.9/c
        gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/root/.cache/pre-commit/reponmshav7x/py_env-python3.9/include -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
        error: command 'gcc' failed: No such file or directory
        ----------------------------------------
        ERROR: Failed building wheel for cffi
        Running setup.py clean for cffi
      Failed to build cffi
      Installing collected packages: setuptools, wheel, pycparser, cffi, toml, semantic-version, setuptools-rust
          Running setup.py install for cffi: started
          Running setup.py install for cffi: finished with status 'error'
          ERROR: Command errored out with exit status 1:
           command: /root/.cache/pre-commit/reponmshav7x/py_env-python3.9/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_1btoeh2/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_1btoeh2/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-yx_q6xtg/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-u7scfaiu/overlay --compile --install-headers /tmp/pip-build-env-u7scfaiu/overlay/include/site/python3.9/cffi
               cwd: /tmp/pip-install-_1btoeh2/cffi/
          Complete output (45 lines):

              No working compiler found, or bogus compiler options passed to
              the compiler from Python's standard "distutils" module.  See
              the error messages above.  Likely, the problem is not related
              to CFFI but generic to the setup.py of any Python package that
              tries to compile C code.  (Hints: on OS/X 10.8, for errors about
              -mno-fused-madd see http://stackoverflow.com/questions/22313407/
              Otherwise, see https://wiki.python.org/moin/CompLangPython or
              the IRC channel #python on irc.freenode.net.)

              Trying to continue anyway.  If you are trying to install CFFI from
              a build done in a different context, you can ignore this warning.

          running install
          running build
          running build_py
          creating build
          creating build/lib.linux-x86_64-3.9
          creating build/lib.linux-x86_64-3.9/cffi
          copying cffi/verifier.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/lock.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/cparser.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/api.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/__init__.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/model.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/error.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/commontypes.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/recompiler.py -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/_embedding.h -> build/lib.linux-x86_64-3.9/cffi
          copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.9/cffi
          running build_ext
          building '_cffi_backend' extension
          creating build/temp.linux-x86_64-3.9
          creating build/temp.linux-x86_64-3.9/c
          gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/root/.cache/pre-commit/reponmshav7x/py_env-python3.9/include -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
          error: command 'gcc' failed: No such file or directory
          ----------------------------------------
      ERROR: Command errored out with exit status 1: /root/.cache/pre-commit/reponmshav7x/py_env-python3.9/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-_1btoeh2/cffi/setup.py'"'"'; __file__='"'"'/tmp/pip-install-_1btoeh2/cffi/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-yx_q6xtg/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-u7scfaiu/overlay --compile --install-headers /tmp/pip-build-env-u7scfaiu/overlay/include/site/python3.9/cffi Check the logs for full command output.
      ----------------------------------------
    ERROR: Command errored out with exit status 1: /root/.cache/pre-commit/reponmshav7x/py_env-python3.9/bin/python /root/.cache/pre-commit/reponmshav7x/py_env-python3.9/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-u7scfaiu/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.6.0' wheel 'cffi>=1.12; platform_python_implementation != '"'"'PyPy'"'"'' 'setuptools-rust>=0.11.4' Check the logs for full command output.

Check the log at /root/.cache/pre-commit/pre-commit.log

Smoking gun: error: command 'gcc' failed: No such file or directory

This Dockerfile does not have gcc installed, and so therefore cannot be used with this action, or any that suffer the same issue.
@tjamet
Copy link
Contributor

tjamet commented May 27, 2021

thanks for raising.
I will try to take a look sometime soon

@tjamet
Copy link
Contributor

tjamet commented Feb 8, 2022

Sorry about the delay taking a look at the issue.

The problem here is that some python modules need to be installed to be able to be run.
Usually, pre-commit is able to do it on a single machine and this is not a big deal because it is done only once.

Running pre-commit in a container would mean that the container is re-built every time pre-commit is run. Hence probably too often.

We could evaluate whether adding the relevant volume bindings to the pre-commit image and gcc would help. But I tend to foresee complications moving forward.

My suggestion here would be to try to pre-bake an image with all the pre-commit tools inside it. Not sure how to do so though.

Hope this helps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@m-roberts @tjamet