Unable to build python3.8 language module in a conda environment #563

synodriver · 2021-07-04T17:05:57Z

Hi,
I'm trying compiling Unit's source code with Python (3.8) language module in a conda environment, but an error occurs at module configuration step:

./configure python --module=py38 --config="/root/miniconda3/bin/python3.8-config --libs --embed" --lib-path=/root/miniconda3/lib

with the following output

configuring Python module
checking for Python ... not found

In the log file build\autoconf.err

configuring Python module ...
-lpython3.8 -lcrypt -lpthread -ldl  -lutil -lrt -lm -lm 
/root/miniconda3
----------------------------------------
checking for Python
/usr/bin/ld: cannot find -lpython3.8
collect2: error: ld returned 1 exit status
/bin/sh: 2: -I/root/miniconda3/include/python3.8: not found
/bin/sh: 3: -L/root/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu: not found
----------

        #include <Python.h>

        int main() {
            Py_Initialize();
            return 0;
        }
----------
cc -pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -lpython3.8 -lcrypt -lpthread -ldl -lutil -lrt -lm -lm -I/root/miniconda3/include/python3.8 -I/root/miniconda3/include/python3.8 -o build/autotest build/autotest.c -lpython3.8 -lcrypt -lpthread -ldl -lutil -lrt -lm -lm -L/root/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu -L/root/miniconda3/lib -lpython3.8 -lcrypt -lpthread -ldl -lutil -lrt -lm -lm -L/root/miniconda3/lib -Wl,-rpath,/root/miniconda3/lib
----------

But all files mentioned above do exists.

The command cat /etc/issue

Debian GNU/Linux 10

The command cat /proc/version

Linux version 4.19.0-11-amd64 ([email protected]) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.146-1 (2020-09-17)

The command /root/miniconda3/bin/python3.8-config --libs --embed

-lpython3.8 -lcrypt -lpthread -ldl  -lutil -lrt -lm -lm

The conda version

conda 4.10.1

Full steps to reproduce error:

login as root (In order to get rid of sudo)
install miniconda from office website
apt install build-essential
apt install libssl-dev
apt install libpcre2-dev
git clone https://github.com/nginx/unit
In Unit directory execute ./configure
./configure python --module=py38 --config="/root/miniconda3/bin/python3.8-config --libs --embed" --lib-path=/root/miniconda3/lib

Any ideas what can be wrong?

The text was updated successfully, but these errors were encountered:

synodriver · 2021-07-04T17:09:35Z

I think there maybe something special in conda environment, but after commandconda activate base,the same problem still occurs. Besides, system python works fine and can be properly compiled to a language module.

mar0x · 2021-07-05T12:11:52Z

Hello,
Please try the following configure command:

./configure python --module=py38 --config=/root/miniconda3/bin/python3.8-config --lib-path=/root/miniconda3/lib

synodriver · 2021-07-05T16:51:38Z

Hello,
Please try the following configure command:

./configure python --module=py38 --config=/root/miniconda3/bin/python3.8-config --lib-path=/root/miniconda3/lib

Hello,
the command doesn't work either, with the following output

configuring Python module
checking for Python ... not found

./configure: error: no Python found.

And the build\autoconf.err

configuring Python module ...
/root/miniconda3
----------------------------------------
checking for Python
lto1: fatal error: bytecode stream in file '/root/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a' generated with LTO version 6.0 instead of the expected 7.1
compilation terminated.
lto-wrapper: fatal error: cc returned 1 exit status
compilation terminated.
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
----------

        #include <Python.h>

        int main() {
            Py_Initialize();
            return 0;
        }
----------
cc -pipe -fPIC -fvisibility=hidden -O -W -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-prototypes -Werror -g -I/root/miniconda3/include/python3.8 -I/root/miniconda3/include/python3.8 -o build/autotest build/autotest.c -L/root/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu -L/root/miniconda3/lib -lpython3.8 -lcrypt -lpthread -ldl -lutil -lrt -lm -lm -L/root/miniconda3/lib -Wl,-rpath,/root/miniconda3/lib
----------

The command gcc -v

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 8.3.0-6' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 8.3.0 (Debian 8.3.0-6)

Is the gcc version too high?

synodriver · 2021-07-05T17:19:46Z

Find something interesting with conda here. Both of us used conda. Is this the same reason?

mar0x · 2021-07-05T23:13:45Z

Yes, the reason is newer current gcc version than the version used to build libpython.

The simplest way for you is to disable LTO with ./configure --ld-opt="-fno-lto".

Here is the similar issue and the long story with various suggestions and attempts to fix it. I've tried some of suggested approaches (installing gxx_linux-64, recipe creation, conda-build) with no luck :(

synodriver · 2021-07-06T01:18:55Z

Yes, the reason is newer current gcc version than the version used to build libpython.

The simplest way for you is to disable LTO with ./configure --ld-opt="-fno-lto".

Here is the similar issue and the long story with various suggestions and attempts to fix it. I've tried some of suggested approaches (installing gxx_linux-64, recipe creation, conda-build) with no luck :(

Hi,
I successed at last and I‘d like to share what I'm doing when using conda.

./configure --ld-opt="-fno-lto"
./configure python --module=py38 --config=/root/miniconda3/bin/python3.8-config --lib-path=/root/miniconda3/lib
make

then move py38.unit.so to modules
start unitd with the following config write to config.json

{
  "listeners": {
    "*:9001": {
      "pass": "applications/fastapi"
    }
  },
  "applications": {
    "fastapi": {
      "type": "python 3.8",
      "path": "your app path",
      "module": "main",
      "callable": "app"
    }
  }
}

Currently I'm using main.py with

# -*- coding: utf-8 -*-
from fastapi import FastAPI
from fastapi.responses import HTMLResponse

app = FastAPI()


@app.get("/")
async def _():
    return HTMLResponse("<p>hello, this is nginx unit</p>", 200)

then

 curl -X PUT --data-binary @config.json --unix-socket ./control.unit.sock http://localhost/config/

then

curl http://127.0.0.1:9001

It works!

Notice (Very impotant)

Unlike that in example Do not set applications.home otherwise you will get

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

I don't know why but it just works.

synodriver · 2021-07-06T01:22:32Z

This app is used to server some deeplearning project that is hard to configure with normal venv. In this case, conda is necessary. Hoping to help someone.

mar0x · 2021-07-06T08:35:02Z

Notes: --lib-path configure option is not required, configure will deduce it from config path. --module option is helpful only to avoid file name collision. Full path for python3.8-config is only required if python3.8-config cannot be found using PATH environment variable. So, the shortest possible command is ./configure python --config=python3.8-config.

Regarding No module named 'encodings'. The reason is the different python version in virtual environment and build module.

synodriver closed this as completed Jul 6, 2021

synodriver mentioned this issue Oct 16, 2021

Unit对Python包的搜索路径有哪些？ #589

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to build python3.8 language module in a conda environment #563

Unable to build python3.8 language module in a conda environment #563

synodriver commented Jul 4, 2021

synodriver commented Jul 4, 2021 •

edited

Loading

mar0x commented Jul 5, 2021

synodriver commented Jul 5, 2021 •

edited

Loading

synodriver commented Jul 5, 2021

mar0x commented Jul 5, 2021

synodriver commented Jul 6, 2021 •

edited

Loading

synodriver commented Jul 6, 2021

mar0x commented Jul 6, 2021

Unable to build python3.8 language module in a conda environment #563

Unable to build python3.8 language module in a conda environment #563

Comments

synodriver commented Jul 4, 2021

synodriver commented Jul 4, 2021 • edited Loading

mar0x commented Jul 5, 2021

synodriver commented Jul 5, 2021 • edited Loading

synodriver commented Jul 5, 2021

mar0x commented Jul 5, 2021

synodriver commented Jul 6, 2021 • edited Loading

Notice (Very impotant)

synodriver commented Jul 6, 2021

mar0x commented Jul 6, 2021

synodriver commented Jul 4, 2021 •

edited

Loading

synodriver commented Jul 5, 2021 •

edited

Loading

synodriver commented Jul 6, 2021 •

edited

Loading