python3 + openssl compilation fail

[AndreMiras]

Versions

• Python: Host Python 3.6.8, target Python 3.7.1
• OS: Ubuntu 18.04
• Kivy: 1.10.1
• Cython: 0.28.6

Description

Compilation is failing if python3 and openssl are both in the recipe list, but python3 gets compiled first

Command:

# Command failed: /usr/bin/python3 -m pythonforandroid.toolchain create --dist_name=etheroll --bootstrap=sdl2 --requirements=android,cffi,openssl,python3==3.7.1 --ndk-api 27 --arch armeabi-v7a --copy-libs --local-recipes /home/andre/workspace/EtherollApp/src/python-for-android/recipes --color=always --storage-dir="/home/andre/workspace/EtherollApp/.buildozer/android/platform/build" --ndk-api=27

Logs

Console log:

configure: WARNING: unrecognized options: --with-openssl
checking build system type... x86_64-pc-linux-gnu
checking host system type... arm-unknown-linux-androideabi
checking for python3.6... python3.6
checking for python interpreter for cross build... python3.6
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... linux
checking for --without-gcc... no
checking for --with-icc... no
checking for arm-linux-androideabi-gcc... /home/andre/.buildozer/android/platform/android-ndk-r17c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang -target armv7a-none-linux-androideabi -gcc-toolchain /home/andre/.buildozer/android/platform/android-ndk-r17c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64
checking whether the C compiler works... no
configure: error: in `/home/andre/workspace/EtherollApp/.buildozer/android/platform/build/build/other_builds/python3-libffi-openssl/armeabi-v7a__ndk_target_27/python3/android-build':
configure: error: C compiler cannot create executables
See `config.log' for more details

Relevant .buildozer/android/platform/build/build/other_builds/python3-libffi-openssl/armeabi-v7a__ndk_target_27/python3/android-build/config.log part:

/home/andre/.buildozer/android/platform/android-ndk-r17c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lcrypto1.1
/home/andre/.buildozer/android/platform/android-ndk-r17c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lssl1.1

And listing the directory .buildozer/android/platform/build/build/other_builds/openssl-python3/armeabi-v7a__ndk_target_27/openssl1.1 I couldn't indeed find the shared libssl and libcrypto objects.

This is because python3 was compiled before openssl was.
The fix should be fairly simple, basically in python3 recipe we need to dynamically add openssl to the depends list if the openssl recipe is present in the requirements list

[opacam]

Ei, @AndreMiras, I remember that I read a long time ago (I don't know where...I tried to find in p4a docs...unsuccessfully...still may be there...) that the requirements has to have certain order:

--requirements=<pure libraries><python><python dependencies>

I always use that order...can you try it with this?

--requirements=openssl,python3==3.7.1,android,cffi

I think that this should work, even so maybe we have to take a look at what you propose plus this problem it may happen also with libffi and sqlite3 (our current python dependencies) or properly document that the requirements needs the order I mention...if I'm right about that...which I'm not sure...

Anyone can confirm/refuse something about the order of the requirements?

[AndreMiras]

Thanks for your feedback! As for the order thingy I didn't know that one.
The order I had it was: android,cffi,openssl,python3==3.7.1 and well the CI is seems to be running fine with: libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools
I think it would be great if that was handled seamlessly by p4a recipes, just like pip is resolving dependency order himself seamlessly.
In the meantime I'll play with Docker to confirm and give more insight about it

[AndreMiras]

So I couldn't reproduce in a Docker environment.
I've tried messing up with the order by first doing:

python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements python3==3.7.1,openssl

then:

python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements openssl,python3==3.7.1

And deleting ~/.local/share/python-for-android/ in between, but I always got a green build 🤔

[ghost]

We discussed this here #1576 but I think python3 should always pull in openssl as a dependency unless some tiny option is set.

This could be achieved e.g. by renaming python3 to corepython3, and making a new python3 recipe that depends on both corepython3 and openssl - so the default of python3 would pull in all the deps like openssl, while people who want a tiny app can still use just corepython3 (or tinypython3 or however you want to name it)

[ghost]

(Sorry, I forgot to add the reason: I think it should do that because on desktop you can't specify python standard modules as extra dependency either, so if we move closer to pip/standard dependency mechanisms/... which I suggest we do in the long term, then requiring standard library parts to be listed explicitly in this extra list moves us away from that goal, and on top of things is counter-intuitive for beginners)

[ghost]

@AndreMiras wrong order won't stop the build, but the _ssl module won't be added so at runtime you simply can't use it (ImportError). So this isn't something that could be tested without actually running the app

Edit: or let's say, will not necessarily stop the build. so what you see up there could be something else, maybe messed up order + no openssl headers on the host trigger this or some other more unusual combination. nevertheless, the build succeeding doesn't mean the wrong order hasn't caused a problem

[inclement]

As a general note, python-for-android should not require any special ordering of the requirements, as long as the dependencies are specified correctly in the recipes it should always construct a good build order.

It's clear that this doesn't always happen for some reason, but I'm not clear what causes this or how to reproduce it.

[inclement]

Does anyone have a reproducible test case to look at for the openssl ordering thing? I've tried a few, but can't get p4a to try to build openssl after python3.

[inclement]

On further examination of the graph.py code, I think I might have broken opt_depends a very long time ago when I replaced the old algorithm. I'm not sure why this is only now causing issues.

[inclement]

Closing as this is quite possibly fixed by #1617, please reopen or make a new issue if it turns out it does still happen.