Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need libpython3.7m.so.1.0 in android phone #1501

Closed
homdx opened this issue Dec 2, 2018 · 26 comments · Fixed by #1568
Closed

Need libpython3.7m.so.1.0 in android phone #1501

homdx opened this issue Dec 2, 2018 · 26 comments · Fixed by #1568

Comments

@homdx
Copy link

homdx commented Dec 2, 2018

Hello.
I build with this instruction:

https://groups.google.com/forum/#!topic/kivy-users/Lf4zlYmLVPo

Apk compiled and builded. After setup and launch in my Android phone, I have a error:

SDL Error

An error occurred while trying to start the applicaion. Please try again and/or reinstall.
Error: dl open failed: could not load library: "libpython3.7m.so.1.0" needed by "libmain.so"; caused by library "libpython3.7m.so.1.0" not foud

log from logcat:

V/pythonutil(11885): Checking pattern libcrystax\.so against libSDL2_mixer.so
V/pythonutil(11885): Checking pattern libcrystax\.so against libSDL2_ttf.so
V/pythonutil(11885): Checking pattern libcrystax\.so against libmain.so
V/pythonutil(11885): Checking pattern libcrystax\.so against libSDL2.so
V/pythonutil(11885): Checking pattern libcrystax\.so against libSDL2_image.so
V/pythonutil(11885): Checking pattern libcrystax\.so against libpython3.7m.so
V/pythonutil(11885): Checking pattern libsqlite3\.so against libSDL2_mixer.so
V/pythonutil(11885): Checking pattern libsqlite3\.so against libSDL2_ttf.so
V/pythonutil(11885): Checking pattern libsqlite3\.so against libmain.so
V/pythonutil(11885): Checking pattern libsqlite3\.so against libSDL2.so
V/pythonutil(11885): Checking pattern libsqlite3\.so against libSDL2_image.so
V/pythonutil(11885): Checking pattern libsqlite3\.so against libpython3.7m.so
V/pythonutil(11885): Checking pattern libssl.*\.so against libSDL2_mixer.so
V/pythonutil(11885): Checking pattern libssl.*\.so against libSDL2_ttf.so
V/pythonutil(11885): Checking pattern libssl.*\.so against libmain.so
V/pythonutil(11885): Checking pattern libssl.*\.so against libSDL2.so
V/pythonutil(11885): Checking pattern libssl.*\.so against libSDL2_image.so
V/pythonutil(11885): Checking pattern libssl.*\.so against libpython3.7m.so
V/pythonutil(11885): Checking pattern libcrypto.*\.so against libSDL2_mixer.so
V/pythonutil(11885): Checking pattern libcrypto.*\.so against libSDL2_ttf.so
V/pythonutil(11885): Checking pattern libcrypto.*\.so against libmain.so
V/pythonutil(11885): Checking pattern libcrypto.*\.so against libSDL2.so
V/pythonutil(11885): Checking pattern libcrypto.*\.so against libSDL2_image.so
V/pythonutil(11885): Checking pattern libcrypto.*\.so against libpython3.7m.so
V/pythonutil(11885): Loading library: SDL2
V/pythonutil(11885): Loading library: SDL2_image
V/pythonutil(11885): Loading library: SDL2_mixer
V/pythonutil(11885): Loading library: SDL2_ttf
V/pythonutil(11885): Loading library: python2.7
V/pythonutil(11885): Library loading error: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.kivymd.test-1/base.apk"],nativeLibraryDirectories=[/data/app/com.kivymd.test-1/lib/arm, /vendor/lib, /system/lib]]] couldn't find "libpython2.7.so"
V/pythonutil(11885): Loading library: python3.5m
V/pythonutil(11885): Library loading error: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.kivymd.test-1/base.apk"],nativeLibraryDirectories=[/data/app/com.kivymd.test-1/lib/arm, /vendor/lib, /system/lib]]] couldn't find "libpython3.5m.so"
V/pythonutil(11885): Loading library: python3.6m
V/pythonutil(11885): Library loading error: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.kivymd.test-1/base.apk"],nativeLibraryDirectories=[/data/app/com.kivymd.test-1/lib/arm, /vendor/lib, /system/lib]]] couldn't find "libpython3.6m.so"
V/pythonutil(11885): Loading library: python3.7m
V/pythonutil(11885): Loading library: main
E/art     (11885): dlopen("/data/app/com.kivymd.test-1/lib/arm/libmain.so", RTLD_LAZY) failed: dlopen failed: could not load library "libpython3.7m.so.1.0" needed by "libmain.so"; caused by library "libpython3.7m.so.1.0" not found
V/pythonutil(11885): Library loading error: dlopen failed: could not load library "libpython3.7m.so.1.0" needed by "libmain.so"; caused by library "libpython3.7m.so.1.0" not found
V/pythonutil(11885): An UnsatisfiedLinkError occurred loading main
W/System.err(11885): dlopen failed: could not load library "libpython3.7m.so.1.0" needed by "libmain.so"; caused by library "libpython3.7m.so.1.0" not found``` 
@ghost
Copy link

ghost commented Dec 2, 2018

It seems like something with the linker options might have gone wrong, so it might be helpful to see the build. Could you enable p4a's --debug option, rerun the build, and then attach the output as a text file? (It's going to be very long)

@homdx
Copy link
Author

homdx commented Dec 2, 2018

build.log

@ghost
Copy link

ghost commented Dec 2, 2018

@inclement could you take a look at this? This is something related to the new python3 recipe, and it seems main.so (the start.c bootstrap entry point of the sdl2 bootstrap) is built wrong.

This is the main.so linker line from the log:

[DEBUG]:   	[armeabi-v7a] SharedLibrary  : libmain.so
[DEBUG]:   	/home/user/hostcwd/manbuild/android-ndk-r17c/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -Wl,-soname,libmain.so -shared --sysroot=/home/user/hostcwd/manbuild/android-ndk-r17c/platforms/android-21/arch-arm /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2-python3/obj/local/armeabi-v7a/objs/main/__/__/SDL/src/main/android/SDL_android_main.o /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2-python3/obj/local/armeabi-v7a/objs/main/start.o -lgcc -Wl,--exclude-libs,libgcc.a -latomic -Wl,--exclude-libs,libatomic.a /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2-python3/obj/local/armeabi-v7a/libSDL2.so  -gcc-toolchain /home/user/hostcwd/manbuild/android-ndk-r17c/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64 -no-canonical-prefixes -target armv7-none-linux-androideabi21 -Wl,--fix-cortex-a8 -L/home/user/.local/share/python-for-android/build/other_builds/python3/armeabi-v7a__ndk_target_21/python3/android-build -Wl,--build-id -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--warn-shared-textrel -Wl,--fatal-warnings   -L/home/user/hostcwd/manbuild/android-ndk-r17c/platforms/android-21/arch-arm/usr/lib -lGLESv1_CM -lGLESv2 -llog -lpython3.7m -lc -lm -o /home/user/.local/share/python-for-android/build/bootstrap_builds/sdl2-python3/obj/local/armeabi-v7a/libmain.so

I suspect the -lpython3.7m is potentially the issue. I wonder, is clang linking the full name libpython3.7m.so.1.0 when it should just link libpython3.7m.so (which was already loaded as per the runtime log in the original post, Loading library: python3.7m)? Might we need either an option for clang to not do that, or a symlink of that full name back to the short named actual binary?

Maybe this is an issue with the specific newer/older clang from this particular NDK version?

@inclement
Copy link
Member

I'm not sure exactly what is wrong here, but both libpython3.7m.so and libpython3.7m.so.1.0 should be present in the APK. I think it is expected that libpython3.7m.so relies on symbols loaded from libpython3.7m.so.1.0. I wondered if this would cause issues with linking when I first set this up, but forgot about it when it didn't appear to do so.

I wonder, does this still happen if libpython3.7m.so.1.0 is loaded explicitly when the app is started, as we do for the other library files? I think this would require slightly more modification to PythonUtil.java than one might hope, because the Java method called to load the libraries automatically finds them and only supports a .so suffix (although I could be wrong about that).

@homdx
Copy link
Author

homdx commented Dec 2, 2018

@inclement @Jonast You can reproduce this? (This error launch in phone)?

--dist_name=py3-sink
--private .
--package=com.kivymd.test
--name KivyMD
--bootstrap=sdl2
--requirements=python3,kivy
--arch=armeabi-v7a
--icon /home/user/hostcwd/app/kivymd/images/kivymd_logo.png
--sdk_dir /home/user/hostcwd/manbuild/sdk
--ndk_dir /home/user/hostcwd/manbuild/android-ndk-r17c
--android_api 28
--ndk_version 17c
--ndk-api 21
--version 0.0.1

Also this error aviable, then I compile Hello World. This apk in phone start with error (see bottom message) :-(

@homdx
Copy link
Author

homdx commented Dec 2, 2018

py3-sink.zip

You can uncompress for sea this result apk

@ghost
Copy link

ghost commented Dec 2, 2018

I just checked, there is only lib/armeabi-v7a/libpython3.7m.so - no trace of a libpython3.7m.so.1.0

@inclement
Copy link
Member

Ah, quite interesting. I wonder at what point in the build process it failed to appear or be copied around.

@homdx it looks like you are using p4a directly? What files are present in `~/.local/share/python-for-android/dists/$YOURDISTNAME/libs/armeabi-v7a?

@homdx
Copy link
Author

homdx commented Dec 2, 2018

I can reproduce for you Docker file, if you need?)

@homdx
Copy link
Author

homdx commented Dec 2, 2018

yes. I run with:
$p4a apk
@Jonast @inclement
compiled files:

:~/.local/share/python-for-android/dists/py3-sink$ ls -la libs/armeabi-v7a/
total 13820
drwxr-xr-x. 2 user user     164 Dec  2 07:09 .
drwxr-xr-x. 3 user user      25 Dec  2 07:07 ..
-rwxr-xr-x. 1 user user   18112 Dec  2 07:09 libmain.so
-rwxr-xr-x. 1 user user 2776024 Dec  2 07:09 libpython3.7m.so
-rwxr-xr-x. 1 user user 9357004 Dec  2 07:08 libpython3.7m.so.1.0
-rwxr-xr-x. 1 user user  415396 Dec  2 07:09 libSDL2_image.so
-rwxr-xr-x. 1 user user  282740 Dec  2 07:09 libSDL2_mixer.so
-rwxr-xr-x. 1 user user  878352 Dec  2 07:09 libSDL2.so
-rwxr-xr-x. 1 user user  407204 Dec  2 07:09 libSDL2_ttf.so

@ghost
Copy link

ghost commented Dec 2, 2018

@homdx I just attempted to configure your exact build combination and see if compiling the kivy demo app will include libpython3.7m.so.1.0 for me or not, but I ran into another error #1502 - oops!

Basically all I'm saying is I'll tell you if I can reproduce this as soon as I manage to actually get through the build 😆 hang in there

@ghost
Copy link

ghost commented Dec 3, 2018

@inclement I can confirm that libpython3.7m.so.1.0 isn't added even for a trivial test app:

Steps to reproduce manually:

  1. Install NDK r17c (NDK API 21), SDK 28
  2. Install p4a master, python3, cython and such
  3. Fetch https://github.com/kivy/python-for-android/tree/master/testapps/testapp_keyboard
  4. Build test app with: p4a apk --arch=armeabi-v7a --name test --package com.example.test --version 1 --requirements=kivy,python3 --private /path/to/app and all the necessary options to use your NDK r17c with API 21, and the Android SDK with API 28 with their given paths and such

Steps to reproduce in docker environment: (with my little experimental build environment tool)

  1. Install docker, launch docker daemon
  2. sudo pip3 install https://github.com/JonasT/p4a-build-spaces/archive/master.zip (installs p4a-build-spaces, system-wide to get the binary installed to $PATH)
  3. sudo p4aspaces shell p4a-py3-api28ndk21 --p4a master (launches the build env - sudo only needed if your regular user isn't in the docker group)
  4. In the build shell: cd ~/testapp && p4a apk --arch=armeabi-v7a --name test --package com.example.test --version 1 --requirements=kivy,python3 --private . (no more options required, SDK/NDK are preconfigured via env)

This is the result when examining the .apk (note the missing libpython3.7m.so):

# unzip ../unnamed_dist_1-1-debug.apk 
Archive:  ../unnamed_dist_1-1-debug.apk
  inflating: AndroidManifest.xml     
  inflating: META-INF/CERT.RSA       
  inflating: META-INF/CERT.SF        
  inflating: META-INF/MANIFEST.MF    
 extracting: assets/private.mp3      
  inflating: classes.dex             
  inflating: lib/armeabi-v7a/libSDL2.so  
  inflating: lib/armeabi-v7a/libSDL2_image.so  
  inflating: lib/armeabi-v7a/libSDL2_mixer.so  
  inflating: lib/armeabi-v7a/libSDL2_ttf.so  
  inflating: lib/armeabi-v7a/libmain.so  
  inflating: lib/armeabi-v7a/libpython3.7m.so  
 extracting: res/drawable-hdpi-v4/ic_launcher.png  
 extracting: res/drawable-mdpi-v4/ic_launcher.png  
 extracting: res/drawable-xhdpi-v4/ic_launcher.png  
 extracting: res/drawable-xxhdpi-v4/ic_launcher.png  
 extracting: res/drawable/icon.png   
 extracting: res/drawable/presplash.jpg  
  inflating: res/layout/chooser_item.xml  
  inflating: res/layout/main.xml     
  inflating: res/layout/project_chooser.xml  
  inflating: res/layout/project_empty.xml  
 extracting: resources.arsc

However, the .so is still present in the original dist folder:

# ls /root/.local/share/python-for-android/dists/unnamed_dist_1/libs/armeabi-v7a/
libSDL2.so  libSDL2_image.so  libSDL2_mixer.so  libSDL2_ttf.so  libmain.so  libpython3.7m.so  libpython3.7m.so.1.0

@inclement
Copy link
Member

Wow, that's quite interesting. Am I right in thinking that you've successfully used the python3 recipe, so it isn't a generic problem that happens to everyone except me?
I wonder if there's some gradle setting changed or similar, affecting what libraries get bundled.
I'd hope that it should be simple enough to make gradle include the file, unless it has some specific reason that it doesn't want to. What version of the build-tools is this using?

@ghost
Copy link

ghost commented Dec 3, 2018

@inclement the exact environment p4a-py3-api28ndk21's details are specified here:

https://github.com/JonasT/p4a-build-spaces/blob/master/environments/p4a-py3-api28ndk21/Dockerfile

Edit: I don't know if it's a generic problem, because I haven't tried any other environment with the python3 recipe. I use python3crystax with the TLS hack rebuild usually, which has no such issues but I guess that's also packaged differently

Please also note that sudo p4aspaces shell p4a-py3-api28ndk21 --p4a master gives you, after some build waiting time, a bash with everything isolated & installed in the affected versions in docker (needs docker installed of course). Just in case that would help with debugging. I can also assist with that on discord if that helps, both debugging and launching the environment

@homdx
Copy link
Author

homdx commented Dec 5, 2018

@Jonast Thank you. You release very interesting tool p4a-build-spaces:

Step 27/27 : CMD bash -c "`python3 /cmdline.py`"
 ---> Running in 1db4bc5c74d7
 ---> ae0bb8e5ac24
Removing intermediate container 1db4bc5c74d7
Successfully built ae0bb8e5ac24

To build a kivy demo app, use this command:
cd ~/testapp && p4a apk --arch=armeabi-v7a --name test --package com.example.test --version 1 --requirements=kivy,python3 --private .
root@c528c50f1cd9:~#

Now wait fix by @inclement for python p4a apk build

@maho
Copy link
Contributor

maho commented Jan 3, 2019

while trying to build with Dockerfile.py3 - I have identical error msg.

cmd within docker is

~/venv/bin/p4a apk --sdk-dir=/opt/android/android-sdk/ --android-api=27 --ndk-dir=/opt/android/android-ndk-r16b/ --ndk-version=16b --requirements=python3,kivy --package=maho.test --name=pong --version=0.0.1 --private=$PWD --debug --copy-libs

@maho
Copy link
Contributor

maho commented Jan 3, 2019

short investigation: gradle is guilty for such situation. Why? it's total mysterious tool for me, so I have no idea. However - so.1.0 exists in libs/ dir, but is absent in .apk

opacam added a commit to opacam/python-for-android that referenced this issue Jan 6, 2019
The Java method called to load the libraries only supports a .so suffix, so we must remove the version for our python libraries, or we will get linkage errors

Resolves: kivy#1501
@opacam
Copy link
Member

opacam commented Jan 6, 2019

Ok, I solved the linkage problem with libpython3.7.so.1.0, and @inclement was in the right direction:

...the Java method called to load the libraries automatically finds them and only supports a .so suffix

So...there is no way that the java loader can load a versioned library...therefore... is no problem with gradle, is our build method what is causing the problems, but this can be easily solved by forcing that the python library gets built without the version, so this way we will have the proper suffix, and we will avoid all this linkage problems.

The fix is already applied in pr #1537, but here I leave a link to a patch that will solve the problem for the current master branch: https://gist.github.com/opacam/10cef1ca8182a39b9e5ebffe0adcdc90#file-fix-python3-linkage-patch

opacam added a commit to opacam/python-for-android that referenced this issue Jan 7, 2019
…NSTSONAME`)

We fix this issue by re-introducing `INSTSONAME`. This variable has been used in all our python recipes, because allow us to remove the version of the compiled python library which is mandatory, at the time of writing, because android does not support versioned libraries (the Java method called to load the libraries only supports a .so suffix).

A little history, because I didn't find any official documentation for this variable, and **it's quite important for us**:

  - This variable it's not hacked, it's official and was introduced a long time ago, forms part of the python's shared library building process (python/cpython@1142de3)
  - @tito introduced this variable almost at the beginning of the p4a project in (bdefea1)
  - @inclement also make use of it when he reworked the python2 recipe (4c8b5bc)
  - this variable still exists in the current python2 recipe and we make use of it precisely to solve the mentioned android's libraries version problem
  - Somehow, during the process of making the new python3 recipe, we loose this variable and then, we begin to have linkage problems at runtime with our python library as described in issue kivy#1501

Note: ¡¡¡Special thanks to @Jonast!!!...to force me to search this information ;)

References: python/cpython@1142de3, bdefea1 and 4c8b5bc
Resolves: kivy#1501
@homdx
Copy link
Author

homdx commented Jan 7, 2019

@opacam I comfimed. Now app are start in mobile phone without this error)

I/python  (20877): [INFO   ] [Kivy        ] v1.10.1
I/python  (20877): [INFO   ] [Python      ] v3.7.1 (default, Jan  7 2019, 21:35:27) 
I/python  (20877): [Clang 5.0.300080 ]

But I compile only with buildozer 0.37: api = 27, sdk=24, nkd 16b,
How use and build with p4a ? Wait merge to master branch python-for-android ?
Or you can add instruction for me?

@homdx
Copy link
Author

homdx commented Jan 8, 2019

@opacam Thank you! I pushed you patch for testing in branch fix/python37
https://github.com/homdx/p4a-build-spaces
@Jonast, @inclement You can probe build )

@opacam
Copy link
Member

opacam commented Jan 8, 2019

How use and build with p4a ?

My method consist in set the variable p4a.source_dir of my app's buildozer spec file to point to my pythonforandroid development folder, so I can build with any branch of my local fork of p4a...but maybe someone has a better method...

@homdx
Copy link
Author

homdx commented Jan 8, 2019

@opacam I think, need feature for p4a with path like p4a.source_dir in buildozer

@opacam
Copy link
Member

opacam commented Jan 8, 2019

This option is commented in the default buildozers spec file...I suppose that you modified this file from a previous version of buildozers...so...if you don't find this option commented in your buildozers file just add it and put the corresponding path (look at almost the end of the section app, before buildozer)

@tito
Copy link
Member

tito commented Jan 9, 2019

Guys, about this, previous Python 2.7 compilation was forcing INSTSONAME to be just libpython2.7.so, while now, none of the Python 2 and 3 recipes set it.
In consequence, the versionned version of the library will be used for linkage for all the library that do -lpython.

The right fix is either set INSTSONAME=libpythonXX.so as before, either patch Python's configure to remove the versionning (like i do in 15a3770). Then when compiling, it will use just libpythonXX.so, and everything will be fine.

@ghost
Copy link

ghost commented Jan 9, 2019

This might not be actually fixed, since the commit was backed out into the separate #1568 mostly because I requested it (because I asked to not bloat #1537 further). So I think this issue still persists, and will be fixed by #1568

opacam added a commit to opacam/python-for-android that referenced this issue Jan 9, 2019
…STSONAME`)

We fix this issue by re-introducing `INSTSONAME`. This variable has been used in all our python recipes, because allow us to remove the version of the compiled python library which is mandatory, at the time of writing, because android does not support versioned libraries (the Java method called to load the libraries only supports a .so suffix).

A little history, because I didn't find any official documentation for this variable, and **it's quite important for us**:

  - This variable it's not hacked, it's official and was introduced a long time ago, forms part of the python's shared library building process (python/cpython@1142de3)
  - @tito introduced this variable almost at the beginning of the p4a project in (bdefea1)
  - @inclement also make use of it when he reworked the python2 recipe (4c8b5bc)
  - this variable still exists in the current python2 recipe and we make use of it precisely to solve the mentioned android's libraries version problem
  - Somehow, during the process of making the new python3 recipe, we loose this variable and then, we begin to have linkage problems at runtime with our python library as described in issue kivy#1501

Note: ¡¡¡Special thanks to @Jonast!!!...to force me to search this information ;)

References: python/cpython@1142de3, bdefea1 and 4c8b5bc
Resolves: kivy#1501
@KeyWeeUsr
Copy link
Contributor

Merged #1568. Please test with the current master if the error is still present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants