[CORE UPDATE - PART I] Refactor python recipes + openssl + sqlite3

.travis.yml

@@ -18,7 +18,7 @@ env:
     - ANDROID_SDK_HOME=/opt/android/android-sdk
     - ANDROID_NDK_HOME=/opt/android/android-ndk
   matrix:
-    - COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements sdl2,pyjnius,kivy,python3'
+    - COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools'
     # overrides requirements to skip `peewee` pure python module, see:
     # https://github.com/kivy/python-for-android/issues/1263#issuecomment-390421054
     - COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python2_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements sdl2,pyjnius,kivy,python2,openssl,requests,sqlite3,setuptools'

Dockerfile.py2

@@ -26,7 +26,7 @@ RUN apt -y update -qq \
 
 
 ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
-ENV ANDROID_NDK_VERSION="16b"
+ENV ANDROID_NDK_VERSION="17c"
 ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"
 
 # get the latest version from https://developer.android.com/ndk/downloads/index.html
@@ -104,7 +104,7 @@ RUN dpkg --add-architecture i386 \
 # specific recipes dependencies (e.g. libffi requires autoreconf binary)
 RUN apt -y update -qq \
     && apt -y install -qq --no-install-recommends \
-        autoconf automake cmake gettext libltdl-dev libtool pkg-config \
+        libffi-dev autoconf automake cmake gettext libltdl-dev libtool pkg-config \
     && apt -y autoremove \
     && apt -y clean
 

Dockerfile.py3

@@ -26,7 +26,7 @@ RUN apt -y update -qq \
 
 
 ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
-ENV ANDROID_NDK_VERSION="16b"
+ENV ANDROID_NDK_VERSION="17c"
 ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"
 
 # get the latest version from https://developer.android.com/ndk/downloads/index.html
@@ -104,7 +104,7 @@ RUN dpkg --add-architecture i386 \
 # specific recipes dependencies (e.g. libffi requires autoreconf binary)
 RUN apt -y update -qq \
     && apt -y install -qq --no-install-recommends \
-        autoconf automake cmake gettext libltdl-dev libtool pkg-config \
+        libffi-dev autoconf automake cmake gettext libltdl-dev libtool pkg-config \
     && apt -y autoremove \
     && apt -y clean
 

README.rst

@@ -35,6 +35,25 @@ issues and PRs relating to this branch are still accepted. However, the
 new toolchain contains all the same functionality via the built in
 pygame bootstrap.
 
+In the last quarter of 2018 the python recipes has been changed, the new recipe
+for python3 (3.7.1) has a new build system which has been applied to the ancient
+python recipe, allowing us to bump the python2 version number to 2.7.15. This
+change, unifies the build process for both python recipes, and probably solve
+some issues detected over the years, but unfortunately, this change breaks the
+pygame bootstrap (in a near future we will fix it or remove it). Also should be
+mentioned that the old python recipe is still usable, and has been renamed to
+`python2legacy`. This `python2legacy` recipe allow us to build with a minimum
+api lower than 21, which is not the case for the new python recipes which are
+limited to a minimum api of 21. All this work has been done using android ndk
+version r17c, and your build should success with that version...but be aware
+that the project is in constant development so...the ndk version will change
+at some time.
+
+Those mentioned changes has been done this way to make easier the transition
+between python3 and python2. We will slowly phase out python2 support
+towards 2020...so...if you are using python2 in your projects you should
+consider to migrate it into python3.
+
 Documentation
 =============
 

doc/source/quickstart.rst

@@ -93,11 +93,27 @@ named ``tools``, and you will need to run extra commands to install
 the SDK packages needed. 
 
 For Android NDK, note that modern releases will only work on a 64-bit
-operating system. If you are using a 32-bit distribution (or hardware),
-the latest useable NDK version is r10e, which can be downloaded here:
+operating system. The minimal, and recommended, NDK version to use is r17c:
+
+ - `Go to ndk downloads page <https://developer.android.com/ndk/downloads/>`_
+ - Windows users should create a virtual machine with an GNU Linux os
+   installed, and then you can follow the described instructions from within
+   your virtual machine.
+
+If you are using a 32-bit distribution (or hardware),
+the latest usable NDK version is r10e, which can be downloaded here:
 
 - `Legacy 32-bit Linux NDK r10e <http://dl.google.com/android/ndk/android-ndk-r10e-linux-x86.bin>`_
 
+.. warning::
+    **32-bit distributions**
+
+    Since the python2 recipe updated to version 2.7.15, the build system has
+    been changed and you should use an old release of python-for-android, which
+    contains the legacy python recipe (v2.7.2). The last python-for-android
+    release with the legacy version of python is version
+    `0.6.0 <https://github.com/kivy/python-for-android/archive/0.6.0.zip>`_.
+
 First, install a platform to target (you can also replace ``27`` with
 a different platform number, this will be used again later)::
 
@@ -113,8 +129,8 @@ Then, you can edit your ``~/.bashrc`` or other favorite shell to include new env
 
     # Adjust the paths!
     export ANDROIDSDK="$HOME/Documents/android-sdk-27"
-    export ANDROIDNDK="$HOME/Documents/android-ndk-r10e"
-    export ANDROIDAPI="27"  # Target API version of your application
+    export ANDROIDNDK="$HOME/Documents/android-ndk-r17c"
+    export ANDROIDAPI="26"  # Target API version of your application
     export NDKAPI="19"  # Minimum supported API version of your application
     export ANDROIDNDKVER="r10e"  # Version of the NDK you installed
 

pythonforandroid/archs.py

@@ -1,5 +1,6 @@
-from os.path import (exists, join, dirname)
+from os.path import (exists, join, dirname, split)
 from os import environ, uname
+from glob import glob
 import sys
 from distutils.spawn import find_executable
 
@@ -30,13 +31,30 @@ def include_dirs(self):
                 d.format(arch=self))
             for d in self.ctx.include_dirs]
 
-    def get_env(self, with_flags_in_cc=True):
+    @property
+    def target(self):
+        target_data = self.command_prefix.split('-')
+        return '-'.join(
+            [target_data[0], 'none', target_data[1], target_data[2]])
+
+    def get_env(self, with_flags_in_cc=True, clang=False):
         env = {}
 
-        env['CFLAGS'] = ' '.join([
-            '-DANDROID', '-mandroid', '-fomit-frame-pointer'
-            ' -D__ANDROID_API__={}'.format(self.ctx.ndk_api),
-            ])
+        cflags = [
+            '-DANDROID',
+            '-fomit-frame-pointer',
+            '-D__ANDROID_API__={}'.format(self.ctx.ndk_api)]
+        if not clang:
+            cflags.append('-mandroid')
+        else:
+            cflags.append('-target ' + self.target)
+            toolchain = '{android_host}-{toolchain_version}'.format(
+                android_host=self.ctx.toolchain_prefix,
+                toolchain_version=self.ctx.toolchain_version)
+            toolchain = join(self.ctx.ndk_dir, 'toolchains', toolchain, 'prebuilt', 'linux-x86_64')
+            cflags.append('-gcc-toolchain {}'.format(toolchain))
+
+        env['CFLAGS'] = ' '.join(cflags)
         env['LDFLAGS'] = ' '
 
         sysroot = join(self.ctx._ndk_dir, 'sysroot')
@@ -45,6 +63,8 @@ def get_env(self, with_flags_in_cc=True):
             # https://android.googlesource.com/platform/ndk/+/ndk-r15-release/docs/UnifiedHeaders.md
             env['CFLAGS'] += ' -isystem {}/sysroot/usr/include/{}'.format(
                 self.ctx.ndk_dir, self.ctx.toolchain_prefix)
+            env['CFLAGS'] += ' -I{}/sysroot/usr/include/{}'.format(
+                self.ctx.ndk_dir, self.command_prefix)
         else:
             sysroot = self.ctx.ndk_platform
             env['CFLAGS'] += ' -I{}'.format(self.ctx.ndk_platform)
@@ -82,8 +102,20 @@ def get_env(self, with_flags_in_cc=True):
             env['NDK_CCACHE'] = self.ctx.ccache
             env.update({k: v for k, v in environ.items() if k.startswith('CCACHE_')})
 
-        cc = find_executable('{command_prefix}-gcc'.format(
-            command_prefix=command_prefix), path=environ['PATH'])
+        if clang:
+            llvm_dirname = split(
+                glob(join(self.ctx.ndk_dir, 'toolchains', 'llvm*'))[-1])[-1]
+            clang_path = join(self.ctx.ndk_dir, 'toolchains', llvm_dirname,
+                              'prebuilt', 'linux-x86_64', 'bin')
+            environ['PATH'] = '{clang_path}:{path}'.format(
+                clang_path=clang_path, path=environ['PATH'])
+            exe = join(clang_path, 'clang')
+            execxx = join(clang_path, 'clang++')
+        else:
+            exe = '{command_prefix}-gcc'.format(command_prefix=command_prefix)
+            execxx = '{command_prefix}-g++'.format(command_prefix=command_prefix)
+
+        cc = find_executable(exe, path=environ['PATH'])
         if cc is None:
             print('Searching path are: {!r}'.format(environ['PATH']))
             raise BuildInterruptingException(
@@ -93,20 +125,20 @@ def get_env(self, with_flags_in_cc=True):
                 'installed. Exiting.')
 
         if with_flags_in_cc:
-            env['CC'] = '{ccache}{command_prefix}-gcc {cflags}'.format(
-                command_prefix=command_prefix,
+            env['CC'] = '{ccache}{exe} {cflags}'.format(
+                exe=exe,
                 ccache=ccache,
                 cflags=env['CFLAGS'])
-            env['CXX'] = '{ccache}{command_prefix}-g++ {cxxflags}'.format(
-                command_prefix=command_prefix,
+            env['CXX'] = '{ccache}{execxx} {cxxflags}'.format(
+                execxx=execxx,
                 ccache=ccache,
                 cxxflags=env['CXXFLAGS'])
         else:
-            env['CC'] = '{ccache}{command_prefix}-gcc'.format(
-                command_prefix=command_prefix,
+            env['CC'] = '{ccache}{exe}'.format(
+                exe=exe,
                 ccache=ccache)
-            env['CXX'] = '{ccache}{command_prefix}-g++'.format(
-                command_prefix=command_prefix,
+            env['CXX'] = '{ccache}{execxx}'.format(
+                execxx=execxx,
                 ccache=ccache)
 
         env['AR'] = '{}-ar'.format(command_prefix)
@@ -123,12 +155,13 @@ def get_env(self, with_flags_in_cc=True):
         env['READELF'] = '{}-readelf'.format(command_prefix)
         env['NM'] = '{}-nm'.format(command_prefix)
 
-        hostpython_recipe = Recipe.get_recipe('hostpython2', self.ctx)
-
-        # This hardcodes python version 2.7, needs fixing
+        hostpython_recipe = Recipe.get_recipe(
+            'host' + self.ctx.python_recipe.name, self.ctx)
         env['BUILDLIB_PATH'] = join(
             hostpython_recipe.get_build_dir(self.arch),
-            'build', 'lib.linux-{}-2.7'.format(uname()[-1]))
+            'build', 'lib.linux-{}-{}'.format(
+                uname()[-1], self.ctx.python_recipe.major_minor_version_string)
+        )
 
         env['PATH'] = environ['PATH']
 
@@ -147,12 +180,18 @@ class ArchARM(Arch):
     command_prefix = 'arm-linux-androideabi'
     platform_dir = 'arch-arm'
 
+    @property
+    def target(self):
+        target_data = self.command_prefix.split('-')
+        return '-'.join(
+            ['armv7a', 'none', target_data[1], target_data[2]])
+
 
 class ArchARMv7_a(ArchARM):
     arch = 'armeabi-v7a'
 
-    def get_env(self, with_flags_in_cc=True):
-        env = super(ArchARMv7_a, self).get_env(with_flags_in_cc)
+    def get_env(self, with_flags_in_cc=True, clang=False):
+        env = super(ArchARMv7_a, self).get_env(with_flags_in_cc, clang=clang)
         env['CFLAGS'] = (env['CFLAGS'] +
                          (' -march=armv7-a -mfloat-abi=softfp '
                           '-mfpu=vfp -mthumb'))
@@ -166,8 +205,8 @@ class Archx86(Arch):
     command_prefix = 'i686-linux-android'
     platform_dir = 'arch-x86'
 
-    def get_env(self, with_flags_in_cc=True):
-        env = super(Archx86, self).get_env(with_flags_in_cc)
+    def get_env(self, with_flags_in_cc=True, clang=False):
+        env = super(Archx86, self).get_env(with_flags_in_cc, clang=clang)
         env['CFLAGS'] = (env['CFLAGS'] +
                          ' -march=i686 -mtune=intel -mssse3 -mfpmath=sse -m32')
         env['CXXFLAGS'] = env['CFLAGS']
@@ -180,8 +219,8 @@ class Archx86_64(Arch):
     command_prefix = 'x86_64-linux-android'
     platform_dir = 'arch-x86_64'
 
-    def get_env(self, with_flags_in_cc=True):
-        env = super(Archx86_64, self).get_env(with_flags_in_cc)
+    def get_env(self, with_flags_in_cc=True, clang=False):
+        env = super(Archx86_64, self).get_env(with_flags_in_cc, clang=clang)
         env['CFLAGS'] = (env['CFLAGS'] +
                          ' -march=x86-64 -msse4.2 -mpopcnt -m64 -mtune=intel')
         env['CXXFLAGS'] = env['CFLAGS']
@@ -194,8 +233,8 @@ class ArchAarch_64(Arch):
     command_prefix = 'aarch64-linux-android'
     platform_dir = 'arch-arm64'
 
-    def get_env(self, with_flags_in_cc=True):
-        env = super(ArchAarch_64, self).get_env(with_flags_in_cc)
+    def get_env(self, with_flags_in_cc=True, clang=False):
+        env = super(ArchAarch_64, self).get_env(with_flags_in_cc, clang=clang)
         incpath = ' -I' + join(dirname(__file__), 'includes', 'arm64-v8a')
         env['EXTRA_CFLAGS'] = incpath
         env['CFLAGS'] += incpath

pythonforandroid/bootstrap.py

@@ -269,14 +269,13 @@ def strip_libraries(self, arch):
             return
         strip = sh.Command(strip)
 
-        if self.ctx.python_recipe.name == 'python2':
-            filens = shprint(sh.find, join(self.dist_dir, 'private'),
-                             join(self.dist_dir, 'libs'),
-                             '-iname', '*.so', _env=env).stdout.decode('utf-8')
-        else:
-            filens = shprint(sh.find, join(self.dist_dir, '_python_bundle', '_python_bundle', 'modules'),
-                             join(self.dist_dir, 'libs'),
-                             '-iname', '*.so', _env=env).stdout.decode('utf-8')
+        libs_dir = join(self.dist_dir, '_python_bundle',
+                        '_python_bundle', 'modules')
+        if self.ctx.python_recipe.name == 'python2legacy':
+            libs_dir = join(self.dist_dir, 'private')
+        filens = shprint(sh.find, libs_dir, join(self.dist_dir, 'libs'),
+                         '-iname', '*.so', _env=env).stdout.decode('utf-8')
+
         logger.info('Stripping libraries in private dir')
         for filen in filens.split('\n'):
             try:

pythonforandroid/bootstraps/common/build/build.py

@@ -46,7 +46,6 @@ def get_bootstrap_name():
 # Try to find a host version of Python that matches our ARM version.
 PYTHON = join(curdir, 'python-install', 'bin', 'python.host')
 if not exists(PYTHON):
-    print('Could not find hostpython, will not compile to .pyo (this is normal with python3)')
     PYTHON = None
 
 BLACKLIST_PATTERNS = [
@@ -151,7 +150,6 @@ def make_python_zip():
 
     if not exists('private'):
         print('No compiled python is present to zip, skipping.')
-        print('this should only be the case if you are using the CrystaX python')
         return
 
     global python_files

pythonforandroid/bootstraps/common/build/jni/application/src/start.c

@@ -104,6 +104,10 @@ int main(int argc, char *argv[]) {
   LOGP(env_argument);
   chdir(env_argument);
 
+#if PY_MAJOR_VERSION < 3
+  Py_NoSiteFlag=1;
+#endif
+
   Py_SetProgramName(L"android_python");
 
 #if PY_MAJOR_VERSION >= 3
@@ -144,10 +148,6 @@ int main(int argc, char *argv[]) {
     #if PY_MAJOR_VERSION >= 3
         wchar_t *wchar_paths = Py_DecodeLocale(paths, NULL);
         Py_SetPath(wchar_paths);
-    #else
-        char *wchar_paths = paths;
-        LOGP("Can't Py_SetPath in python2, so crystax python2 doesn't work yet");
-        exit(1);
     #endif
 
         LOGP("set wchar paths...");
@@ -161,6 +161,12 @@ int main(int argc, char *argv[]) {
   Py_Initialize();
 
 #if PY_MAJOR_VERSION < 3
+  // Can't Py_SetPath in python2 but we can set PySys_SetPath, which must
+  // be applied after Py_Initialize rather than before like Py_SetPath
+  #if PY_MICRO_VERSION >= 15
+    // Only for python native-build
+    PySys_SetPath(paths);
+  #endif
   PySys_SetArgv(argc, argv);
 #endif
 
@@ -183,7 +189,9 @@ int main(int argc, char *argv[]) {
    */
   PyRun_SimpleString("import sys, posix\n");
   if (dir_exists("lib")) {
-    /* If we built our own python, set up the paths correctly */
+    /* If we built our own python, set up the paths correctly.
+     * This is only the case if we are using the python2legacy recipe
+     */
     LOGP("Setting up python from ANDROID_APP_PATH");
     PyRun_SimpleString("private = posix.environ['ANDROID_APP_PATH']\n"
                        "argument = posix.environ['ANDROID_ARGUMENT']\n"

pythonforandroid/bootstraps/pygame/build/jni/application/Android.mk

@@ -18,9 +18,7 @@ LOCAL_CFLAGS := $(foreach D, $(APP_SUBDIRS), -I$(LOCAL_PATH)/$(D)) \
 				-I$(LOCAL_PATH)/../jpeg \
 				-I$(LOCAL_PATH)/../intl \
 				-I$(LOCAL_PATH)/.. \
-                -I$(LOCAL_PATH)/../../../../other_builds/$(PYTHON2_NAME)/$(ARCH)/python2/python-install/include/python2.7
-				# -I$(LOCAL_PATH)/../../../../python-install/include/python2.7
-				# -I$(LOCAL_PATH)/../../../build/python-install/include/python2.7
+                -I$(LOCAL_PATH)/../../../../other_builds/$(MK_PYTHON_INCLUDE_ROOT)
 
 
 LOCAL_CFLAGS += $(APPLICATION_ADDITIONAL_CFLAGS)
@@ -40,7 +38,7 @@ LOCAL_LDLIBS := -lpython2.7 -lGLESv1_CM -ldl -llog -lz
 
 # AND: Another hardcoded path that should be templated
 # AND: NOT TEMPALTED! We can use $ARCH
-LOCAL_LDFLAGS += -L$(LOCAL_PATH)/../../../../other_builds/$(PYTHON2_NAME)/$(ARCH)/python2/python-install/lib $(APPLICATION_ADDITIONAL_LDFLAGS)
+LOCAL_LDFLAGS += -L$(LOCAL_PATH)/../../../../other_builds/$(MK_PYTHON_LINK_ROOT) $(APPLICATION_ADDITIONAL_LDFLAGS)
 
 LIBS_WITH_LONG_SYMBOLS := $(strip $(shell \
 	for f in $(LOCAL_PATH)/../../libs/$ARCH/*.so ; do \