Cython projects that don't need any special option should work without a CythonRecipe

[ghost]

Versions

• Python: 3.x
• OS: any should be affected
• Kivy: not used
• Cython: any should be affected

Description

Cython projects that don't need any special option should work without recipe. Right now, an empty Cython recipe with zero useful information is needed even in case no patching, no special options, no anything needs to be done, because otherwise p4a apparently won't set up things like the Cython being available in the first place correctly. I think that's unnecessary and shouldn't be required, since it makes using projects with p4a that don't require any of this way more complicated (since they need a recipe when they would work fine without, if p4a actually had reasonable default behavior).

In general, I find over-reliance on recipes quite a roadblock for external projects that are well-tested with p4a but developed independently, since they would somehow need to make users install recipes or get them into the main repo which just eats away at everyone's time...

buildozer.spec

not used, but should be irrelevant. I tested with a p4a one-liner with two projects of mine as requirements, one of them requiring Cython: --requirements=python3crystax,https://github.com/JonasT/wobblui/archive/master.zip,https://github.com/JonasT/nettools/archive/master.zip

Logs

Basically, the Cython part of the setup.py just plain doesn't work as you can see here (Cython doesn't seem to be present in the virtual env p4a sets up in the first place, if not using a CythonRecipe):

p4a apk --minsdk=18 --release --debug --private /my-app/ --bootstrap=sdl2 --requirements=python3crystax,https://github.com/JonasT/wobblui/archive/master.zip,https://github.com/JonasT/nettools/archive/master.zip --arch=armeabi-v7a
[INFO]:    Will compile for the following archs: armeabi-v7a
[INFO]:    Found Android API target in $ANDROIDAPI
[INFO]:    Available Android APIs are (19)
[INFO]:    Requested API target 19 is available, continuing.
[INFO]:    Found NDK dir in $ANDROIDNDK
[INFO]:    Got NDK version from $ANDROIDNDKVER
[INFO]:    Got Android NDK version from the NDK dir: it is 10.3.2
[INFO]:    Using Crystax NDK 10.3.2
[INFO]:    Found virtualenv at /usr/bin/virtualenv
[INFO]:    ccache is missing, the build will not be optimized in the future.
[INFO]:    Found the following toolchain versions: ['4.9', '5', 'clang3.6', 'clang3.7']
[INFO]:    Picking the latest gcc toolchain, here 5
[INFO]:    No existing dists meet the given requirements!
[INFO]:    No dist exists that meets your requirements, so one will be built.
[INFO]:    Found a single valid recipe set: ['hostpython3crystax', 'https://github.com/JonasT/nettools/archive/master.zip', 'https://github.com/JonasT/wobblui/archive/master.zip', 'python3crystax']
[INFO]:    The selected bootstrap is sdl2_gradle
[INFO]:    # Creating dist with sdl2_gradle bootstrap
[INFO]:    Dist will have name unnamed_dist_3 and recipes (python3crystax, https://github.com/JonasT/wobblui/archive/master.zip, https://github.com/JonasT/nettools/archive/master.zip)
[INFO]:    Dist will also contain modules (https://github.com/JonasT/nettools/archive/master.zip, https://github.com/JonasT/wobblui/archive/master.zip) installed from pip
[DEBUG]:   -> running cp -r /usr/local/lib/python3.6/dist-packages/pythonforandroid/bootstraps/sdl2/build /root/.local/share/python-for-android/build/bootstrap_builds/sdl2_gradle-python3crystax
[INFO]:    -> directory context /root/.local/share/python-for-android/build/bootstrap_builds/sdl2_gradle-python3crystax
[INFO]:    <- directory context /
[INFO]:    Recipe build order is ['hostpython3crystax', 'python3crystax']
[INFO]:    The requirements (https://github.com/JonasT/nettools/archive/master.zip, https://github.com/JonasT/wobblui/archive/master.zip) were not found as recipes, they will be installed with pip.
[INFO]:    # Downloading recipes 
[INFO]:    Downloading hostpython3crystax
[INFO]:    Skipping hostpython3crystax download as no URL is set
[INFO]:    Downloading python3crystax
[DEBUG]:   -> running mkdir -p /root/.local/share/python-for-android/packages/python3crystax
[INFO]:    -> directory context /root/.local/share/python-for-android/packages/python3crystax
[DEBUG]:   -> running basename 
[DEBUG]:   	
[DEBUG]:   Downloading python3crystax from 
[DEBUG]:   -> running rm -f .mark-
[DEBUG]:   -> running touch .mark-
[INFO]:    <- directory context /
[INFO]:    # Building all recipes for arch armeabi-v7a
[INFO]:    # Unpacking recipes
[INFO]:    Unpacking hostpython3crystax for armeabi-v7a
[INFO]:    Skipping hostpython3crystax unpack as no URL is set
[INFO]:    Unpacking python3crystax for armeabi-v7a
[DEBUG]:   -> running basename 
[DEBUG]:   	
[INFO]:    -> directory context /root/.local/share/python-for-android/build/other_builds/python3crystax-version3.6/armeabi-v7a
[DEBUG]:   -> running cp -Rv /root/.local/share/python-for-android/packages/python3crystax/.mark- /root/.local/share/python-for-android/build/other_builds/python3crystax-version3.6/armeabi-v7a/python3crystax
[DEBUG]:   	'/root/.local/share/python-for-android/packages/python3crystax/.mark-' -> '/root/.local/share/python-for-android/build/other_builds/python3crystax-version3.6/armeabi-v7a/python3crystax/.mark-'
[INFO]:    <- directory context /
[INFO]:    # Prebuilding recipes
[INFO]:    Prebuilding hostpython3crystax for armeabi-v7a
[INFO]:    hostpython3crystax has no prebuild_armeabi_v7a, skipping
[INFO]:    Prebuilding python3crystax for armeabi-v7a
[INFO]:    python3crystax has no prebuild_armeabi_v7a, skipping
[INFO]:    # Building recipes
[INFO]:    Building hostpython3crystax for armeabi-v7a
[DEBUG]:   -> running mkdir -p /root/.local/share/python-for-android/build/other_builds/hostpython3crystax/desktop/hostpython3crystax/build
[DEBUG]:   -> running ln -sf /usr/bin/python3.6 /root/.local/share/python-for-android/build/other_builds/hostpython3crystax/desktop/hostpython3crystax/hostpython
[INFO]:    Building python3crystax for armeabi-v7a
[INFO]:    The NDK does not have a prebuilt Python 3.6, trying to obtain one.
[DEBUG]:    + temp directory used /tmp/tmpf67icxf5
[INFO]:    Downloading python3crystax from https://github.com/inclement/crystax_python_builds/releases/download/0.1/crystax_python_3.6_armeabi_armeabi-v7a.tar.gz
[DEBUG]:   -> running tar xf /tmp/tmpf67icxf5/downloaded_python --directory /crystax-ndk/sources/python
[DEBUG]:    - temp directory deleted /tmp/tmpf67icxf5
[INFO]:    # Biglinking object files
[INFO]:    NDK is crystax, skipping biglink (will this work?)
[INFO]:    # Postbuilding recipes
[INFO]:    Postbuilding hostpython3crystax for armeabi-v7a
[INFO]:    Postbuilding python3crystax for armeabi-v7a
[INFO]:    # Installing pure Python modules
[INFO]:    The requirements (https://github.com/JonasT/nettools/archive/master.zip, https://github.com/JonasT/wobblui/archive/master.zip) don't have recipes, attempting to install them with pip
[INFO]:    If this fails, it may mean that the module has compiled components and needs a recipe.
[INFO]:    -> directory context /root/.local/share/python-for-android/build
[DEBUG]:   -> running virtualenv --python=python2.7 venv
[DEBUG]:   	Running virtualenv with interpreter /usr/bin/python2.7
[DEBUG]:   	New python executable in /root/.local/share/python-for-android/build/venv/bin/python2.7
[DEBUG]:   	Also creating executable in /root/.local/share/python-for-android/build/venv/bin/python
[DEBUG]:   	Installing setuptools, pkg_resources, pip, wheel...done.
[INFO]:    Creating a requirements.txt file for the Python modules
[INFO]:    Installing Python modules with pip
[INFO]:    If this fails with a message about /bin/false, this probably means the package cannot be installed with pip as it needs a compilation recipe.
[DEBUG]:   -> running bash -c source venv/bin/activate && env CC=/bin/false CXX=/bin/false PYTHONPATH=/root/.local/share/python-for-android/build/python-installs/unnamed_dist_3 pip install --target '/root/.local/share/python-for-android/build/python-installs/unnamed_dist_3' --no-deps -r requirements.txt
[DEBUG]:   	Collecting https://github.com/JonasT/nettools/archive/master.zip (from -r requirements.txt (line 1))
[DEBUG]:   	  Downloading https://github.com/JonasT/nettools/archive/master.zip
     \ 143kB 955kB/s
[DEBUG]:   	Collecting https://github.com/JonasT/wobblui/archive/master.zip (from -r requirements.txt (line 2))
[DEBUG]:   	  Downloading https://github.com/JonasT/wobblui/archive/master.zip
     | 2.0MB 18.0MB/s
[DEBUG]:   	    Complete output from command python setup.py egg_info:
[DEBUG]:   	    Traceback (most recent call last):
[DEBUG]:   	      File "<string>", line 1, in <module>
[DEBUG]:   	      File "/tmp/pip-req-build-CE0nSM/setup.py", line 5, in <module>
[DEBUG]:   	        from Cython.Build import cythonize
[DEBUG]:   	    ImportError: No module named Cython.Build
[DEBUG]:   	
[DEBUG]:   	    ----------------------------------------
[DEBUG]:   	Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-CE0nSM/
Exception in thread background thread for pid 843:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/sh.py", line 1540, in wrap
    fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/sh.py", line 2459, in background_thread
    handle_exit_code(exit_code)
  File "/usr/local/lib/python3.6/dist-packages/sh.py", line 2157, in fn
    return self.command.handle_command_exit_code(exit_code)
  File "/usr/local/lib/python3.6/dist-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /bin/bash -c source venv/bin/activate && env CC=/bin/false CXX=/bin/false PYTHONPATH=/root/.local/share/python-for-android/build/python-installs/unnamed_dist_3 pip install --target '/root/.local/share/python-for-android/build/python-installs/unnamed_dist_3' --no-deps -r requirements.txt

  STDOUT:
Collecting https://github.com/JonasT/nettools/archive/master.zip (from -r requirements.txt (line 1))
  Downloading https://github.com/JonasT/nettools/archive/master.zip
     \ 143kB 955kB/s
Collecting https://github.com/JonasT/wobblui/archive/master.zip (from -r requirements.txt (line 2))
  Downloading https://github.com/JonasT/wobblui/archive/master.zip
     | 2.0MB 18.0MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-CE0nSM/setup.py", line 5, in <module>
        from Cython.Build import cythonize
    ImportError: No module named Cython.Build
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-CE0nSM/


  STDERR:


Traceback (most recent call last):
  File "/usr/local/bin/p4a", line 11, in <module>
    load_entry_point('python-for-android==0.6.0', 'console_scripts', 'p4a')()
  File "/usr/local/lib/python3.6/dist-packages/pythonforandroid/toolchain.py", line 999, in main
    ToolchainCL()
  File "/usr/local/lib/python3.6/dist-packages/pythonforandroid/toolchain.py", line 532, in __init__
    getattr(self, args.subparser_name.replace('-', '_'))(args)
  File "/usr/local/lib/python3.6/dist-packages/pythonforandroid/toolchain.py", line 145, in wrapper_func
    build_dist_from_args(ctx, dist, args)
  File "/usr/local/lib/python3.6/dist-packages/pythonforandroid/toolchain.py", line 189, in build_dist_from_args
    build_recipes(build_order, python_modules, ctx)
  File "/usr/local/lib/python3.6/dist-packages/pythonforandroid/build.py", line 599, in build_recipes
    run_pymodules_install(ctx, python_modules)
  File "/usr/local/lib/python3.6/dist-packages/pythonforandroid/build.py", line 640, in run_pymodules_install
    ).format(ctx.get_site_packages_dir()))
  File "/usr/local/lib/python3.6/dist-packages/pythonforandroid/logger.py", line 176, in shprint
    for line in output:
  File "/usr/local/lib/python3.6/dist-packages/sh.py", line 863, in next
    self.wait()
  File "/usr/local/lib/python3.6/dist-packages/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/local/lib/python3.6/dist-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /bin/bash -c source venv/bin/activate && env CC=/bin/false CXX=/bin/false PYTHONPATH=/root/.local/share/python-for-android/build/python-installs/unnamed_dist_3 pip install --target '/root/.local/share/python-for-android/build/python-installs/unnamed_dist_3' --no-deps -r requirements.txt

  STDOUT:
Collecting https://github.com/JonasT/nettools/archive/master.zip (from -r requirements.txt (line 1))
  Downloading https://github.com/JonasT/nettools/archive/master.zip
     \ 143kB 955kB/s
Collecting https://github.com/JonasT/wobblui/archive/master.zip (from -r requirements.txt (line 2))
  Downloading https://github.com/JonasT/wobblui/archive/master.zip
     | 2.0MB 18.0MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-CE0nSM/setup.py", line 5, in <module>
        from Cython.Build import cythonize
    ImportError: No module named Cython.Build
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-CE0nSM/


  STDERR:

[ghost]

Ok since this turned out to be way more horrible than anticipated, let me share my suffering with you with some of the things I had to do to get my project to build:

1. Write and test a recipe installer: https://github.com/JonasT/wobblui/blob/master/src/wobblui/p4arecipes/__init__.py

2. Write recipes that don't actually have useful additional information (like custom build flags): https://github.com/JonasT/wobblui/blob/master/src/wobblui/p4arecipes/wobblui.py and another

3. Make a recipe for my local project with a placeholder, because apparently I need to specify a remote URL even if I don't want to

4. Add the following steps to my Dockerfile to install the recipes, and insert placeholder for local project (I wanted to put a file URL into the placeholder, but that doesn't work, see file:// urls in a recipe are not processed correctly #1407 )

   RUN pip3 install -U Pillow==5.1.0 && pip3 install -U https://github.com/JonasT/nettools/archive/master.zip && pip3 install -U https://github.com/JonasT/wobblui/archive/master.zip && apt update &&  apt install -y libsdl2-2.0-0 libsdl2-image-2.0-0 libsdl2-ttf-2.0-0
   RUN python3 -c "from wobblui.p4arecipes import install; install();"
   RUN mkdir /myapp-recipe/
   COPY p4arecipes/myrecipe.py /myapp-recipe/myrecipe.py
   RUN apt install -y nginx
   RUN sed -i 's/{INSERT_URL}/http:\/\/127.0.0.1\/archive.zip/g' /myapp-recipe/myrecipe.py
   RUN python3 -c "from wobblui.p4arecipes import install; install(from_source='/myapp-recipe/');
   

   Note this step includes installing a webserver. Ugh.

5. Add this to my container launch script:

   os.system("mkdir -p /var/www/html/")
   os.system("service nginx start")
   

6. Add this to my build start script:

   rm -f /var/www/html/archive.zip
   cd / && zip -r -9 /var/www/html/archive.zip my-app-lib/
   

I can't even imagine what I would have done without Docker! Install obscure recipes and a webserver and somehow hope I don't trash the entire install on my machine?? How are other people supposed to do this?

If recipes weren't necessary for Cython, I would have been done with this: --requirements=https://github.com/JonasT/wobblui/archive/master.zip - done.

Right now, the recipes are an absolute disaster for external projects, because:

1. The recipes can't be picked up automatically from the repo, you have to get the user to somehow fetch them separately, e.g. with an installer as written above

2. The recipes cannot not contain an URL, which means the user needs to hardcode the proper URL or it needs to be inserted on-the-fly if they e.g. want to change a branch

3. The recipes cannot contain a file URL, so even if dynamically inserted, it's not possible to build a local copy of the project - which is usually what you want if you just fetched the whole external lib in a repository and installed the recipe from that, since you might want to play around with it & modify things, or change the local branch etc.

4. The recipes aren't automatically used if a package with that name is in the requirements. Basically if the requirements contain an URL to a package which pip recognizes & installs with a specific name, p4a won't be smart enough to match this to the recipe, but instead the exact name needs to be in the requirements list, which is simply stupid for anything that isn't published exactly like that on pypi, because it unnecessarily obscures the install source.

Given this situation, it would help improve the situation massively if Cython with default settings could be used without relying on this recipe mess.

Disclaimer: Maybe I missed something and all of this is a lot easier. That'd be nice. But right now, the mess above is the only way I managed to make it build.

[inclement]

Rather than setting up a webserver, it would be much simpler to write a recipe class that uses local files. There's already a simple mixin that does this, IncludedFilesBehaviour, which is only used internally by the android recipe.

The method for using it is something like:

class YourRecipe(IncludedFilesBehaviour, CythonRecipe):
    src_filename = '/path/to/module/dir'
    url = None  # May not need to set this explicitly

It would be nice to support a more generic api, if it's possible to easily get pip to resolve paths for us then that would be ideal.

[inclement]

Thinking about it, I think it might be fairly easy to use pip for much more of the dependency resolution than is currently the case. I'll maybe look into it once the python3 stuff is working.

[AndreMiras]

Yes I also feel there're way to lower our current boilerplate a lot. But I want us to make sure we don't break things up, e.g. via integration testing.

[ghost]

@AndreMiras yeah I think the main problem at hand here is still that Cython projects require a recipe in the first place. Because it is feasible with the additional use of ctypes to completely avoid any special options even when interacting with libs (because with ctypes, no linking is necessary) while taking most of the performance gains of Cython - so with a bit of care, the need of any special options can be avoided in projects that care to do so. But unless p4a uses proper defaults without a recipe, that gives no real benefits...

[ghost]

@inclement that is useful to know, thanks! However, as suggested in #1407 I think the standard recipe should just support file URLs and/or file paths out of the box. Everything else is just again overcomplicating things and quite non-intuitive. Why should I need a mixin class for something that is completely unambiguous regarding my intent like the file URL? (it's not even a different type of thing, it's still a completely bog standard URL)

[ghost]

I made a pull request to fix this: #1483