[revamp] Can't load unicodedata module

[str4d]

Just to test it out, I tried building a Twisted distribution without any recipes, simply passing in --requirements twisted and letting pip handle the dependencies. It actually managed to install fine, but another error turned up (output trimmed):

$ p4a apk --private src/ --package org.example.twistedtest --name "Hello Twisted" --version 1.0 --requirements twisted --dist_name twisted
[INFO]:    # Creating dist with sdl2 bootstrap
[INFO]:    # Downloading recipes 
[INFO]:    # Building all recipes for arch armeabi
[INFO]:    # Unpacking recipes
[INFO]:    # Prebuilding recipes
[INFO]:    # Building recipes
[INFO]:    # Biglinking object files
[INFO]:    # Postbuilding recipes
[INFO]:    # Installing pure Python modules
[INFO]:    The requirements (twisted) don't have recipes, attempting to install them with pip
[INFO]:    If this fails, it may mean that the module has compiled components and needs a recipe.
[INFO]:    -> directory context /home/str4d/.local/share/python-for-android/build
[INFO]:    running virtualenv --python=python2.7 venv
           working ... Installing setuptools, pip, wheel...done. ...                                               
[INFO]:    Creating a requirements.txt file for the Python modules
[INFO]:    Installing Python modules with pip
[INFO]:    If this fails with a message about /bin/false, this probably means the package cannot be installed with pip as it needs a compilation recipe.
[INFO]:    running bash -c source venv/bin/activate && env CC=/bin/false CXX=/bin/falsePYTHONPATH= pip install ... (and 132 more)
           working ... Target directory /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/ ...
[INFO]:    <- directory context /home/str4d/dev/git/tahoe-lafs/kivytest
[INFO]:    # Creating Android project from build and sdl2 bootstrap
[INFO]:    This currently just copies the SDL2 build stuff straight from the build dir.
[INFO]:    running rm -rf /home/str4d/.local/share/python-for-android/dists/twisted
[INFO]:    running cp -r /home/str4d/.local/share/python-for-android/build/bootstrap_builds/sdl2 /home/str4d/.l... (and 43 more)
[INFO]:    -> directory context /home/str4d/.local/share/python-for-android/dists/twisted
[INFO]:    <- directory context /home/str4d/dev/git/tahoe-lafs/kivytest
[INFO]:    -> directory context /home/str4d/.local/share/python-for-android/dists/twisted
[INFO]:    Copying python distribution
[INFO]:    running mkdir private
[INFO]:    running mkdir assets
[INFO]:    running python.host -OO -m compileall /home/str4d/.local/share/python-for-android/build/python-insta... (and 11 more)
           working ... Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/share/man/man1 ... ...Traceback (most recent call last):
  File "/home/str4d/.local/bin/p4a", line 11, in <module>
    sys.exit(main())
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2797, in main
    ToolchainCL()
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2442, in __init__
    getattr(self, args.command)(unknown)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 169, in wrapper_func
    args = build_dist_from_args(ctx, dist, args)
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 2223, in build_dist_from_args
    ctx.bootstrap.run_distribute()
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/bootstraps/sdl2/__init__.py", line 34, in run_distribute
    self.ctx.get_python_install_dir())
  File "/home/str4d/.local/lib/python2.7/site-packages/pythonforandroid/toolchain.py", line 129, in shprint
    for line in output:
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 565, in next
    self.wait()
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "/home/str4d/.local/lib/python2.7/site-packages/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_1: 

  RAN: '/home/str4d/.local/share/python-for-android/build/python-installs/twisted/bin/python.host -OO -m compileall /home/str4d/.local/share/python-for-android/build/python-installs/twisted'

  STDOUT:
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/bin ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/include ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/include/python2.7 ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/pkgconfig ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7 ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/... (22027 more, please see e.stdout)

  STDERR:

Modifying sh.py to print out e.stdout uncovers the following (trimmed):

Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/tap ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test ...
Compiling /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_amp.py ...
SyntaxError: ("(unicode error) \\N escapes not supported (can't load unicodedata module)", ('/home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_amp.py', 2732, None, '"multiple": [u"\\N{SNOWMAN}", u"Hello", u"world"]}\n'))

Compiling /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_compat.py ...
SyntaxError: ("(unicode error) \\N escapes not supported (can't load unicodedata module)", ('/home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_compat.py', 596, None, 'UnicodeError, networkString, u"\\N{SNOWMAN}".encode(\'utf-8\'))\n'))

Compiling /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_log.py ...
SyntaxError: ("(unicode error) \\N escapes not supported (can't load unicodedata module)", ('/home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_log.py', 347, None, "str(u'\\N{VULGAR FRACTION ONE HALF}')\n"))

Compiling /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_process.py ...
SyntaxError: ("(unicode error) \\N escapes not supported (can't load unicodedata module)", ('/home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_process.py', 656, None, "badUnicode = u'\\N{SNOWMAN}'\n"))

Compiling /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_sslverify.py ...
SyntaxError: ("(unicode error) \\N escapes not supported (can't load unicodedata module)", ('/home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/test/test_sslverify.py', 1638, None, 'hello = u"h\\N{LATIN SMALL LETTER A WITH ACUTE}llo.example.com"\n'))

Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/topfiles ...
Listing /home/str4d/.local/share/python-for-android/build/python-installs/twisted/lib/python2.7/site-packages/twisted/trial ...

So setuptools, zope.interface and twisted are all successfully installed as Python modules, but when it comes to compiling twisted fails because the compiler can't load the unicodedata module.

[inclement]

You can also use --debug to print all the sh stdout.

[inclement]

I don't think twisted will work without a recipe in any case, because the android build env vars won't be set, but I'll keep an eye out for this.

[inclement]

I guess this is the same problem as with _io.so and that your new recipes resolve the problem, so I'll close this issue. Let me know if I'm wrong about that.