Proof of concept - A bunch of tests for library recipes #1982

opacam · 2019-09-12T12:05:38Z

The main idea is to test as many recipes as we can with the less code possible and without creating any file/directory so our tests can be performed as fast as possible (all this tests will only add between 2 and 3 seconds to our CI tests and will cover almost 100% of the code for each tested recipe)

To do so, we create a couple of modules:

tests.recipe_ctx: allow us to create a proper Context to test our recipes
tests.recipe_lib_test: which holds some base classes to be used to test a recipe depending on the build method used. For starters we introduce two kind of base classes:
- BaseTestForMakeRecipe: To test an standard library build (this will iinclude the recipes which requires the classical build commandsconfigure/make)
- BaseTestForCmakeRecipe: To test an library recipe which is compiled with cmake

We also refactor the existing recipes tests, so we can remove some lines in there...the ones that creates a Context.

Notes:

I'm aware that we could go a little further with the testing...but for starters I thing it's fine...we can do it later
This will cover most of the library recipes, but still we will have to cover a little more
I tried to create a generic tests depending on the build system, I'm sure that we could do something similar with PythonRecipes, BootstrapNDKRecipes...)
This could be split into multiple dependant PRs but, first let's see if the general idea is good enough ...then we will deal with the details if convenient 😄
@AndreMiras, sorry man, this time I made use of Python Inheritance...I just feel that is the best we can do in this case, but I'm always open to other solutions 😉

AndreMiras · 2019-09-14T10:02:48Z

Absolutely awesome 💕 I can't wait to review!
I actually wanted to do so for so long, I started in a local branch and never finished. I'll try to review ASAP and most likely start now.

AndreMiras

Man, you went again full nuts and you did an awesome work.
I wanted to do something similar for long, but you actually nailed it 👏
Core devs can retire 😂
I wrote some comments, but it's mostly minor remarks or questions, so feel free to merge already.

AndreMiras · 2019-09-14T10:04:24Z

pythonforandroid/recipes/jpeg/__init__.py

-                        toolchain=env['TOOLCHAIN']),
+                    '-DCMAKE_C_COMPILER={cc}'.format(cc=arch.get_clang_exe()),
+                    '-DCMAKE_CXX_COMPILER={cc_plus}'.format(
+                        cc_plus=arch.get_clang_exe(plus_plus=True)),


So you actually did find a bug already while covering it with your tests? 👏

Aaa...this is more a little refactor (since at the time we changed JPEG to be built with Cmake, the function arch.get_clang_exe does not existed, we were forced to got these values in get_recipe_env, since our default compiler was gcc, but with the NDK-r19 migration thing, we use clang, so we can make use of arch.get_clang_exe and remove some lines 😄)

AndreMiras · 2019-09-14T10:15:54Z

tests/recipes/recipe_lib_test.py

+            print(f"We are testing recipe: {obj.recipe_name}")
+            obj.recipes.append(obj.recipe_name)
+            obj.recipe_build_order.insert(1, obj.recipe_name)
+            return obj


I guess you're using the __new__ over the __init__() for a reason, but I have to ask. Why can't this be dealt with it? Something like:

def __init__(self, *args, **kwargs): if self.recipe_name is not None: ...

Aaa... I faced several issues along the way with this: duplicated tests, tests not initialized...:laughing:..finally using __new__ is what it worked and I forgot about this lines until now...

I just made a test with __init__ and it also can work so I'm considering to change __new__ with __init__ (I think it's better to not mess with __new__ if we can do it with __init__)

...so...@AndreMiras, let's go with __init__?

I think it's better to not mess with __new__ if we can do it with __init__

That recaps my feeling pretty well. I'm not against __new__ in general, but it needs to be clearly justified. But yes again if you can't make it work consistently with __init__ then I'm go for it.

AndreMiras · 2019-09-14T10:19:21Z

tests/recipes/recipe_lib_test.py

+        obj = super(BaseTestForMakeRecipe, cls).__new__(cls)
+        if obj.recipe_name is not None:
+            print(f"We are testing recipe: {obj.recipe_name}")
+            obj.recipes.append(obj.recipe_name)


It always confuse me how Python can deal with class attributes vs instance attributes.
For instance initially recipes is a class attribute, right? Then we access it from the instance to alter it. Would this modify also the class attribute or just for this instance? 😄

Well, class attributes are supposed to be shared for all instances...so I guess that the answer is yes, we are also modifying class attributes (the thing is that recipes will grow up as long as we perform the tests...but shouldn't affect the test itself at all)

Exactly, in this case why not making it explicit and access it through cls? That would avoid confusion. Do you see what I mean? This is rather nitpicking, but at the same time I personally feel it's important to get the habit about thinking/clarifying theses when possible since it can cause nasty side effects sometimes

Yes you are right, I'm preparing another PR to address all this we recently talk that cannot be done here. Later I will push it.

Thanks again @AndreMiras, your reviews always makes the code better 👍

AndreMiras · 2019-09-14T10:22:21Z

tests/recipes/recipe_lib_test.py

+            self.recipe.build_arch(self.arch)
+
+            # make sure that the mocked methods are actually called
+            mock_glob.assert_called()


Only the call needs to be in the context manager. The rest can be outside

AndreMiras · 2019-09-14T10:23:14Z

tests/recipes/recipe_lib_test.py

+            self.recipe.build_arch(self.arch)
+
+            # make sure that the mocked methods are actually called
+            mock_glob.assert_called()


Same here, you can take this one level down (if you want). It doesn't need to be in the context manager

AndreMiras · 2019-09-14T10:24:16Z

tests/recipes/test_freetype.py

+from tests.recipes.recipe_lib_test import BaseTestForMakeRecipe
+
+
+class TestFreetypeRecipe(BaseTestForMakeRecipe, unittest.TestCase):


Why do we need to inherit from TestCase suddenly?

We actually need it because nor RecipeCtx nor BaseTestForMakeRecipe inherits from unittest.TestCase

<nitpick-troll>
What is TestCase providing that you need in this case? Would tests fail or not run if we drop it?
We don't even use setup/teardown here. Yes I know it's hard to say unittest.TestCase goodbye 😂
Honestly, feel free to not address that comment, I'm just challenging your habits 😉
</nitpick-troll>

Jajajaja...No man, we need this or the tests won't run...we are using setUp/teardown for all library recipes we introduced/modified, since all of them inherits from RecipeCtx...or directly or via BaseTestForMakeRecipe/BaseTestForCmakeRecipe

...this fuc*** Python Inheritance makes the things hard to track sometimes...:laughing:

Damn it, you got the point 😂
But yes definitely OOP sometimes 😅
Thanks man

AndreMiras · 2019-09-14T10:24:55Z

tests/recipes/test_gevent.py



-class TestGeventRecipe(unittest.TestCase):
+class TestGeventRecipe(RecipeCtx, unittest.TestCase):


Also here, I think we can kill the TestCase inheritance, can't we?

The same than this

AndreMiras · 2019-09-14T10:26:13Z

tests/recipes/test_libgeos.py

+    ):
+        # We overwrite the base test method because we
+        # want to avoid any file/directory creation
+        super(TestLibgeosRecipe, self).test_build_arch()


I think we can actually ditch all theses Python2 style super() call and use the Python3 one if we wanted. At least in unit tests

AndreMiras · 2019-09-14T10:32:23Z

My only regret, is I didn't see the coverage report appearing in GitHub, I wanted to see that satisfying coverall +x% reported here.
Can't wait to see that merged anyway

opacam · 2019-09-14T16:59:35Z

@AndreMiras, thanks for the review 😄

About this coverage thing...I already noticed that...and I don't know what the hell happen, but I remember that at the same time that I pushed the changes I noticed that the coveralls page went down, so maybe the report wasn't send back to github with the coverage report... because the coveralls job for the last commit is done at:

https://coveralls.io/jobs/53165250

Apparently we will increases the coverage by a 3.3% so we will pass the barrier of 60% 🎉

Anyway, I will address your comments which will require a couple of commits, which I hope that brings the coverage report back in here 🤞

AndreMiras

Minor remarks, but I'm all good for a merge

AndreMiras · 2019-09-14T17:41:27Z

tests/recipes/recipe_lib_test.py

+                    mock.call(command),
+                    mock_sh_command.mock_calls,
+                )
+            mock_make.assert_called()


FYI actually this check can also be outside the context manager. The only thing that needs to be in the context manager is the call that you want to be mocked, hence self.recipe.build_arch(self.arch)

AndreMiras · 2019-09-14T17:43:50Z

tests/recipes/recipe_lib_test.py

+            self.recipe.build_arch(self.arch)
+
+            mock_cmake.assert_called()
+            mock_make.assert_called()


Same here it could also be outside. I know it can feel weird at first because we assume the variable is scoped to the that bock, but not. Unlike javascript let this Python variable is function scoped, just like javascript var.

Thanks man, you are right 😄, let me change this and try if a force-push with a rebase on top of develop branch triggers the build...

AndreMiras · 2019-09-14T17:46:10Z

Travis is completely lost it seems. Not even the CI is reporting to the ticket 🤔

opacam · 2019-09-14T18:01:49Z

Mmmmm...yes it's odd, I just pushed my develop branch and no build is triggered (and it should...I was 1 commit behind)...

I have a bad feeling on this...

It maybe related with this new github feature Actions?

I did not enabled this feature in my personal account, neither in here... but I see the Actions tab in here (not in my fork)...so maybe someone enabled this beta feature for python-for-android repo?

A helper module to test recipes. Here we will initialize a `Context` to test recipes.

A helper module to test recipes which will allow to test any recipe using `configure/make` commands.

We can remove the `get_recipe_env` because the environment that we use is already using a clang as default compiler (not the case when we migrated the jpeg recipe to use `cmake`)...so we can do a little refactor :)

Because there is no need to do it there. Also rewrote the inline comments.

AndreMiras

All good

AndreMiras · 2019-09-14T18:34:28Z

It maybe related with this new github feature Actions?

Yes maybe, I can't tell 😕

opacam · 2019-09-14T18:36:51Z

Mmmm...well then...let's try to merge this and see what happens...ok?

AndreMiras · 2019-09-14T18:38:35Z

Yes go for it 💥

opacam · 2019-09-14T18:40:00Z

Ok, then I will squash and merge 👍

opacam · 2019-09-14T22:10:10Z

Finally, the travis builds appeared (with two hours of delay...more or less)...:<
so it seems that there is some delay between pushes and travis requests...it looks like that travis was lazy/busy...

Since this delay had side effects for us, we loosed the travis/coveralls report in this github pull request, here they are:

Travis build succeed
Coveralls report: +3.3%

@AndreMiras, thanks for the reviews, patience and approvals 😄, tomorrow I will properly answer the unanswered questions (you touched some sensitive points that will require some well thought words)...as well I think that I have some pending words in issues section...I also will deal with that...vacations leaved me out of game, I made an halt and catch fire... ☺️

This is a follow up of #1982 In here we do: - Move from `__new__` to `__init__` for `BaseTestForMakeRecipe` ([discussed at here](#1982 (comment))) - Convert cls attributes to instance attributes ([discussed at here](#1982 (comment))) - Remove `mock` dependency for py3 tests (Because, as per Python 3.3+, `mock` it's a built-in-module). (Unfortunately we still will have a couple of `import mock` entries that we cannot remove until we completely remove python2 from our tests) * [test] From `__new__` to `__init__` for `BaseTestForMakeRecipe` * [test] Remove mock dependency for py3 tests Because, as per Python 3.3+, `mock` it's a built-in-module * [test] Convert cls attributes to instance attributes Because we may have some side effects without willing, considering that cls attributes are shared between instances.

The main idea is to test as many recipes as we can with the less code possible and without creating any file/directory so our tests can be performed as fast as possible (all this tests will only add between 2 and 3 seconds to our CI tests and will cover almost 100% of the code for each tested recipe) To do so, we create a couple of modules: tests.recipe_ctx: allow us to create a proper Context to test our recipes tests.recipe_lib_test: which holds some base classes to be used to test a recipe depending on the build method used. For starters we introduce two kind of base classes: BaseTestForMakeRecipe: To test an standard library build (this will iinclude the recipes which requires the classical build commandsconfigure/make) BaseTestForCmakeRecipe: To test an library recipe which is compiled with cmake We also refactor the existing recipes tests, so we can remove some lines in there...the ones that creates a Context. * [test] Add module `tests.recipe_ctx` A helper module to test recipes. Here we will initialize a `Context` to test recipes. * [test] Refactor `setUp/tearDown` for `test_icu` * [test] Refactor `setUp/tearDown` for `test_pyicu` * [test] Refactor `setUp` for `test_reportlab` * [test] Refactor `setUp` for `test_gevent` * [test] Add module `tests.recipe_lib_test` A helper module to test recipes which will allow to test any recipe using `configure/make` commands. * [test] Add test for `libffi` * [test] Add test for `libexpat` * [test] Add test for `libcurl` * [test] Add test for `libiconv` * [test] Add test for `libogg` * [test] Add test for `libpq` * [test] Add test for `libsecp256k1` * [test] Add test for `libshine` * [test] Add test for `libvorbis` * [test] Add test for `libx264` * [test] Add test for `libxml2` * [test] Add test for `libxslt` * [test] Add test for `png` * [test] Add test for `freetype` * [test] Add test for `harfbuzz` * [test] Add test for `openssl` * [test] Add `BaseTestForCmakeRecipe` * [test] Add test for `jpeg` and clean code We can remove the `get_recipe_env` because the environment that we use is already using a clang as default compiler (not the case when we migrated the jpeg recipe to use `cmake`)...so we can do a little refactor :) * [test] Add test for `snappy` * [test] Add test for `leveldb` * [test] Add test for `libgeos` * [test] Add test for `libmysqlclient` * [test] Add test for `openal` * [test] Make the `super` calls Python3 style * [test] Move mock tests outside context manager... Because there is no need to do it there. Also rewrote the inline comments.

This is a follow up of kivy#1982 In here we do: - Move from `__new__` to `__init__` for `BaseTestForMakeRecipe` ([discussed at here](kivy#1982 (comment))) - Convert cls attributes to instance attributes ([discussed at here](kivy#1982 (comment))) - Remove `mock` dependency for py3 tests (Because, as per Python 3.3+, `mock` it's a built-in-module). (Unfortunately we still will have a couple of `import mock` entries that we cannot remove until we completely remove python2 from our tests) * [test] From `__new__` to `__init__` for `BaseTestForMakeRecipe` * [test] Remove mock dependency for py3 tests Because, as per Python 3.3+, `mock` it's a built-in-module * [test] Convert cls attributes to instance attributes Because we may have some side effects without willing, considering that cls attributes are shared between instances.

@TheSin-

* [recipes] Fix compilation for regex recipe The error was: build/other_builds/hostpython3/desktop/hostpython3/Include/Python.h:39:19: fatal error: crypt.h: No such file or directory * [recipes] Update regex's recipe * put the sdk manager command in a code block (#1956) to avoid quotes being replaced by fancy ones * Updated version number for develop branch following 2019.08.09 release (#1960) * Fix build for case-insensitive FS and add CI test for OSX (#1951) * [python] Fix build for case-insensitive FS 🍎 It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way To achieve this goal we refactor a little our `HostPythonRecipe`: - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version) - add public property `HostPythonRecipe.python_exe` (the full path of the python executable) - implement `HostPythonRecipe.should_build` And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`: - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...) - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS We also need: - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version - fix python2 interpreter (fix-interpreter-version.patch) Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions) Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!! * [ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation To do so we: - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk` - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk` - refactor docker files: make use of android's makefile - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build. * customizability (#1869) * customizability options - added compile-options * apptheme option for all bootstraps * Initial migration to NDK r19 (Part I - The core) (#1722) * [ndk19] Rework of the arch environment to work with ndk r19+ Things done here: - Modifies target so we can use the toolchains `in-place` (as if we were using a standalone toolchain) - Removes some cflags that we don't need anymore - Move `macros` and `includes` from `CFLAGS` to `CPPFLAGS` (because this is the way it's supposed to be) - Move libraries from `LDFLAGS` to `LDLIBS` (again, the same as above) - Refactor `Arch`'s variables into class attributes, to make `get_env` smaller and to be more readable/maintainable: + class attributes for flags: * common_cflags * common_cppflags * common_ldflags * common_ldlibs * common_ldshared * arch_cflags + class attributes for compiler: * clang_path * clang_exe * clang_exe_cxx - Remove hardcoded cpu count in `MAKE` - Remove `gcc` of environment (because it has been partially removed in ndk r18) - Shorten long line to be pep8 friendly (line where we define env['BUILDLIB_PATH']) - Commented lines for `ArchAarch_64` that it seems tha we don't need anymore (not removed because I'm not sure if we need them) Note: This will force us to use a minimal ndk version of 19 See also: https://developer.android.com/ndk/guides/other_build_systems * [ndk19] Update `test_archs` to ndk's r19+ build system Also add some more tests. Now we check if some of the functions we mock, `glob` and `find_executable`, are called with the expected arguments * [ndk19] Update `test_bootstrap` to ndk's r19+ build system Because `clang` is the new default compiler and the affected tests were made thinking in `gcc` compiler * [ndk19] Remove `clang` of `get_recipe_env` Because as per ndk r19 the default compiler is clang and gcc has been partially removed * [ndk19] Adapt python recipes to ndk's r19+ build system We move python recipe to use a minimal ndk version of 19 because it will simplify the python's cross compilation (this way we can remove some environment variables that we don't need anymore). The only side effect detected is that the python's `zlib` module isn't built anymore. This is caused because python3 and python2 build systems contains libz's hardcoded version, which does not match the version used in ndk 19 (may be this the cause of some `zip/unzip` errors reported in python-for-android's issues?). We will take care later of this zlib problem, or we will get a runtime crash...so it seems that zlib is an `essential module` to successfully run a python-for-android app * [ndk19] Update libffi to version post `v3.3RC0` and adapt it to be ndk r19+ compatible * [ndk19] Fix build of python's zlib module Because python has a hardcoded zlib version which does not match android's ndk r19, and that prevents the build of zlib's module Note: it seems that we have to point to the right lib and headers, otherwise we won't succeed in building python's zlib module * [ndk19] Fix build of python2's _ctypes module for ndk r19+ * [ndk19] Fix numpy for ndk r19+ Note: because of the new build system recently introduced, we can safely remove some of the patches that we needed because it seems that it builds and runs fine without them * [ndk19] Update docs with the new ndk requirements * [ndk19] Set min/max/recommended android's NDK * [ndk19] Make CI tests with android's NDK `r19b` Because it's our minimum supported android NDK * [ndk19] Fix png recipe for NDK r19+ We remove `get_recipe_env` because we don't need anymore to specify the flags set in there. Note: As per new NDK r19+ build system implemented in p4a, the removed `flags` should be automatically set by the android's NDK * [ndk19] Add `-fPIC` to `armeabi-v7a`'s arch cflags... ...force `Position-Independent Code` To solve the kivy's build error detected in travis `OSX` test (https://travis-ci.org/kivy/python-for-android/jobs/576204088#L1105). But also detected the same error in my local tests (linux os in this case) for `freetype` and `harfbuzz` recipes...so I have the feeling that this is better to set as global 🤞..is it? * [LIBS - PART I] Initial refactor of library recipes (#1944) This commit will introduce a new cls attribute: `built_libraries`. This `built_libraries` attribute must be a dictionary. The keys are the full library name, e.g.: `libffi.so`. And the values must be the relative path of the library, e.g: `.libs`. So the this cls attribute for libffi recipe it would look like: ```built_libraries = {'libffi.so': '.libs'}``` This new cls attribute will allow us to detect library recipes and to refactor common operations that we do for those kind of recipes: - copy library into right location - implement `should_build` for library recipes Those two things will make that our rebuilds to be more consistent and I hope that it will be easier to maintain. **So, once explained a little this PR, in here we do:** - [x] Add attribute `built_libraries` - [x] Add methods to refactor common library operations: - [x] install_libraries - [x] get_libraries - [x] implement `should_build` for library recipes - [x] implement basic tests for the newly introduced `Recipe` methods - [x] Make use of those Library attributes/methods for some basic set of recipes: - [x] libffi - [x] openssl - [x] png - [x] jpeg - [x] freetype - [x] harfbuzz - [x] libcurl - [x] libzbar - [x] libiconv - [x] libexpat - [x] libogg - [x] libxml2 - [x] libxslt - [x] libshine - [x] libx264 - [x] libglob - [x] libsodium - [x] libsecp256k1 **Things to do in separate PRs**, to be easy to review and because those points will start to conflict with the `NDK r19 migration` thing just merged: - Make use of Library attributes/methods introduced in this PR for un covered library recipes in here. Here we have two situations: - library recipes that depends on android's STL library (almost all the work done in here will depend of the `NDK r19 migration` thing) - all remaining library recipes, which are not STL lib dependent and that are not implemented in here (because I was unable to perform a successful build with them using arch `arm64-v8a`...so I think it would be better to deal with those recipes in separate PRs and later...with the `NDK r19 migration` thing merged) **Notes about changed recipes:** all the recipes touched in here almost have the same changes: - remove `should_build` method (to make use of the one implemented in base class for library recipes) - remove the copy of the library (because that will be done automatically by the method `install_libraries` implemented in base class) - fixed the imports due to refactoring * [recipe-lib] Add attribute Recipe.built_libraries so we can refactor some essential operations performed for library recipes (like copy the library into the right location, or decide if we should build the library recipe again) * [recipe-lib] Add method Recipe.get_libraries This function will allows us to get a list of the built/context libraries. This will be useful to check if we have to built a library recipe or which library should be rebuild * [recipe-lib] Make `Recipe.should_build` decide the library build In case that the attribute built_libraries has been set, we will check the need of the recipe build, otherwise we will act as usual, forcing the build unless overwrote in subclass * [recipe-lib] Add method `Recipe.install_libraries` So we can: - control the copy of the library recipe - refactor the install of the lib for all library recipes * [recipe-lib] Make libffi a library recipe * [recipe-lib] Make openssl a library recipe and ... also make the imports from the right module * [recipe-lib] Make png a library recipe * [recipe-lib] Make jpeg a library recipe * [recipe-lib] Make freetype a library recipe and ... also make the imports from the right module * [recipe-lib] Make harfbuzz a library recipe and ... also make the imports from the right module * [recipe-lib] Make libcurl a library recipe and ... also make the imports from the right module * [recipe-lib] Make libzbar a library recipe and ... also make the imports from the right module * [recipe-lib] Make libiconv a library recipe and ... also make the imports from the right module * [recipe-lib] Make libexpat a library recipe and ... also: - make the imports from the right module - remove hardcoded arch * [recipe-lib] Make libogg a library recipe * [recipe-lib] Make libxml2 a library recipe and ... also make the imports from the right module * [recipe-lib] Make libxslt a library recipe and ... also make the imports from the right module * [recipe-lib] Make libshine a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libx264 a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libglob a library recipe * [recipe-lib] Make libsodium a library recipe and ... also: - make the imports from the right module - fix hardcoded arch - enable cpu count when compiling * [recipe-lib] Make libsecp256k1 a library recipe and ... also make the imports from the right module * [tests] Add tests for library recipe * [NDK19] Fix libglob for android NDK r19 - change the `ARG_MAX` define, because it's already defined at `sysroot/usr/include/linux/limits.h` - Replaced `size_t` by Including the <stddef.h> header. Because found the solution at stackoverflow: As per C99, §7.17, size_t is not a builtin type but defined in <stddef.h> See also: - https://travis-ci.org/kivy/python-for-android/jobs/576392711#L5992-L6013 - https://stackoverflow.com/questions/26410466/gcc-linaro-compiler-throws-error-unknown-type-name-size-t * [NDK19] Remove `--sysroot` from LDFLAGS for cffi and pymunk (#1965) Because `--sysroot` flag is not needed anymore and make it fails the build since we migrated to new android's build system (implemented in NDK r19) See also: - https://developer.android.com/ndk/guides/other_build_systems * Added libffi headers troubleshooting note to doc (#1972) * [LIBS - PART II] Part II of NDK r19 migration - Initial STL lib migration (#1947) * [recipe-stl] Add android's STL lib support to `Recipe` To allow us to refactor some common operations that we use in our recipes that depends on android's STL library. Note: This commit will allow us to begin the migration to `c++_shared`. This is a must when we move to android's NDK r19+, because as for android NDK >= 18 is the only one supported STL library. * [recipe-stl] Make CppCompiledComponentsPythonRecipe use Recipe's STL support * [recipe-stl] Make icu a library recipe with STL support (rework) Also done here: - Remove hardcoded version in url - Disable versioned shared libraries - Make it to be build as a shared libraries (instead of static) - Disable the build of static libraries (because we build them as shared ones, so we avoid to link with them without our consents) - Shorten long lines to be pep8's friendly - Remove icu from ci/constants - Remove `RuntimeError` about the need to use NDK api <= 19 (because that is not the case anymore) - consider host's number of cpu to perform the build * [recipe-stl] Rework pyicu recipe to match latest icu changes Also done here: - Remove icu.patch because now we don't have the version in our icu libraries - Shorten long lines to be pep8's friendly * [tests] Add tests for recipe with STL support * [tests] Add tests for icu recipe * [tests] Add tests for pyicu recipe * [WIP][LIBS - PART III] Rework of pyleveldb, leveldb and snappy (#1966) * [recipe-lib] Make snappy a library recipe * [recipe-stl] Make leveldb a library recipe and... make it work with the reworked snappy recipe * [recipe-stl] Fix pyleveldb for reworked leveldb/snappy * [WIP][LIBS - PART IV] Rework of shapely and libgeos (#1967) * [recipe-stl] Make libgeos a library recipe * [recipe-stl] Fix shapely for reworked libgeos * [recipe-stl] Make libzmq a library recipe (#1968) * [recipe-stl] Rework of libtorrent and boost (#1971) which: - fix build for android's NDK r19+ - allow us to build `boost+libtorrent` for any arch - Update boost to version `1.69.0` - update libtorrent to version `1.2.1` The build method needs to be changed because one of the scripts used in `boost+libtorrent` build (make-standalone-toolchain.sh), has been removed from android's NDK. [This is because](https://developer.android.com/ndk/guides/standalone_toolchain): `As of r19, the NDK's default toolchains are standalone toolchains, which renders this process unnecessary.` Note: since 3887d2b, `python-for-android` uses android's NDK r19 as the minimum supported and also changed the build method used by p4a as [described in here](https://developer.android.com/ndk/guides/other_build_systems) * Also copy the service/main.py when building with setup.py (#1936) * [recipe-stl] Rework of protobuf_cpp recipe (#1969) In here we do: - inherit from CppCompiledComponentsPythonRecipe Because depends on android's STL library - make use of the base class methods for library recipes - Split build_arch into proper methods - Shorten some long lines (to be PEP8 friendly) - make generated library shared - remove recipe from CI/constants * [CI] Fix CI's test for `arm64-v8a` (#1977) * Updated README.md to clarify NDK versions (#1981) * 📚 Updated README.dm to clarify NDK versions * 🔧 Change back to r19b * A bunch of tests for library recipes (#1982) The main idea is to test as many recipes as we can with the less code possible and without creating any file/directory so our tests can be performed as fast as possible (all this tests will only add between 2 and 3 seconds to our CI tests and will cover almost 100% of the code for each tested recipe) To do so, we create a couple of modules: tests.recipe_ctx: allow us to create a proper Context to test our recipes tests.recipe_lib_test: which holds some base classes to be used to test a recipe depending on the build method used. For starters we introduce two kind of base classes: BaseTestForMakeRecipe: To test an standard library build (this will iinclude the recipes which requires the classical build commandsconfigure/make) BaseTestForCmakeRecipe: To test an library recipe which is compiled with cmake We also refactor the existing recipes tests, so we can remove some lines in there...the ones that creates a Context. * [test] Add module `tests.recipe_ctx` A helper module to test recipes. Here we will initialize a `Context` to test recipes. * [test] Refactor `setUp/tearDown` for `test_icu` * [test] Refactor `setUp/tearDown` for `test_pyicu` * [test] Refactor `setUp` for `test_reportlab` * [test] Refactor `setUp` for `test_gevent` * [test] Add module `tests.recipe_lib_test` A helper module to test recipes which will allow to test any recipe using `configure/make` commands. * [test] Add test for `libffi` * [test] Add test for `libexpat` * [test] Add test for `libcurl` * [test] Add test for `libiconv` * [test] Add test for `libogg` * [test] Add test for `libpq` * [test] Add test for `libsecp256k1` * [test] Add test for `libshine` * [test] Add test for `libvorbis` * [test] Add test for `libx264` * [test] Add test for `libxml2` * [test] Add test for `libxslt` * [test] Add test for `png` * [test] Add test for `freetype` * [test] Add test for `harfbuzz` * [test] Add test for `openssl` * [test] Add `BaseTestForCmakeRecipe` * [test] Add test for `jpeg` and clean code We can remove the `get_recipe_env` because the environment that we use is already using a clang as default compiler (not the case when we migrated the jpeg recipe to use `cmake`)...so we can do a little refactor :) * [test] Add test for `snappy` * [test] Add test for `leveldb` * [test] Add test for `libgeos` * [test] Add test for `libmysqlclient` * [test] Add test for `openal` * [test] Make the `super` calls Python3 style * [test] Move mock tests outside context manager... Because there is no need to do it there. Also rewrote the inline comments. * Recipes tests enhancements (#1984) This is a follow up of #1982 In here we do: - Move from `__new__` to `__init__` for `BaseTestForMakeRecipe` ([discussed at here](#1982 (comment))) - Convert cls attributes to instance attributes ([discussed at here](#1982 (comment))) - Remove `mock` dependency for py3 tests (Because, as per Python 3.3+, `mock` it's a built-in-module). (Unfortunately we still will have a couple of `import mock` entries that we cannot remove until we completely remove python2 from our tests) * [test] From `__new__` to `__init__` for `BaseTestForMakeRecipe` * [test] Remove mock dependency for py3 tests Because, as per Python 3.3+, `mock` it's a built-in-module * [test] Convert cls attributes to instance attributes Because we may have some side effects without willing, considering that cls attributes are shared between instances. * Fixes test_virtualenv and test_venv failing, closes #1994 (#1995) Recent pep517 release seems to break tests. * Made p4a use per-arch dist build dirs (#1986) * Made p4a use per-arch dist build dirs * Fixed up check on existing folder when overwriting distributions * Improved Distribution.get_distribution api * Fixed test_archs.py * Fixed test_bootstrap.py * Fixed test_distribution.py * Fixed recipes tests * Fixed formatting * Made sure whole build process accesses dist_dir correctly * Fixed apk output name formatting by moving version to end of string * Further test fixes following dist_dir changes * Removed unnecessary variable * Fixes libiconv & libzbar configure host Having `--host=x86` was making the configure script fail at enabling shared library build on `x86`. This was resulting in failing to copy `libiconv.so` and `libzbar.so` on `x86` arch. * Updates Java version troubleshooting (#1991) Adds the NoClassDefFoundError XmlSchema case and groups related cases together. Gives instructions for both Ubuntu and macOS to fix. * Made on-device unit tests app use the develop branch by default (#1985) * Updated version to 2019.10.06 Co-authored-by: Pol Canelles <[email protected]> Co-authored-by: Gabriel Pettier <[email protected]> Co-authored-by: Philipp Auersperg-Castell <[email protected]> Co-authored-by: Richard Larkin <[email protected]> Co-authored-by: Andre Miras <[email protected]>

@TheSin-

* Release 2019.10.06 (#1998) * [recipes] Fix compilation for regex recipe The error was: build/other_builds/hostpython3/desktop/hostpython3/Include/Python.h:39:19: fatal error: crypt.h: No such file or directory * [recipes] Update regex's recipe * put the sdk manager command in a code block (#1956) to avoid quotes being replaced by fancy ones * Updated version number for develop branch following 2019.08.09 release (#1960) * Fix build for case-insensitive FS and add CI test for OSX (#1951) * [python] Fix build for case-insensitive FS 🍎 It turns out that the generated python binary for some builds are named `python.exe` instead of `python`. This depends on the File System where the build happens. It will be named `python.exe` when the FS is case-insensitive (Mac OSX and Cygwin), but it will be named `python` when the FS is case-sensitive (most GNU Linux distributions). The proposed solution consists in make a copy of the generated python binary with a given name (python3 or python2) so this way To achieve this goal we refactor a little our `HostPythonRecipe`: - add private property `HostPythonRecipe._exe_name` (the name of the python executable based on major version) - add public property `HostPythonRecipe.python_exe` (the full path of the python executable) - implement `HostPythonRecipe.should_build` And also it's affected the `GuestPythonRecipe`, because we need to use the generated python executable by `HostPythonRecipe`: - add private property `GuestPythonRecipe._libpython` (python's library name with extension...hardcoded for now...) - implement `GuestPythonRecipe.should_build`... to check the library instead of the executable so we avoid conflicts with case-insensitive FS We also need: - fix `PythonRecipe.real_hostpython_location` because the name of our host python executable will depend on major version - fix python2 interpreter (fix-interpreter-version.patch) Note: the variation of the name of the python's executable is mentioned at python's build docs (https://github.com/python/cpython/blob/3.7/README.rst#build-instructions) Note: @TheSin- , ¡¡¡thanks for your debugging sessions!!! * [ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation To do so we: - create a makefile to install the necessary dependencies for Mac OS X: `ci/makefiles/osx.mk` - create a makefile to install android's SDK/NDK: `ci/makefiles/android.mk` - refactor docker files: make use of android's makefile - change OS, from `linux` to `osx`, for CI test `Python 3 armeabi-v7a`, so we don't increase the overall build time and jobs - rename the `Python 2` test to `Python 2 armeabi-v7a (with numpy)` to reflect the build arch and numpy's build. * customizability (#1869) * customizability options - added compile-options * apptheme option for all bootstraps * Initial migration to NDK r19 (Part I - The core) (#1722) * [ndk19] Rework of the arch environment to work with ndk r19+ Things done here: - Modifies target so we can use the toolchains `in-place` (as if we were using a standalone toolchain) - Removes some cflags that we don't need anymore - Move `macros` and `includes` from `CFLAGS` to `CPPFLAGS` (because this is the way it's supposed to be) - Move libraries from `LDFLAGS` to `LDLIBS` (again, the same as above) - Refactor `Arch`'s variables into class attributes, to make `get_env` smaller and to be more readable/maintainable: + class attributes for flags: * common_cflags * common_cppflags * common_ldflags * common_ldlibs * common_ldshared * arch_cflags + class attributes for compiler: * clang_path * clang_exe * clang_exe_cxx - Remove hardcoded cpu count in `MAKE` - Remove `gcc` of environment (because it has been partially removed in ndk r18) - Shorten long line to be pep8 friendly (line where we define env['BUILDLIB_PATH']) - Commented lines for `ArchAarch_64` that it seems tha we don't need anymore (not removed because I'm not sure if we need them) Note: This will force us to use a minimal ndk version of 19 See also: https://developer.android.com/ndk/guides/other_build_systems * [ndk19] Update `test_archs` to ndk's r19+ build system Also add some more tests. Now we check if some of the functions we mock, `glob` and `find_executable`, are called with the expected arguments * [ndk19] Update `test_bootstrap` to ndk's r19+ build system Because `clang` is the new default compiler and the affected tests were made thinking in `gcc` compiler * [ndk19] Remove `clang` of `get_recipe_env` Because as per ndk r19 the default compiler is clang and gcc has been partially removed * [ndk19] Adapt python recipes to ndk's r19+ build system We move python recipe to use a minimal ndk version of 19 because it will simplify the python's cross compilation (this way we can remove some environment variables that we don't need anymore). The only side effect detected is that the python's `zlib` module isn't built anymore. This is caused because python3 and python2 build systems contains libz's hardcoded version, which does not match the version used in ndk 19 (may be this the cause of some `zip/unzip` errors reported in python-for-android's issues?). We will take care later of this zlib problem, or we will get a runtime crash...so it seems that zlib is an `essential module` to successfully run a python-for-android app * [ndk19] Update libffi to version post `v3.3RC0` and adapt it to be ndk r19+ compatible * [ndk19] Fix build of python's zlib module Because python has a hardcoded zlib version which does not match android's ndk r19, and that prevents the build of zlib's module Note: it seems that we have to point to the right lib and headers, otherwise we won't succeed in building python's zlib module * [ndk19] Fix build of python2's _ctypes module for ndk r19+ * [ndk19] Fix numpy for ndk r19+ Note: because of the new build system recently introduced, we can safely remove some of the patches that we needed because it seems that it builds and runs fine without them * [ndk19] Update docs with the new ndk requirements * [ndk19] Set min/max/recommended android's NDK * [ndk19] Make CI tests with android's NDK `r19b` Because it's our minimum supported android NDK * [ndk19] Fix png recipe for NDK r19+ We remove `get_recipe_env` because we don't need anymore to specify the flags set in there. Note: As per new NDK r19+ build system implemented in p4a, the removed `flags` should be automatically set by the android's NDK * [ndk19] Add `-fPIC` to `armeabi-v7a`'s arch cflags... ...force `Position-Independent Code` To solve the kivy's build error detected in travis `OSX` test (https://travis-ci.org/kivy/python-for-android/jobs/576204088#L1105). But also detected the same error in my local tests (linux os in this case) for `freetype` and `harfbuzz` recipes...so I have the feeling that this is better to set as global 🤞..is it? * [LIBS - PART I] Initial refactor of library recipes (#1944) This commit will introduce a new cls attribute: `built_libraries`. This `built_libraries` attribute must be a dictionary. The keys are the full library name, e.g.: `libffi.so`. And the values must be the relative path of the library, e.g: `.libs`. So the this cls attribute for libffi recipe it would look like: ```built_libraries = {'libffi.so': '.libs'}``` This new cls attribute will allow us to detect library recipes and to refactor common operations that we do for those kind of recipes: - copy library into right location - implement `should_build` for library recipes Those two things will make that our rebuilds to be more consistent and I hope that it will be easier to maintain. **So, once explained a little this PR, in here we do:** - [x] Add attribute `built_libraries` - [x] Add methods to refactor common library operations: - [x] install_libraries - [x] get_libraries - [x] implement `should_build` for library recipes - [x] implement basic tests for the newly introduced `Recipe` methods - [x] Make use of those Library attributes/methods for some basic set of recipes: - [x] libffi - [x] openssl - [x] png - [x] jpeg - [x] freetype - [x] harfbuzz - [x] libcurl - [x] libzbar - [x] libiconv - [x] libexpat - [x] libogg - [x] libxml2 - [x] libxslt - [x] libshine - [x] libx264 - [x] libglob - [x] libsodium - [x] libsecp256k1 **Things to do in separate PRs**, to be easy to review and because those points will start to conflict with the `NDK r19 migration` thing just merged: - Make use of Library attributes/methods introduced in this PR for un covered library recipes in here. Here we have two situations: - library recipes that depends on android's STL library (almost all the work done in here will depend of the `NDK r19 migration` thing) - all remaining library recipes, which are not STL lib dependent and that are not implemented in here (because I was unable to perform a successful build with them using arch `arm64-v8a`...so I think it would be better to deal with those recipes in separate PRs and later...with the `NDK r19 migration` thing merged) **Notes about changed recipes:** all the recipes touched in here almost have the same changes: - remove `should_build` method (to make use of the one implemented in base class for library recipes) - remove the copy of the library (because that will be done automatically by the method `install_libraries` implemented in base class) - fixed the imports due to refactoring * [recipe-lib] Add attribute Recipe.built_libraries so we can refactor some essential operations performed for library recipes (like copy the library into the right location, or decide if we should build the library recipe again) * [recipe-lib] Add method Recipe.get_libraries This function will allows us to get a list of the built/context libraries. This will be useful to check if we have to built a library recipe or which library should be rebuild * [recipe-lib] Make `Recipe.should_build` decide the library build In case that the attribute built_libraries has been set, we will check the need of the recipe build, otherwise we will act as usual, forcing the build unless overwrote in subclass * [recipe-lib] Add method `Recipe.install_libraries` So we can: - control the copy of the library recipe - refactor the install of the lib for all library recipes * [recipe-lib] Make libffi a library recipe * [recipe-lib] Make openssl a library recipe and ... also make the imports from the right module * [recipe-lib] Make png a library recipe * [recipe-lib] Make jpeg a library recipe * [recipe-lib] Make freetype a library recipe and ... also make the imports from the right module * [recipe-lib] Make harfbuzz a library recipe and ... also make the imports from the right module * [recipe-lib] Make libcurl a library recipe and ... also make the imports from the right module * [recipe-lib] Make libzbar a library recipe and ... also make the imports from the right module * [recipe-lib] Make libiconv a library recipe and ... also make the imports from the right module * [recipe-lib] Make libexpat a library recipe and ... also: - make the imports from the right module - remove hardcoded arch * [recipe-lib] Make libogg a library recipe * [recipe-lib] Make libxml2 a library recipe and ... also make the imports from the right module * [recipe-lib] Make libxslt a library recipe and ... also make the imports from the right module * [recipe-lib] Make libshine a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libx264 a library recipe and ... also: - make the imports from the right module - remove the hardcoded cpu count when compiling * [recipe-lib] Make libglob a library recipe * [recipe-lib] Make libsodium a library recipe and ... also: - make the imports from the right module - fix hardcoded arch - enable cpu count when compiling * [recipe-lib] Make libsecp256k1 a library recipe and ... also make the imports from the right module * [tests] Add tests for library recipe * [NDK19] Fix libglob for android NDK r19 - change the `ARG_MAX` define, because it's already defined at `sysroot/usr/include/linux/limits.h` - Replaced `size_t` by Including the <stddef.h> header. Because found the solution at stackoverflow: As per C99, §7.17, size_t is not a builtin type but defined in <stddef.h> See also: - https://travis-ci.org/kivy/python-for-android/jobs/576392711#L5992-L6013 - https://stackoverflow.com/questions/26410466/gcc-linaro-compiler-throws-error-unknown-type-name-size-t * [NDK19] Remove `--sysroot` from LDFLAGS for cffi and pymunk (#1965) Because `--sysroot` flag is not needed anymore and make it fails the build since we migrated to new android's build system (implemented in NDK r19) See also: - https://developer.android.com/ndk/guides/other_build_systems * Added libffi headers troubleshooting note to doc (#1972) * [LIBS - PART II] Part II of NDK r19 migration - Initial STL lib migration (#1947) * [recipe-stl] Add android's STL lib support to `Recipe` To allow us to refactor some common operations that we use in our recipes that depends on android's STL library. Note: This commit will allow us to begin the migration to `c++_shared`. This is a must when we move to android's NDK r19+, because as for android NDK >= 18 is the only one supported STL library. * [recipe-stl] Make CppCompiledComponentsPythonRecipe use Recipe's STL support * [recipe-stl] Make icu a library recipe with STL support (rework) Also done here: - Remove hardcoded version in url - Disable versioned shared libraries - Make it to be build as a shared libraries (instead of static) - Disable the build of static libraries (because we build them as shared ones, so we avoid to link with them without our consents) - Shorten long lines to be pep8's friendly - Remove icu from ci/constants - Remove `RuntimeError` about the need to use NDK api <= 19 (because that is not the case anymore) - consider host's number of cpu to perform the build * [recipe-stl] Rework pyicu recipe to match latest icu changes Also done here: - Remove icu.patch because now we don't have the version in our icu libraries - Shorten long lines to be pep8's friendly * [tests] Add tests for recipe with STL support * [tests] Add tests for icu recipe * [tests] Add tests for pyicu recipe * [WIP][LIBS - PART III] Rework of pyleveldb, leveldb and snappy (#1966) * [recipe-lib] Make snappy a library recipe * [recipe-stl] Make leveldb a library recipe and... make it work with the reworked snappy recipe * [recipe-stl] Fix pyleveldb for reworked leveldb/snappy * [WIP][LIBS - PART IV] Rework of shapely and libgeos (#1967) * [recipe-stl] Make libgeos a library recipe * [recipe-stl] Fix shapely for reworked libgeos * [recipe-stl] Make libzmq a library recipe (#1968) * [recipe-stl] Rework of libtorrent and boost (#1971) which: - fix build for android's NDK r19+ - allow us to build `boost+libtorrent` for any arch - Update boost to version `1.69.0` - update libtorrent to version `1.2.1` The build method needs to be changed because one of the scripts used in `boost+libtorrent` build (make-standalone-toolchain.sh), has been removed from android's NDK. [This is because](https://developer.android.com/ndk/guides/standalone_toolchain): `As of r19, the NDK's default toolchains are standalone toolchains, which renders this process unnecessary.` Note: since 3887d2b, `python-for-android` uses android's NDK r19 as the minimum supported and also changed the build method used by p4a as [described in here](https://developer.android.com/ndk/guides/other_build_systems) * Also copy the service/main.py when building with setup.py (#1936) * [recipe-stl] Rework of protobuf_cpp recipe (#1969) In here we do: - inherit from CppCompiledComponentsPythonRecipe Because depends on android's STL library - make use of the base class methods for library recipes - Split build_arch into proper methods - Shorten some long lines (to be PEP8 friendly) - make generated library shared - remove recipe from CI/constants * [CI] Fix CI's test for `arm64-v8a` (#1977) * Updated README.md to clarify NDK versions (#1981) * 📚 Updated README.dm to clarify NDK versions * 🔧 Change back to r19b * A bunch of tests for library recipes (#1982) The main idea is to test as many recipes as we can with the less code possible and without creating any file/directory so our tests can be performed as fast as possible (all this tests will only add between 2 and 3 seconds to our CI tests and will cover almost 100% of the code for each tested recipe) To do so, we create a couple of modules: tests.recipe_ctx: allow us to create a proper Context to test our recipes tests.recipe_lib_test: which holds some base classes to be used to test a recipe depending on the build method used. For starters we introduce two kind of base classes: BaseTestForMakeRecipe: To test an standard library build (this will iinclude the recipes which requires the classical build commandsconfigure/make) BaseTestForCmakeRecipe: To test an library recipe which is compiled with cmake We also refactor the existing recipes tests, so we can remove some lines in there...the ones that creates a Context. * [test] Add module `tests.recipe_ctx` A helper module to test recipes. Here we will initialize a `Context` to test recipes. * [test] Refactor `setUp/tearDown` for `test_icu` * [test] Refactor `setUp/tearDown` for `test_pyicu` * [test] Refactor `setUp` for `test_reportlab` * [test] Refactor `setUp` for `test_gevent` * [test] Add module `tests.recipe_lib_test` A helper module to test recipes which will allow to test any recipe using `configure/make` commands. * [test] Add test for `libffi` * [test] Add test for `libexpat` * [test] Add test for `libcurl` * [test] Add test for `libiconv` * [test] Add test for `libogg` * [test] Add test for `libpq` * [test] Add test for `libsecp256k1` * [test] Add test for `libshine` * [test] Add test for `libvorbis` * [test] Add test for `libx264` * [test] Add test for `libxml2` * [test] Add test for `libxslt` * [test] Add test for `png` * [test] Add test for `freetype` * [test] Add test for `harfbuzz` * [test] Add test for `openssl` * [test] Add `BaseTestForCmakeRecipe` * [test] Add test for `jpeg` and clean code We can remove the `get_recipe_env` because the environment that we use is already using a clang as default compiler (not the case when we migrated the jpeg recipe to use `cmake`)...so we can do a little refactor :) * [test] Add test for `snappy` * [test] Add test for `leveldb` * [test] Add test for `libgeos` * [test] Add test for `libmysqlclient` * [test] Add test for `openal` * [test] Make the `super` calls Python3 style * [test] Move mock tests outside context manager... Because there is no need to do it there. Also rewrote the inline comments. * Recipes tests enhancements (#1984) This is a follow up of #1982 In here we do: - Move from `__new__` to `__init__` for `BaseTestForMakeRecipe` ([discussed at here](#1982 (comment))) - Convert cls attributes to instance attributes ([discussed at here](#1982 (comment))) - Remove `mock` dependency for py3 tests (Because, as per Python 3.3+, `mock` it's a built-in-module). (Unfortunately we still will have a couple of `import mock` entries that we cannot remove until we completely remove python2 from our tests) * [test] From `__new__` to `__init__` for `BaseTestForMakeRecipe` * [test] Remove mock dependency for py3 tests Because, as per Python 3.3+, `mock` it's a built-in-module * [test] Convert cls attributes to instance attributes Because we may have some side effects without willing, considering that cls attributes are shared between instances. * Fixes test_virtualenv and test_venv failing, closes #1994 (#1995) Recent pep517 release seems to break tests. * Made p4a use per-arch dist build dirs (#1986) * Made p4a use per-arch dist build dirs * Fixed up check on existing folder when overwriting distributions * Improved Distribution.get_distribution api * Fixed test_archs.py * Fixed test_bootstrap.py * Fixed test_distribution.py * Fixed recipes tests * Fixed formatting * Made sure whole build process accesses dist_dir correctly * Fixed apk output name formatting by moving version to end of string * Further test fixes following dist_dir changes * Removed unnecessary variable * Fixes libiconv & libzbar configure host Having `--host=x86` was making the configure script fail at enabling shared library build on `x86`. This was resulting in failing to copy `libiconv.so` and `libzbar.so` on `x86` arch. * Updates Java version troubleshooting (#1991) Adds the NoClassDefFoundError XmlSchema case and groups related cases together. Gives instructions for both Ubuntu and macOS to fix. * Made on-device unit tests app use the develop branch by default (#1985) * Updated version to 2019.10.06 Co-authored-by: Pol Canelles <[email protected]> Co-authored-by: Gabriel Pettier <[email protected]> Co-authored-by: Philipp Auersperg-Castell <[email protected]> Co-authored-by: Richard Larkin <[email protected]> Co-authored-by: Andre Miras <[email protected]> * Bumps openssl to 1.1.1f * Updated version to 2019.10.06.post0 Co-authored-by: Pol Canelles <[email protected]> Co-authored-by: Gabriel Pettier <[email protected]> Co-authored-by: Philipp Auersperg-Castell <[email protected]> Co-authored-by: Richard Larkin <[email protected]> Co-authored-by: Andre Miras <[email protected]> Co-authored-by: Andre Miras <[email protected]>

AndreMiras previously approved these changes Sep 14, 2019

View reviewed changes

opacam dismissed AndreMiras’s stale review via 4f54abc September 14, 2019 17:30

AndreMiras previously approved these changes Sep 14, 2019

View reviewed changes

opacam added 22 commits September 14, 2019 20:20

[test] Add module tests.recipe_ctx

11df8e0

A helper module to test recipes. Here we will initialize a `Context` to test recipes.

[test] Refactor setUp/tearDown for test_icu

b242289

[test] Refactor setUp/tearDown for test_pyicu

bd1018b

[test] Refactor setUp for test_reportlab

add66c2

[test] Refactor setUp for test_gevent

52978e1

[test] Add module tests.recipe_lib_test

04be9cb

A helper module to test recipes which will allow to test any recipe using `configure/make` commands.

[test] Add test for libffi

5e2c08f

[test] Add test for libexpat

5616cbd

[test] Add test for libcurl

3b0478e

[test] Add test for libiconv

bbf4d64

[test] Add test for libogg

c4deb10

[test] Add test for libpq

7061cc5

[test] Add test for libsecp256k1

18bda8e

[test] Add test for libshine

054eb02

[test] Add test for libvorbis

6f3758a

[test] Add test for libx264

a68522a

[test] Add test for libxml2

6370455

[test] Add test for libxslt

bc93051

[test] Add test for png

ed8524f

[test] Add test for freetype

fd4cbc0

[test] Add test for harfbuzz

d3d6e1d

[test] Add test for openssl

5cca1f5

opacam added 9 commits September 14, 2019 20:20

[test] Add BaseTestForCmakeRecipe

a95b7e9

[test] Add test for jpeg and clean code

0d48b40

We can remove the `get_recipe_env` because the environment that we use is already using a clang as default compiler (not the case when we migrated the jpeg recipe to use `cmake`)...so we can do a little refactor :)

[test] Add test for snappy

47bba46

[test] Add test for leveldb

f77d693

[test] Add test for libgeos

13a48ea

[test] Add test for libmysqlclient

2e73dc2

[test] Add test for openal

b27376b

[test] Make the super calls Python3 style

fcc71f2

[test] Move mock tests outside context manager...

80104c4

Because there is no need to do it there. Also rewrote the inline comments.

opacam dismissed AndreMiras’s stale review via 80104c4 September 14, 2019 18:25

opacam force-pushed the feature-test-libs branch from 4f54abc to 80104c4 Compare September 14, 2019 18:25

AndreMiras approved these changes Sep 14, 2019

View reviewed changes

opacam merged commit be7abe9 into kivy:develop Sep 14, 2019

opacam deleted the feature-test-libs branch September 14, 2019 22:11

opacam mentioned this pull request Sep 15, 2019

Recipes tests enhancements #1984

Merged

		from tests.recipes.recipe_lib_test import BaseTestForMakeRecipe


		class TestFreetypeRecipe(BaseTestForMakeRecipe, unittest.TestCase):



		class TestGeventRecipe(unittest.TestCase):
		class TestGeventRecipe(RecipeCtx, unittest.TestCase):

Proof of concept - A bunch of tests for library recipes #1982

Proof of concept - A bunch of tests for library recipes #1982

Conversation

opacam commented Sep 12, 2019

AndreMiras commented Sep 14, 2019

AndreMiras left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreMiras commented Sep 14, 2019

opacam commented Sep 14, 2019

AndreMiras left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AndreMiras commented Sep 14, 2019

opacam commented Sep 14, 2019

AndreMiras left a comment

Choose a reason for hiding this comment

AndreMiras commented Sep 14, 2019

opacam commented Sep 14, 2019

AndreMiras commented Sep 14, 2019

opacam commented Sep 14, 2019

opacam commented Sep 14, 2019 • edited Loading

opacam commented Sep 14, 2019 •

edited

Loading