Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Work manager integration #119

Merged
merged 39 commits into from
Apr 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
42b6324
Define default tasks setup and hooks.
rtibbles Jun 9, 2022
f1ef772
Remove use of kolibri workers.
rtibbles Aug 25, 2022
4860e43
Add basic task worker integration.
rtibbles Jun 9, 2022
831403a
Lint and reorganize code.
rtibbles Jun 9, 2022
f07c67d
Update to latest p4a commit.
rtibbles Feb 25, 2023
5f489b5
Add expedited processing for high priority tasks.
rtibbles Feb 28, 2023
2864e56
Commit built bootstrap to allow for direct editing in Android Studio.
rtibbles Mar 21, 2023
a0f1097
Move notification channel creation into main activity.
rtibbles Mar 21, 2023
d9d785d
Show notification statuses when available.
rtibbles Mar 21, 2023
8f4b901
Set update hooks to ensure that changes in status trigger appropriately.
rtibbles Mar 21, 2023
d02edc0
Update build scripts to preserve local edits to bootstrap.
rtibbles Mar 22, 2023
83b796d
Update kolibri notification icon files.
rtibbles Mar 22, 2023
426ac58
Make long running tasks get set to foreground.
rtibbles Mar 22, 2023
f2db879
Update README and make command.
rtibbles Mar 22, 2023
ac09bf8
Update python for android to latest release.
rtibbles Mar 22, 2023
8fc35e8
Add a notification color.
rtibbles Mar 23, 2023
290ec04
Make notification explicitly silent.
rtibbles Mar 28, 2023
0703d2a
Remove unnecessary asset unpacking.
rtibbles Mar 28, 2023
3d5c953
Properly handle finite repeating task rescheduling.
rtibbles Apr 3, 2023
bb44d03
Delete the kolibri dist before rebuilding.
rtibbles Apr 3, 2023
679f489
Simplify loadLibraries and prevent unnecessary repeated load library …
rtibbles Apr 4, 2023
7fb01af
Do not just exit the system process, interrupt the Python thread inst…
rtibbles Apr 4, 2023
ebab523
Add more specific logging.
rtibbles Apr 4, 2023
a509b4f
Count active python threads in the process to prevent premature teard…
rtibbles Apr 4, 2023
9e60047
Add util for getting context.
rtibbles Apr 5, 2023
002a696
Add informative tags to queued tasks.
rtibbles Apr 5, 2023
04e91a9
Update hooks to use storage hook.
rtibbles Apr 5, 2023
f890001
Handle task clearing.
rtibbles Apr 5, 2023
cd06417
Properly setup android app plugin.
rtibbles Apr 5, 2023
8276578
Ensure job isn't currently running before trying to clear it.
rtibbles Apr 5, 2023
c5b0f14
Remove code that fails to remove notifications on clear.
rtibbles Apr 5, 2023
3dad265
Update clear task handler to consistent signature.
rtibbles Apr 5, 2023
803f639
Remove potentially destructive System.exit call.
rtibbles Apr 6, 2023
14a7159
Move handling of notifications to a separate class.
rtibbles Apr 6, 2023
f24b159
Re add system.exit to destroy process once we've torn down.
rtibbles Apr 6, 2023
ffaffd2
Update to latest python for android.
rtibbles Apr 6, 2023
8de5db7
Prevent module level autoclass declarations that could be shared acro…
rtibbles Apr 17, 2023
897bb77
Defer library loading and teardown to parent service.
rtibbles Apr 17, 2023
7e74447
Update makefile and readme for more explanation and effective pyc del…
rtibbles Apr 17, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .github/workflows/build_apk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,12 @@ jobs:
# This is where python for android puts its intermediary build
# files - we cache this to improve build performance, but be
# aggressive in clearing the cache whenever any file changes
# in the repository.
path: ~/.local
key: ${{ runner.os }}-local-${{ hashFiles('*') }}
# in the repository, especially as we commit files to this folder
# too, so we don't want the cache to override these files.
path: ./python-for-android
key: ${{ runner.os }}-python-for-android-${{ hashFiles('*') }}
restore-keys: |
${{ runner.os }}-local-
${{ runner.os }}-python-for-android-
- uses: actions/cache@v2
with:
path: ~/.cache/pip
Expand Down
16 changes: 16 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,22 @@
src/kolibri
tmpenv
whl/*
python-for-android/build/
python-for-android/packages/**
python-for-android/dists/kolibri/_python*
python-for-android/dists/kolibri/.gradle
python-for-android/dists/kolibri/build
python-for-android/dists/kolibri/gradle
python-for-android/dists/kolibri/jni
python-for-android/dists/kolibri/libs
python-for-android/dists/kolibri/templates
python-for-android/dists/kolibri/obj
python-for-android/dists/kolibri/src/res_initial
python-for-android/dists/kolibri/src/main/assets/private.tar
python-for-android/dists/kolibri/src/main/AndroidManifest.xml
python-for-android/dists/kolibri/webview_includes
python-for-android/dists/kolibri/*.*
python-for-android/dists/kolibri/gradlew

# File format for signing key
*.jks
Expand Down
7 changes: 2 additions & 5 deletions .p4a
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,9 @@
--ndk-api 23
--permission ACCESS_NETWORK_STATE
--permission FOREGROUND_SERVICE
--worker TaskWorker:taskworker.py
--service remoteshell:remoteshell.py
--presplash assets/icon.png
--presplash-color #FFFFFF
--icon assets/icon.png
--fileprovider-paths src/fileprovider_paths.xml
--add-asset assets/_load.html:_load.html
--whitelist ./allowlist.txt
--blacklist ./blocklist.txt
--add-source src/java/
--storage-dir ./python-for-android
49 changes: 36 additions & 13 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ needs-android-dirs:
# Clear out apks
clean:
- rm -rf dist/*.apk src/kolibri tmpenv
- find ./src -name '*.pyc' -exec rm -f {} +

deepclean: clean
$(PYTHON_FOR_ANDROID) clean_dists
Expand Down Expand Up @@ -72,9 +73,34 @@ src/kolibri: clean
# patch Django to allow migrations to be pyc files, as p4a compiles and deletes the originals
sed -i 's/if name.endswith(".py"):/if name.endswith(".py") or name.endswith(".pyc"):/g' src/kolibri/dist/django/db/migrations/loader.py

# Checks to see if we have any uncommitted changes in the Android project
# use this to prevent losing uncommitted changes when updating or rebuilding the P4A project
.PHONY: check-android-clean
check-android-clean:
@git diff --quiet --exit-code python-for-android || (echo "python-for-android directory has uncommitted changes in the working tree" && exit 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this just displayed as a warning? Or does the exit 1 make it bail, I guess? Would one not want to be able to build while the P4A is still being iterated.

Probably I'm also misinterpreting these new make targets. Could you add a one-line comment on each as to what it does?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can build while the Java files are being actively edited - we just can't reimport from the bootstrap while that is the case. Will add some comments to make it clearer.


# Create the python-for-android project bootstrap from scratch
.PHONY: p4a_android_distro
p4a_android_distro: needs-android-dirs
p4a_android_distro: needs-android-dirs check-android-clean
rm -rf python-for-android/dists/kolibri
$(P4A) create $(ARCH_OPTIONS)
# Stash any changes to our python-for-android directory
@git stash push --quiet --include-untracked -- python-for-android

# Update the python-for-android project bootstrap, discarding any changes that are made to committed files
# this should be the usually run command in normal workflows.
.PHONY: p4a_android_project
p4a_android_project: p4a_android_distro src/kolibri needs-version
$(P4A) bootstrap $(ARCH_OPTIONS) --version=$(APK_VERSION) --numeric-version=$(BUILD_NUMBER)
# Stash any changes to our python-for-android directory
@git stash push --quiet --include-untracked -- python-for-android

# Update the python-for-android project bootstrap, keeping any changes that are made to committed files
# this command should only be run when it is known there is an update from the upstream p4a bootstrap
# that is needed, although it will probably normally be easier to manually vendor the changes.
.PHONY: update_project_from_p4a
update_project_from_p4a: p4a_android_distro src/kolibri needs-version
$(P4A) bootstrap $(ARCH_OPTIONS) --version=$(APK_VERSION) --numeric-version=$(BUILD_NUMBER)

.PHONY: needs-version
needs-version:
Expand All @@ -83,38 +109,35 @@ needs-version:

.PHONY: kolibri.apk
# Build the signed version of the apk
# For some reason, p4a defauls to adding a final '-' to the filename, so we remove it in the final step.
kolibri.apk: p4a_android_distro src/kolibri needs-version
kolibri.apk: p4a_android_project
$(MAKE) guard-P4A_RELEASE_KEYSTORE
$(MAKE) guard-P4A_RELEASE_KEYALIAS
$(MAKE) guard-P4A_RELEASE_KEYSTORE_PASSWD
$(MAKE) guard-P4A_RELEASE_KEYALIAS_PASSWD
@echo "--- :android: Build APK"
$(P4A) apk --release --sign $(ARCH_OPTIONS) --version=$(APK_VERSION) --numeric-version=$(BUILD_NUMBER)
cd python-for-android/dists/kolibri && ./gradlew clean assembleRelease
mkdir -p dist
mv kolibri-release-$(APK_VERSION).apk dist/kolibri-release-$(APK_VERSION).apk
cp python-for-android/dists/kolibri/build/outputs/apk/release/kolibri-release.apk dist/kolibri-release-$(APK_VERSION).apk

.PHONY: kolibri.apk.unsigned
# Build the unsigned debug version of the apk
# For some reason, p4a defauls to adding a final '-' to the filename, so we remove it in the final step.
kolibri.apk.unsigned: p4a_android_distro src/kolibri needs-version
kolibri.apk.unsigned: p4a_android_project
@echo "--- :android: Build APK (unsigned)"
$(P4A) apk $(ARCH_OPTIONS) --version=$(APK_VERSION) --numeric-version=$(BUILD_NUMBER)
cd python-for-android/dists/kolibri && ./gradlew clean assembleDebug
mkdir -p dist
mv kolibri-debug-$(APK_VERSION).apk dist/kolibri-debug-$(APK_VERSION).apk
cp python-for-android/dists/kolibri/build/outputs/apk/debug/kolibri-debug.apk dist/kolibri-debug-$(APK_VERSION).apk

.PHONY: kolibri.aab
# Build the signed version of the aab
# For some reason, p4a defauls to adding a final '-' to the filename, so we remove it in the final step.
kolibri.aab: p4a_android_distro src/kolibri needs-version
kolibri.aab: p4a_android_project
$(MAKE) guard-P4A_RELEASE_KEYSTORE
$(MAKE) guard-P4A_RELEASE_KEYALIAS
$(MAKE) guard-P4A_RELEASE_KEYSTORE_PASSWD
$(MAKE) guard-P4A_RELEASE_KEYALIAS_PASSWD
@echo "--- :android: Build AAB"
$(P4A) aab --release --sign $(ARCH_OPTIONS) --version=$(APK_VERSION) --numeric-version=$(BUILD_NUMBER)
cd python-for-android/dists/kolibri && ./gradlew clean bundleRelease
mkdir -p dist
mv kolibri-release-$(APK_VERSION).aab dist/kolibri-release-$(APK_VERSION).aab
cp python-for-android/dists/kolibri/build/outputs/bundle/release/kolibri-release.aab dist/kolibri-release-$(APK_VERSION).aab

# DOCKER BUILD

Expand Down
43 changes: 30 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@ This project was primarily developed on Docker, so this method is more rigorousl

4. The generated APK will end up in the `bin/` folder.

## Building for Development
## Development Flow

1. Install the Android SDK and Android NDK.
1. Setup a Python virtual environment in which to do development. The Kolibri developer documentation has a [How To guide for doing this with pyenv](https://kolibri-dev.readthedocs.io/en/develop/howtos/pyenv_virtualenv.html) but any Python virtualenv should work.

2. Install the Android SDK and Android NDK.

Set the `ANDROID_HOME` environment variable to the location you would like these to be installed, e.g.:
`export ANDROID_HOME=./android_root`
Expand All @@ -25,22 +27,33 @@ Run `make setup`.

Follow the instructions from the command to set the additional environment variables.

2. Install the Python dependencies:
3. Install the Python dependencies:

`pip install -r requirements.txt`

3. Ensure you have all [necessary packages for Python for Android](https://python-for-android.readthedocs.io/en/latest/quickstart/#installing-dependencies).
4. Ensure you have all [necessary packages for Python for Android](https://python-for-android.readthedocs.io/en/latest/quickstart/#installing-dependencies).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth above these pieces suggesting a venv?

5. Build or download a Kolibri WHL file, and place it in the `whl/` directory.

To download a Kolibri WHL file, you can use `make get-whl whl=<URL>` from the command line. It will download it and put it in the correct directory.

6. By default the APK/AAB will be built for most architectures supported by Python for Android. To build for a smaller set of architectures, set the `ARCHES` environment variable. Run `p4a archs` to see the available targets.

7. Run `make p4a_android_project` this will do all of the Python for Android setup up until the point of actually building an APK or AAB.

N.B. You will need to rerun this step any time you update the Kolibri WHL file you are using, or any time you update the Python code in this repository.

8. You can now run Android Studio and open the folder `python-for-android/dists/kolibri` as the project folder to work from. You should be able to make updates to Java code, resource files, etc. using Android Studio, and build and run the project using Android Studio, including launching into emulators and real physical devices.

4. Build or download a Kolibri WHL file, and place it in the `whl/` directory.
N.B. When you rerun step 7, it will complain loudly and exit early if you have uncommitted changes in the python-for-android folder. Any changes should be committed (even if in a temporary commit) before rerunning this step, as we use git stash to undo any changes in the Android project caused by the Python for Android project bootstrapping process. Also, when rerunning step 5, the Android version will not have incremented, meaning that any emulator or physical device will need to have Kolibri explicitly uninstalled for any changes to Python code to be updated on install.

To download a Kolibri WHL file, you can use `make whl=<URL>` from the command line. It will download it and put it in the correct directory.
## Debugging the app

1. When running the app from Android Studio, if you are using an emulator, it is possible that there will be many warning messages due to GPU emulation. In the logcat tab, update the filter to this `package:mine & -tag:eglCodecCommon` to hide those errors from the logcat output.

5. By default the APK will be built for most architectures supported by
Python for Android. To build for a smaller set of architectures, set
the `ARCHES` environment variable. Run `p4a archs` to see the
available targets.
## Building from the commandline

6. Run `make kolibri.apk.unsigned` to build the apk. Watch for success at the end, or errors, which might indicate missing build dependencies or build errors. If successful, there should be an APK in the `dist/` directory.
1. Run `make kolibri.apk.unsigned` to build the development apk. Watch for success at the end, or errors, which might indicate missing build dependencies or build errors. If successful, there should be an APK in the `dist/` directory.

## Installing the apk
1. Connect your Android device over USB, with USB Debugging enabled.
Expand All @@ -52,8 +65,6 @@ To download a Kolibri WHL file, you can use `make whl=<URL>` from the command li

1. Run `adb shell am start -n org.learningequality.Kolibri/org.kivy.android.PythonActivity`

## Debugging the app

### Server Side
Run `adb logcat -v brief python:D *:F` to get all debug logs from the Kolibri server

Expand Down Expand Up @@ -102,3 +113,9 @@ Host kolibri-android
HostkeyAlgorithms +ssh-rsa
```
Then, you should be able to just do “ssh kolibri-android”

## Updating Python for Android

We maintain a fork of Python for Android that includes various changes we have made to the source code to support our specific needs. As P4A make new releases, we make a branch from the latest release tag, and then replay the commits on top of this tag using an interactive rebase. Sometimes, this allows us to drop commits as new features are merged into P4A. Our naming convention for the branch on our fork is `from_upstream_<tag_name>`. Any time we push new commits to this branch, we must also update the pinned commit in `requirements.txt`, so that we are always building with a completely predictable version of Python for Android.

By default we stash any updates to our bootstrap coming from Python for Android, because mostly we have overwritten their bootstrap code to make the relevant changes for us. If there are upstream changes to code we have committed in this repo from the bootstraps, then if the diff is small, it is probably simplest to manually copy in these changes to our committed code. If the diff is larger, or the developer fancies exercising some git-fu, then the make command `make update_project_from_p4a` will update the bootstrap from Python for Android, and not stash any changes that introduces. Through judicious change reversion and diffing, the appropriate changes can then be applied. Here be dragons.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@

h1 {
font-size: 30px;
color: blue;
font-weight: bold;
text-align:center;
}

h2 {
text-align:center;
}

button {
margin-left: auto;
margin-right: auto;
display: block;
margin-top: 50px;
font-size: 30px;
}


/* Loader from http://projects.lukehaas.me/css-loaders/#load1 */

.loader,
.loader:before,
.loader:after {
background: #aaaaff;
-webkit-animation: load1 1s infinite ease-in-out;
animation: load1 1s infinite ease-in-out;
width: 1em;
height: 4em;
}
.loader:before,
.loader:after {
position: absolute;
top: 0;
content: '';
}
.loader:before {
left: -1.5em;
}
.loader {
text-indent: -9999em;
margin: 8em auto;
position: relative;
font-size: 11px;
-webkit-animation-delay: -0.16s;
animation-delay: -0.16s;
}
.loader:after {
left: 1.5em;
-webkit-animation-delay: -0.32s;
animation-delay: -0.32s;
}
@-webkit-keyframes load1 {
0%,
80%,
100% {
box-shadow: 0 0 #aaaaff;
height: 4em;
}
40% {
box-shadow: 0 -2em #aaaaff;
height: 5em;
}
}
@keyframes load1 {
0%,
80%,
100% {
box-shadow: 0 0 #aaaaff;
height: 4em;
}
40% {
box-shadow: 0 -2em #aaaaff;
height: 5em;
}
}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
package org.jnius;
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;

public class NativeInvocationHandler implements InvocationHandler {
static boolean DEBUG = false;
private long ptr;

public NativeInvocationHandler(long ptr) {
this.ptr = ptr;
}

public Object invoke(Object proxy, Method method, Object[] args) {
if ( DEBUG ) {
System.out.print("+ java:invoke(<proxy>, ");
// don't call it, or recursive lookup/proxy will go!
//System.out.print(proxy);
//System.out.print(", ");
System.out.print(method);
System.out.print(", ");
System.out.print(args);
System.out.println(")");
System.out.flush();
}

Object ret = invoke0(proxy, method, args);

if ( DEBUG ) {
System.out.print("+ java:invoke returned: ");
System.out.println(ret);
}

return ret;
}

public long getPythonObjectPointer() {
return ptr;
}

native Object invoke0(Object proxy, Method method, Object[] args);
}
Loading