Improve Android package management for the NDK

[DanAlbert]

This bug tracks the Easier access to common open-source libraries item on our roadmap.

There are many other commonly-used libraries (such as Curl and BoringSSL) that are currently difficult to build/package, let alone keep updated. We should offer (a) a tool to build open source projects, (b) a repository of prebuilts, (c) a command-line tool to add prebuilts to an ndk-build/cmake project, and (d) Studio integration to add prebuilts via a GUI.

We currently have cdep, but it doesn't have a very large corpus and could use some build system integration polish. If not cdep, adding support for exposing native libraries from AARs would be an alternative.

Some questions about this came up on another bug, specifically in what we were planning to use and why, so I wanted to publish the requirements that we had for this, show what we'd considered, and explain the current plan.

Note that this isn't the full design document. I've spared the nitty gritty details since I don't think they're particularly interesting at this time, but if anyone has questions just ask and I can elaborate :)

Requirements

1. Works on Linux, Darwin, and Windows build hosts.
2. Supports arbitrary build systems.
3. Able to deliver projects employing a variety of architectures:
   • Header only libraries.
   • Single library projects.
   • Multi library packages.
4. Able to describe complex library use:
   • Export include paths to library users.
   • Export cflags and ldflags to library users.
   • Complex include/link ordering requirements such as those found the C++ standard library.
5. Supports binary distributions.
6. Able to fetch from arbitrary sources.
7. Support IDE integration.
8. Supports cross package dependencies.
9. Support for non-Android targets.
10. Support for multi-language projects.

Options

With these requirements, none of the existing options we know of will work as-is. We considered cdep, AAR (Android packages via Maven), vcpkg, and hunter. cdep is C/C++ only, which means packages like GTestJNI can't be distributed with it. AAR doesn't support C/C++ well (it supports C/C++ implementation, but the interface to the library must be Java), which obviously won't work for the NDK. vcpkg doesn't seem to support more complex build system features such as exporting required flags to dependents. Hunter is CMake only.

Of these options, AAR seems like our best bet. It doesn't meet the single most obvious need (being able to actually expose a C++ interface), but unlike the other options it's the only one that could be reasonably modified to fit these requirements. It also is the extant method for distributing Java dependencies for Android, and reusing existing infrastructure is a big plus.

Design

The plan is to build some low level tooling (which I'm currently calling Prefab) for generating build system integration for prebuilt packages and some metadata describing usage requirements and then extending gradle and AAR to make use of that.

When I say low level, what I mean is that Prefab only solves the build system integration part of the problem. For an Android gradle project, gradle is still responsible for resolving and fetching dependencies. When gradle fetches an AAR that contains Prefab artifacts, it extracts those to a directory and calls prefab to have it generate build system integration. The generated output is then passed to ndk-build or CMake to be used as you would any other dependency. This means that if your organization uses some other mechanism for distributing dependencies, Prefab artifacts can be distributed that way as well (including something as simple as a git submodule).

For example, your build.gradle would contain something like the following:

dependencies {
    implementation 'com.google.gtest:gtest:1.0'
    implementation 'com.google.gtestjni:gtestjni:1.0'
}

and your CMakeLists.txt would contain:

find_package(gtest 1.0 REQUIRED MODULE)
find_package(gtestjni 1.0 REQUIRED MODULE)

# Additional targets for building the actual app library...

add_library(apptest
  SHARED
    src/test/cpp/app_test.cpp
)

target_link_libraries(apptest
  PRIVATE
    app
    gtest::gtest
    gtestjni::gtestjni
)

And that's it. Gradle will fetch the dependencies, call Prefab to generate the CMake package definitions, and find_package will find the dependencies.

Build system integration

Build system integration is provided by plugins for Prefab, and we'll include CMake and ndk-build plugins by default. If you're using a custom build system, you can provide a path to a plugin jar on the command line to teach Prefab how to generate what you need.

Non-Android targets

Support for non-Android targets will be simple to add (basically just a class recognizing and perhaps disambiguating libraries for the platform, such as Android needs to do with multiple ABIs, minSdkVersion targets, etc). We're only planning on implementing support for Android, but patches adding other platforms are welcome.

So, that's the rough outline. If I've missed any major requirements that make this approach not viable for your project, please speak up. We're still in the very early phases of this so it's not too late for large changes.

[Zingam]

Could you post a link to AAR? A simple google search for "aar package manager" doesn't reveal anything.

[DanAlbert]

https://developer.android.com/studio/projects/android-library

tl;dr it's like a jar, but has Android specific pieces. It's distributed via maven like a jar would be.

[alexcohn]

@DanAlbert, I believe that the androidNativeBundle gradle plugin already implements this for AAR, at least partially.

[Zingam]

So this is basically an Android only solution? It's not meant to provide packages targeting Linux, Mac or Windows? I hoped that AAR were something different than Android Archive (AAR) that I didn't know about.

[DanAlbert]

tl;dr: No, it isn't Android specific (see that "Non-Android targets" section of the original post).

To elaborate and hopefully clear up the confusion, it depends on what you mean by "this". There are two separate pieces here:

1. Distribution mechanism (not the topic of this bug, could be anything)
2. Build system integration (the topic of this bug, called Prefab, not Android specific)

There are a large number of good distribution mechanisms out there, so I figure we're better of leaning on those to do what they do best.

Build system integration in the form that we want it, however, doesn't have many good options. Requirements 2, 4, and 10 in particular ruled out the existing options.

This bug describes a new tool (Prefab) that generates build scripts for a set of prebuilt libraries and headers. It plays no part in dependency resolution or distribution.

So, no, Prefab is not Android specific. As the original post states, Android support will be the first implemented, and I personally don't have plans to add other platforms (though Linux might be added just so I can prove to myself that what I have is easily extended to support additional platforms), but we'll accept patches for other platforms (within reason; if you want VAX support you can maintain your own fork).

[Firefly35]

Considering native builds, did you have a look to conan ?

[DanAlbert]

From a quick read, that seems to check must of the boxes. I'll have a more thorough look later today to see if that can work for us. My initial concern is that it doesn't support multi language packages, but that may be something we can add.

Thanks for the pointer. Hadn't seen that one before.

[DanAlbert]

Huh. Having now read further, this actually looks a lot like what I'd designed, but even includes some of the stretch goals I haven't discussed here (source distribution, namely). The only requirement we have that I don't think this covers is the ability to play well with mixed Java/C++ packages, but at a glance that might still be doable?

Conan looks incredibly well thought out and executed, so I'm going to play with this a bit and see if it can be made to fit all our requirements.

[DanAlbert]

It looks like Conan can also be made to package a jar quite easily. Seems like the only thing we would need to do would be to build a gradle plugin that can handle Conan (it might be a part of the Android gradle plugin, if that integrates better).

[Firefly35]

On my side, I started the development of native dependencies management some years ago, and started to define coherent compile/link rules for cross-platform development, before conan gets stable.
The build rules handle for instance static/dynamic runtime issues on windows.
The next release will also provide a simple macro to switch to brew llvm on Mac OSX platforms.

Nowadays, the system (recently renamed remaken) has evolved (a release will come soon - the main tool is in the finalization step) to a meta-packaging system.

I integrated conan, vcpkg, but also system's dependent packagers (apt, yum, brew, chocolatey) and the binary format defined at the beginning of the project.

Conan was easy to integrate to remaken, and nowadays remaken offers the versatility to choose dependencies among the various binaries available, and depending on the project specific needs.

If you 're interested, I will provide the link to the (future) github repository hosting the tool.

For the moment, the qmake build rules are available from https://github.com/b-com-software-basis/builddefs-qmake.
Cmake rules are on the go.

The remaken tool roadmap will include cmake compile and link flags generation, remaken format packager from a build folder ...

[Zingam]

Whatever solution you decide on, please, don't require external dependencies - like Python, Java...

[DanAlbert]

The top two choices at this time will require one of those. The NDK/SDK already depend on and include Java and Python, so this isn't anything new.

[DanAlbert]

Wanted to give a quick update since it's been a while:

Spent some time evaluating other options and came to the conclusion that the original plan (discussed in the first post of this thread) was going to offer the best UX for Android.

The tooling to make this work and the gradle plugin changes to make it a transparent part of an AAR have been written. There's still a fair amount of polish that needs to be done on both before it's shippable, but we have a working proof of concept.

The current hope is that this will be available alongside the Studio 3.7 release cycle, which will likely go to stable in the first half of next year (I'm not sure when the canaries will first be available, or which stage will be the first one with these features available). Hoping to publish the source for the tooling far ahead of that, which will include design info and usage docs.

[Zingam]

What I find worrisome about creating a completely new delivery system is who is going to add and maintain new packages? I doubt that the audience of NDK is that huge. Would google add a few popular libraries and that would be it.
Unless you also plan to extend it to other systems (not just Android) like Windows, MacOS, IOS, Linux and thus competing with the already established package manager it will also add more maintenance burden to cross-platform projects.
I am always fond of new shiny stuff and project maintenance is also the most annoying part.

How will this plan fit with this: #1011
What do you think about all that? Is this Android only project?

[DanAlbert]

Is this Android only project?

That's covered in the original post. Also #916 (comment)

How will this plan fit with this: #1011

Orthogonal.

[NewProggie]

This is very good news, as all dependencies in a CMake-based project should always use ˋfind_packageˋ for resolving dependencies. I have written about this topic before as well and demonstrated a straight forward approach for Android- as well as external dependencies here in case you are interested. [1]

[1] http://kai-wolf.me/cpp/android/cmake/gradle/2019/02/18/working-with-cpp-in-android-effectively/

[sinhpn92]

Many commonly-used libraries is very need to use (cURL, openSSL, boringSSl, etc,... ). But hard to build for use in ndk.

[DanAlbert]

BoringSSL is particularly annoying (you need a go toolchain to build it...) It's not particularly useful for apps either, so that's not one we intend to publish (but anyone that wants to do so can of course do so themselves). cURL and OpenSSL are ones that we've already prepped for this, along with jsoncpp, since it was easy and helped me make a demo with the other two.

[DanAlbert]

The underlying implementation of this is now up on GitHub: https://github.com/google/prefab (expect a lot of broken URLs; I only just published this so I haven't setup the matching github.io page yet). While it's a big step closer to finishing off this bug, it's just one piece. We still need to:

1. Finish up enough of the TODOs that we can publish something to Maven for the Android Gradle Plugin to pick up.
2. Submit the AGP support.
3. Fix AGP to automatically package CMake import targets (this is an pre-existing issue, but we need it to be fixed for this to have good UX in Studio with CMake).
4. Build up a corpus of packages to make this worth using at launch (currently have OpenSSL, cURL, and jsoncpp ready, but will probably add things like libpng, libjpeg, zlib, etc).
5. Ship it!

Should still be on track to have this available in AGP 4.0.

[sonicdebris]

Out of curiosity, which are the reasons that made you discard Conan in the end?

[H-G-Hristov]

Is there a wishlist/roadmap of which libraries you (Google) will support?

As a part of the NDK users are multimedia developers I'd expect that at least the most popular multimedia/graphics/audio libraries to be included: freetype2, SDL2, OpenAL Soft, opus, vorbis, ffmpeg, some VP9, av1 libraries, etc

[DanAlbert]

Out of curiosity, which are the reasons that made you discard Conan in the end?

Seamless integration with Gradle was one reason.

The largest concern though is that with conan the user's machine has to build the packages, not just fetch them. The building part is one of the things we're trying to avoid doing. It adds unnecessary time to the user's build and almost certainly will result in "works on my machine". With how much code out there still uses autoconf and similar tools, that would likely be a large problem for Windows users. I don't think this has been much of a problem with Conan so far because primarily the people building things on Windows are also the people building things for Windows. For Android development there would likely be many Linux-only packages that would not build on Windows.

We also are able to do a bit better with ABI matching than Conan can. Conan requires an exact match of the build settings across modules, so the common case for Android where a library is built for android-16 but the app is targeting 21 would not work. With Prefab we can currently match that, and by the end of the week it should be improved to allow finding a best match if there's more than one valid match (google/prefab#8).

Is there a wishlist/roadmap of which libraries you (Google) will support?

I've already taken care of OpenSSL, cURL, and jsoncpp. The cURL/OpenSSL use case was our benchmark for "have we done this right at all", and jsoncpp tied those together into a presentable sample.

Someone I spoke with at ADS requested LuaJIT, and @alexcohn has a big list for us over at #456 (comment).

The Prefab bug tracker is going to be the home for package requests. I haven't filed any of those I just mentioned yet, but feel free to start populating it: https://github.com/google/prefab/issues/new/choose (use the "Package request" template).

We're going to have to think about which packages we commit to, since each one will have a recurring cost that comes with it. We fortunately do have some tooling that helps us keep things up to date in AOSP that I'm hoping we can extend to cover this use case, but I'm particularly worried about testing these.

That is to say, please file those requests, but be patient while we figure out what we can and can't afford to support right away.

Based on what I've seen people struggle with on SO, I think that from your list SDL2 and ffmpeg are definitely two that we want to support.

[DanAlbert]

A quick update on where this stands:

• Finish up enough of the TODOs that we can publish something to Maven for the Android Gradle Plugin to pick up.
• Submit the AGP support. (basically done, waiting for final review)
• Fix AGP to automatically package CMake import targets (this is an pre-existing issue, but we need it to be fixed for this to have good UX in Studio with CMake). (Written and reviewed but not submitted)
• Build up a corpus of packages to make this worth using at launch (currently have OpenSSL, cURL, and jsoncpp ready, but will probably add things like libpng, libjpeg, zlib, etc).
• Ship it!

I've just sent a PR adding a sample: android/ndk-samples#678. You won't be able to build it yet since the AGP change hasn't shipped yet, but that should be happening soon (I don't know the exact date, but early January is likely). Once that's done, you'll also need to build the AARs that it imports yourself because those also need to be published, which will happen later in January.

[DanAlbert]

FYI the Gradle portion of this was included in AGP 4.0.0-alpha09. An initial set of packages, docs, and a blog post are still on the way.

[DanAlbert]

The curl, openssl, and jsoncpp packages got pushed to maven today. The sample is still pending (android/ndk-samples#678), but should now be a usable starting point for anyone wanting to poke at this.

Bugs should go to https://issuetracker.google.com/issues/new?component=803099 (if you're sure it's a prefab bug and not a bug in the gradle plugin, can also file at https://github.com/google/prefab/issues, but prefer Issue Tracker if you're not sure).

I'm just waiting for final approval on the docs and the blog post so those should be live within a week.

[Zingam]

I am waiting for the docs to try out this feature. And maybe contribute a couple of packages if possible.
I wasn't going to ask before reading the docs in full but I couldn't wait. Are these prefabs precompiled libraries? Or do they contain source code that then gets compiled before use?

https://github.com/google/prefab/blob/master/docs/index.md

Prefab is a tool for generating build system integrations for prebuilt C/C++ libraries.

I don't understand are only binaries distributed by the system or source code or both?

[DanAlbert]

@Zingam the sample is live now: https://github.com/android/ndk-samples/tree/master/prefab/curl-ssl. The docs provide some explanations for things, but that might be enough to get you started playing with it. I'm hoping to have the docs live before the end of the day, by we're still polishing them up a bit.

As for building your own packages, how easy that is depends on what package you want to ship. https://android.googlesource.com/platform/tools/ndkports/ is what we're using the build curl, openssl, and jsoncpp. It can technically handle any project that you have the source for, but it would be annoying to use for iterative development. Later on (4.1, I'm hoping) I'll get the gradle plugin to do this for regular android library modules, and that'll be the thing that really enables people to build these themselves.

I don't understand are only binaries distributed by the system or source code or both?

It's binary (and headers) only. https://google.github.io/prefab/ explains some of the motivations, and the blog post will elaborate. The simplest answer is that Windows exists and it's really hard to build autotools projects on an arbitrary Windows machine, but that's not the only reason.

[DanAlbert]

btw, for anyone trying to use this with canary 9, apparently canary 9 defaults to prefab-1.0.0-alpha3 which is lacking google/prefab#47 and google/prefab#49, so you'll want to add the following to your gradle.properties alongside the property to enable prefab:

# Will need this until 4.1 or later
android.enablePrefab=true
# Will need this until 4.0 beta 2
android.prefabVersion=1.0.0-alpha5

[alexcohn]

While this feature was under development, support for GitHub Packages has been announced – a solution that helps to distribute built projects, especially the ones hosted on GitHub.

They already have gradle integration, and probably the distance from full support for Prefab should be minimal.

[DanAlbert]

For Android projects, no extra work needed. Prefab packages are already distributed as AARs through Maven, and it looks like GitHub packages already supports that. Definitely an option for folks to consider.

Using it for non-Android projects is a different story :) The format and tooling are amenable to it, at least, it just isn't fleshed out.

[Zingam]

@DanAlbert Thank you very much for taking your time to explain me and I'd like to apologize for the stupid questions. I was confused about the workings of these features.
So as I understand you do not have a goal to provide a source package manager integration behind this infrastructure that can pull from sources, build and provide prefabs (which also include sources) like vcpkg, conan or whatever other there are.
As I don't see others commenting I'd like to share my experience on a project for that very big, very famous platform and their very famous online services supported by that very big software company (not Google). On a quarterly basis they would release a new SDK which we had to upgrade to. We could only lock to a specific SDK at release. Integrating the online services was very troublesome as the provided library had a simple and also very complex API. We used the simple API. Yet with each release new very strange random exceptions would appear. Although they provided sources for the library they didn't provide a proper project file to easily integrate them into hour project and we wished many times to just be able to step into their code and see why where these exceptions happening. What I want to say here with a lot of words is that sometimes it is desirable to also have the source code of the third party library and debug it? Would that be possible to implement with prefabs to provide debugable packages?

What I did for my current project is a standard gradle library module with a special directory where I drop the third party C++ source code. I include the library in my main project and whenever I need to add a new library I just copy that standard library module as necessary.

[DanAlbert]

So as I understand you do not have a goal to provide a source package manager integration behind this infrastructure that can pull from sources, build and provide prefabs (which also include sources) like vcpkg, conan or whatever other there are.

Not all in one piece, no. We will be tackling each of those problems individually, but as far as distribution goes I think distributing binaries and headers is going to be a much better fit for our users given the wide variety of build environments (and everyone's favorite: faster builds).

What I want to say here with a lot of words is that sometimes it is desirable to also have the source code of the third party library and debug it? Would that be possible to implement with prefabs to provide debugable packages?

You don't actually need to build the libraries to debug them; you just need debug info and the sources that match the binaries. Filed google/prefab#65. It's not something I'll be able to look at any time soon, but it's a good idea. Thanks for the suggestion!

[DanAlbert]

Docs are live: https://developer.android.com/studio/preview/features#native-dependencies

They'll be migrated into their own page once 4.0 is out of preview.

[DanAlbert]

https://android-developers.googleblog.com/2020/02/native-dependencies-in-android-studio-40.html

[Zingam]

@DanAlbert Is there an index/an register (planned) where a developer might check for what prefabs are available?

[DanAlbert]

Can find the ones we (the NDK team) own by searching for "ndkports" in the Maven index that's linked in the blog post.

For everything else, the project will probably publish their own links, so they'll be easy to find (Oboe and eventually androidx, for example). We can also keep a list in the prefab wiki.

[alexcohn]

One proposal that should have come to my mind earlier. It would be nice if the same AAR could be used both by C++ and Java developers. I believe that the compiled classes can be inserted into a prefab archive at no cost, but the expected locations of native libraries that should be packaged into the resulting APK or AAB, are different.

[DanAlbert]

That's already covered, and was actually one of the reasons we ended up choosing this over a different option. GTestJni needs that behavior, and its requirements drove that aspect of the design.

If the Java code in the AAR depends on the same libraries that prefab is trying to export, I think that should Just Work, but I haven't tried it. The library will end up duplicated into both the jni and prefab directories, but the size of the AAR won't affect the app at all.

[Zingam]

We can also keep a list in the prefab wiki.

I think that's a good idea.

[alexcohn]

The library will end up duplicated into both the jni and prefab directories

I thought this duplication could be avoided. It looks like zip -yr works fine, and helps to reduce the size of an AAR file signicantly. Should see if the result works correctly on Windows.

[DanAlbert]

The library will end up duplicated into both the jni and prefab directories

I thought this duplication could be avoided.

With some effort it could be done, but I'm not sure it would be worth it.

We can also keep a list in the prefab wiki.

I think that a good idea.

Created https://github.com/google/prefab/wiki/Package-List. Just send a PR if you have a package to add to the list.

[alexcohn]

Created https://github.com/google/prefab/wiki/Package-List. Just send a PR if you have a package to add to the list.

The list should at least link to https://maven.google.com/web/index.html; other packages that will eventually be not on Google's Maven repository, will need more explanations (e.g. GitHub Packages require access token to be imported).

[alexcohn]

The library will end up duplicated into both the jni and prefab directories

I thought this duplication could be avoided.

With some effort it could be done, but I'm not sure it would be worth it.

Unfortunately, this duplication is bad. Not only because it make the AAR almost twice as big (nobody cares about development file sizes these days), not only because maintaining the binaries in two locations to be identical is and additional burden on the developer who chooses to produce the AAR, but mainly because when it is consumed, AGP v.4.0.0-beta04 shows an alarm:

> Task :app:mergeDebugNativeLibs
More than one file was found with OS independent path 'lib/x86_64/libgperf.so'. This version of the Android Gradle Plugin chooses the file from the app or dynamic-feature module, but this can cause unexpected behavior or errors at runtime. Future versions of the Android Gradle Plugin will throw an error in this case.

I tried to use prefab for ndk-samples/hello-libs, but to avoid this warning I generate two separate AAR files for gen-libs module (one of which is a throwaway).

[DanAlbert]

Filed https://issuetracker.google.com/154133927

[alexcohn]

Unfortunately, it seems that the cause of warning is not that the binaries are duplicated inside the AAR, but that the two subsystems in AGP push them to the output bundle after all kinds of transforms having been applied. Note that this happens before packaging, so the standard android.packagingOptions.pickFirst … does not help.

[alexcohn]

Recently I gave a presentation on prefab: https://youtu.be/-QbnbrSUaGA, unfortunately under the limitations of the coronavirus.

[enh-google]

(nice! here's a direct link to alexcohn's slides for that video: https://corecppil.github.io/Meetups/2020-03-37_Core_C++_in_Cyberspace_Online/Package_Management_for_Android_C++_Alex.pdf )