ABI filtering for assets

[henry-filgueiras]

Hello,

My current understanding (please correct me if I'm wrong) is that that today, ABI-based filtering is enabled for the libs folder for native libraries.

In other words, anything placed under lib/<arch>/... will be appropriately selected/delivered, based on the target device's ABI.

I have a use case to select assets based on ABI.

Is there a way to configure bundletool to do the same type of ABI filtering, but within the assets folder? Any folder other than lib would probably work, but assets seems like a good starting point.

If it's possible, is there some guidance and/or practical example that I could work with to accomplish ABI filtering for assets?

If not, are there any thoughts on how this functionality could be added to bundletool?

Looking around, I found this which seemed interesting, but it's unclear how it's expected to be used:

https://github.com/google/bundletool/blob/master/src/main/proto/config.proto#L107

Found this too, which might imply we can't do ABI filtering unless it's explicitly part of that list?

bundletool/src/main/java/com/android/tools/build/bundletool/model/targeting/TargetedDirectorySegment.java

Line 59 in 7425c0d

private static final ImmutableMap<String, TargetingDimension> KEY_TO_DIMENSION =

Any help would be appreciated. Thanks!

[ymakhno]

ABI dimension is only applicable to libs folder because this folder contains native libraries per each binary interface. Could you please share your usecase, why do you want assets to be split by ABI?

[henry-filgueiras]

We want to do specialized encoding of some of our .so files, and ship them within in the assets folder.

Immediate use case is to apply a compression algorithm with better compression factor to a subset .so files, in order to save on apk download size.

Those files require some post-processing within our app (such as decompression), which will write the libraries to disk before triggering a System.load.

Shipping multiple compressed SO files for each architecture within a single APK can easily defeat the purpose of doing the specialized compression in the first place, so we want to do ABI-based filtering of the assets folder.

Let me know if I can clarify anything, thanks.

[ymakhno]

I don't think we plan to support native libraries in the assets folder as assets folder is mainly for raw data which is delivered as-is and files inside this folder don't have any structure.

I recommend storing native libraries inside the libs folder as we are already doing some optimisations for them based on a device. For example on new Android devices native libraries are stored in uncompressed format inside APK which allows them to be loaded without extracting to disk so saving space on the device: https://developer.android.com/topic/performance/reduce-apk-size#extract-false. Please note that it doesn't mean that the library is not compressed during delivery as we are generating patches and compress these patches for delivery.

[henry-filgueiras]

Please note that it doesn't mean that the library is not compressed during delivery as we are generating patches and compress these patches for delivery.

Can you be more specific about this? What kind of compression is used here?

How are "first install" and "subsequent update" different, in terms of download size?

How can we determine actual download size (whether it's for first time or subsequent update) if the play store does a bunch of implicit things for us? It makes it hard for us to optimize for download size if we don't know the impact our app bundle makes on the end user.

At the end of the day, the goal here is to minimize download size (and not necessarily app install size) as much as possible, specifically for native libraries.

I don't think we plan to support native libraries in the assets folder as assets folder is mainly for raw data which is delivered as-is and files inside this folder don't have any structure.

FWIW I don't think we're asking that bundle tool to natively support dealing with native libraries in the assets folder. We're asking for ABI filtering support, in assets folder, in bundle tool as a generic feature.

The rest is implementation detail which we are responsible for, so the lack of well-known folder structure shouldn't be relevant since our solution would be implementation-specific anyways.

Is there any other kind of assets filtering supported by bundle tool today?

Can you at least help us identify the gap in functionality? What would it take to add such filtering to the bundle tool?

Ignoring the native library scenario for the moment, it seems that shipping "actual" assets which are optimized for 32 bit versus 64 bit architectures could maybe be an interesting use case (32 vs 64 bit integer types, different padding/alignments, etc). We don't currently have plans to leverage such a behavior outside of shipping compressed libraries, but it's not outside the realm of possibility to enable more advanced optimizations in the future.

I recommend storing native libraries inside the libs folder

To reiterate: we aren't storing native libraries as-is, we are storing them in a custom compressed format within the APK, to be dealt with at runtime within our own logic.

At a minimum, due to how the Android gradle plugin is set up: we would have to ship the compressed files, which will fail loadLibrary calls, with a ".so" suffix (within the lib/ folder) to avoid it being ignored by the build system when constructing artifacts for the play store. This seems like a hacky way to get around a lack of asset ABI filtering in the bundle tool.

Another problem here (I think you implied this) is that package manager will unpack the entire lib/ folder to disk on startup (even non-SO files if you manage to get them into the lib/ folder), and it seems we can't get around that without extractNativeLibs = false.

I've been told by folks internally that extractNativeLibs = false has potential issues related to Android 6, and we don't yet want to take that leap.

Also, as mentioned elsewhere, zstd (or even XZ) compression tends to beat zlib in terms of compression ratio, especially with native libraries that are built with heavy optimizations turned on.

We can look into asset delivery in parallel. Using this new feature will take some non-trivial effort on our side to make the build setup do the right thing, since we don't already use it, and it's unclear what impact using this feature will have on the app in a broader sense (if it makes things slower/less reliable/etc).

It seems adding functionality to bundletool would be a much more pragmatic solution here, especially if there are other existing asset filters and we can bolt on something similar for ABI filtering.

Thanks in advance for the attention, and let me know if I can clarify anything.