Optimized DEX repair alternative method

[anestisb]

This project is heavily relying to SmaliEx (baksmali/smali wrapper) in order to de-optimize bytecode targets (APKs/JARs) that we want extract from factory system images as part of the vendor proprietary blobs.

While this approach was relatively working until now, it heavily depends into smali & smaliEx maintainers being always up to speed with all recent ART runtime changes (#20, #18). Unfortunately this is practically not the case since teams need a decent amount of time to adjust to Google changes. Plust we need to do a lot of porting / glueing work to upgrade script to match these changes. Considering how often such changes happen in the Android ecosystem, I was looking for an alternative workaround.

Back in Feb 2015 that I've upstreamed oatdump++ to AOSP (https://android-review.googlesource.com/#/c/134380/), I've pushed a DEX export functionality to dump bytecode from .rodata section of the OAT file. Of course the exported DEX bytecode is not the actual original since the DEX-to-DEX transformations have already been applied (https://github.com/anestisb/oatdump_plus#dex-to-dex-optimisations).

However, to my great surprise when I was researching some other aspects of the dex2oat compiler backends, I've noticed that current implementation front-end is not aborting when DEX-to-DEX transformations have been partially or fully applied already to input. Instead the compiler overrides them when it's not happy with the VTABLE indexes. This makes more sense if someone considers the ART profiling functionality where pre-optimized code maybe need to be recompiled without having the original DEX.

This effectively means that the bytecode we want to repair from /system partition can be extracted with oatdump host tool, re-inserted back to APK/JAR and re-compile (pre-optimize) when building from AOSP.

Theoretically dex2oat can happen either at host (pre-optimize) or directly at target when first boot. Until we're confident that this approach works for all bytecode we want to repair for supported devices, we choose to always pre-optimize so potential errors can be spotted as early as possible.

The execution flow of the repairs is:

1. oatdump to dump bytecode from OAT ELF
2. dexrepair (https://github.com/anestisb/dexRepair) to repair DEX CRC checksum
3. Rename bytecode to match naming conventions (classes.dex, classes2.dex, etc.)
4. jar to append bytecode back to source APK/JAR

• zipalign & resign steps are handled automatically from AOSP build chain.

This new method is a work in progress and still needs lots of testing, however so far results (manual repair bytecode samples for N5x target) seems quite promising.

[anestisb]

Undocumented dex2oat --abort-on-hard-verifier-error (https://android.googlesource.com/platform/build/+/android-7.0.0_r4/core/dex_preopt_libart.mk#111) flag is causing issues with some APKs when pre-compiled at host level. Removing that flag and comparing oatdump from original ODEX and regenerated ODEX doesn't reveal any issues of code flow differences.

Looking for alternatives to circumvent this without modifying AOSP makefile.

[anestisb]

AOSP compilation is successfully completed for supported targets. Now testing images against supported devices to ensure that repaired bytecode is working as expected.

[anestisb]

Added support for OS X after building platform/art targeting MAC host tools. Since current Nougat upstream has broken MAC host support at this commit, a small edit had to be introduced to allow oatdump to work under MAC host:

anestisb@blackbird:[art]: git diff
diff --git a/runtime/oat_file.cc b/runtime/oat_file.cc
index 72c0cea..c6a833e 100644
--- a/runtime/oat_file.cc
+++ b/runtime/oat_file.cc
@@ -141,7 +141,7 @@ OatFileBase* OatFileBase::OpenOatFile(const std::string& elf_filename,
                                       std::string* error_msg) {
   std::unique_ptr<OatFileBase> ret(new kOatFileBaseSubType(location, executable));

-  ret->PreLoad();
+  // ret->PreLoad();

   if (!ret->Load(elf_filename,
                  oat_file_begin,

[anestisb]

No issues so far after testing for couple of days with N5x & N6p Nougat builds. Mark as good to go until further notice.