Strategy to convert CompactDex back to StandardDex

[anestisb]

When the Vdex 019 unquicken decompiler is completed (some minor bugs left to be fixed), need to figure out a way to convert CompactDex back to StandardDex. This is required since most tools (e.g. JEB2) and AOSP build infra (see android-prepare-vendor requirements) does not support CompactDex. And most likely they never will since CompactDex is considered an ART internal file.

Some possible approaches are:

• Use libdexlayout from ART
• Try to manually convert CompactDex without implementing a layout IR
• Fork the Dex layout IR from libart and wrap-around a conversion API

All 3 approaches require a significant deal of code reading and testing, with the optimal candidate being the first for the time being. So this might take some time.

[Surge1223]

I'm not sure if this might help, but I was exploring option 1 as well, I was reading code and came across this https://android.googlesource.com/platform/art/+/master/openjdkjvmti/fixed_up_dex_file.cc#113 since it has to return standard dex, and the library itself depends on libdexlayout I thought it might be useful for this goal.

[anestisb]

@Surge1223 I've seen it and already used similar code in an initial patch I tried for the oatdump tool (check it here). Unfortunately it's not very stable (memory OOB issues when processing Vdex files without an ART runtime - aka from a host binary) for reasons I cannot figure out yet. And for the files that it worked, the checksums & file sizes were not matching the original Dex files (garbage 4bytes padding per Dex + shifted data offsets).

I'm currently reading how dexlayout work, and initial evidence indicate that a fully reverted Dex that matches the initial bytecode checksum will not be possible. However, it seems that supporting CompactDex and exporting a functionally identical StandardDex is possible. Will keep the thread updated, in case more people want to contribute.

[anestisb]

I've finished an initial prototype of a new standalone ART utility to convert the CompactDex files back to StandardDex. It is available here. It manually opens input CompactDex files with the DexFileLoader primitive, instead of the ArtDexFileLoader which supports opening only from Vdex containers. Then for each file, libdexlayout is used to process data collections, create IR and generate the standard DexContainers. The main section of the DexContainer is then exported (and optionally verified) to the output directory.

I've tested the tool against a series of vdexExtractor decompiled Android P files I've gathered and so far it works without any major issues being spotted. For each converted DexFile I've also used the dexdump2 utility to verify that the ART dex parsers are happy & can verify the converted file.

It should be noted though that after manually examining a small set of converted Dex files, they're not matching 100% to the original file. Some offsets have been shuffled (e.g. class_data offsets swaped with matching class_annotation offsets), although the Dex files appear to be functionally identical (bytecode verified by diff-ing dexdump2 output against original Dex).

I'll do some more testing and then decide what is the optimal way to distribute the converter tool. Most probably would do something similar to android-prepare vendor, which is compile the baseline libs for Linux & macOS, distribute via a file sharing URL and wrap-around automation via a bash script.

Ideally I would like to create a standalone version of the libdexlayout so I can use it independently of AOSP builds, although handling all the ART lib dependencies atm is very tight due to limited timeframes.

The final step is to raise an issue in the Android tracker and chat with the ART devs on the preferred path to upstream the conversion logic. It would be extremely painful to independently maintain this tool while ART involves. It is very likely that an Android 9.1 will drop with the Pixel 3 release, although hopefully it will get merged in the Android 10 release.

$ vdexExtractor/bin/vdexExtractor -i 019_vdex -o cdex_tmp -f && find cdex_tmp -type f | while read -r file; do /fast-datavault/aosp_prod/out_userdebug_release_Linux_9.0.0_r2/aosp_prod/host/linux-x86/bin/compact_dex_converter -w converted_tmp $file && /fast-datavault/aosp_prod/out_userdebug_release_Linux_9.0.0_r2/aosp_prod/host/linux-x86/bin/dexdump2 -a -d -f -h -o converted_tmp/$(basename "$file").dump converted_tmp/$(basename "$file"); done
[INFO] 151 input files have been added to the list
[INFO] Processing 151 file(s) from 019_vdex
[INFO] 151 out of 151 Vdex files have been processed
[INFO] 77 Dex files have been extracted in total
[INFO] Extracted Dex files are available in 'cdex_tmp'
Opened 'cdex_tmp/BluetoothMidiService.apk_classes.cdex', DEX version '001'
compact_dex_converter I 09-02 11:27:43  1706  1706 compact_dex_converter_main.cc:172] StandardDex file successfully extracted to converted_tmp/BluetoothMidiService.apk_classes.cdex
Opened 'cdex_tmp/CloudPrint2.apk_classes.cdex', DEX version '001'
compact_dex_converter I 09-02 11:27:44  1710  1710 compact_dex_converter_main.cc:172] StandardDex file successfully extracted to converted_tmp/CloudPrint2.apk_classes.cdex
Opened 'cdex_tmp/PrebuiltDeskClockGoogle.apk_classes.cdex', DEX version '001'
compact_dex_converter I 09-02 11:27:44  1714  1714 compact_dex_converter_main.cc:172] StandardDex file successfully extracted to converted_tmp/PrebuiltDeskClockGoogle.apk_classes.cdex
...

[anestisb]

For those interested to experiment, a set of Linux binaries is available here.

[Furniel]

How compact_dex_converter can be build for Windows? "make win_sdk" or Windows build not supported at this moment?

[anestisb]

@Furniel AOSP building in Windows is not currently supported from Google, so compiling the compact_dex_converter for the Windows host is not possible. An alternative would be to compile once on Linux/macOS and then run the arm binaries of the tool in Android devices to do the conversion on the devices/emulators.

I'm looking for DEX IR alternatives that can be built outside AOSP, although couldn't find something decent to work with. dexlib2 primitives are not flexible (plus hate to write Java), facebook redex is strongly coupled with optimization rounds so hard to isolate IR part, and the AOSP ART IR builder is strongly coupled with libart, base libs & Android build env, thus painful to standalonize.

Any further ideas are more than welcome.

[anestisb]

Fixed minor bugs & polished the compact_dex_converter tool (see gist revisions). New release & debug binaries uploaded for Linux and macOS.

[anestisb]

Wrap around scripting has been also pushed to master. You can use the deodex tool to automate the decompilation & conversion of the CompactDex files back to StandardDex.

Will keep the issue open for the time being in case more people want to jump-in with ideas about a usable Dex IR so I can get rid off the AOSP dependencies for the converter tool.

Unfortunately I don't have much more time to work on this project so this is probably as good as it can get for the time being.

[Furniel]

Implemented android P deodexing under windows in my tool.

Strategy to convert CompactDex back to StandardDex under windows(All versions):
You need adb, any android device or emulator and compact_dex_converter tool
Execute from cmd:

adb push compact_dex_converter /data/local/tmp/
adb shell chmod 777 /data/local/tmp/bin/compact_dex_converter
adb push your.cdex /data/local/tmp/classes.cdex
adb shell "/data/local/tmp/compact_dex_converter /data/local/tmp/classes.cdex"
adb pull /data/local/tmp/classes.cdex.new your.dex

Personally i recommend use android x86_64 emulator.

Strategy to convert CompactDex back to StandardDex under Windows 10 x64:
Microsoft introduced WSL(Windows Subsystem for Linux) for windows 10, so windows 10 users can run compact_dex_converter linux binary under windows:

1. Open control panel -> programs and features -> features and check if wsl is enabled
2. Open windows store and install one of linux distributions(Ubuntu, CentOS, Kali and few others supported)
3. Run installed distribution and wait until it will be configured, on last step you will be asked for user name and password
4. Thats all, now you can run compact_dex_converter from wsl command promt

Static compact_dex_converter binaries for linux and android:
compact_dex_converter_android_arm64-v8a.zip
compact_dex_converter_android_armeabi-v7a.zip
compact_dex_converter_android_x86.zip
compact_dex_converter_android_x86_64.zip
compact_dex_converter_linux.zip

Adb Instruction updated, thanks to @anestisb
Added info for windows 10 users

[anestisb]

@Furniel nice. However, you don't need to remount the /data partition nor root access. You just need shell access to upload the cdex files and the compact_dex_converter executable, run it and download the output dex. Simply enabling ADB access should be more than enough.

[Furniel]

@anestisb We can't run binaries from /sdcard on android, so only solution i foud is to use /data partition, but we can't push or pull to data(or even view it) without root. Maybe i missed something? Anyway it's better to use emualtor, not real device.

[anestisb]

@Furniel "/data/local/tmp" should not be under sdcard even for devices that have one. shell user (from adb) should have access to read/write/execute files in that directory based on current DAC/MAC.

[superr]

Thank you @anestisb for the Linux and Mac binaries.
Thank you @Furniel for the static Linux binary.

Unfortunately, none of the Linux binaries work on my 64-bit Ubuntu 16.04.5 install.

$ ./compact_dex_converter services_classes.cdex 
Opened 'services_classes.cdex', DEX version '001'
Illegal instruction (core dumped)

The error is the same with static and shared libs. Any idea what might cause this?
Thanks

[anestisb]

@superr can you run the build with the shared libraries under valgrind and paste the report? It would be great if you can do it on a separate issue so the problem can be tracked properly.

[superr]

Thank you for your response :)

Here comes a glob of my ignorance... I am not familiar with valgrind, and I did not build the compact_dex_converter binaries. I used the ones you provided and the ones @Furniel provided, both with the same results. I did compile vdexExtractor and it is working perfectly.

[Furniel]

@superr
Installed Ubuntu 16.04 on vbox and tested compact_dex_converter - everything works without issues. Are you sure that problem in compact_dex_converter, but not in cdex file?(I've got simillar issue with corrupted cdex file, which was genereted by old vdex extractor version).

[superr]

Thanks @Furniel. I am sure in this case.

1. I am using the latest vdexExtractor source
2. I sent the same file to another user and it converted without issue on their Ubuntu 16.04.

It seems something in my system does not like these binaries, but I am clueless. To be sure, I have attached the cdex file for testing. Thanks :)

services_classes.cdex.zip

[anestisb]

@superr please open a new issue to track the problem you have with the distributed binaries.

valgrind is a dynamic instrumentation framework and can be used to debug applications. Just install it (sudo apt-get install valgrind for your Ubuntu) and run the converter via valgrind (valgrind ./compact_dex_converterd <input_cdex>). Then paste the report in the issue.

[ale8530]

@Furniel great work i have try your compact_dex_converter for linux and work perfect.

@anestisb is possible insert compile tool of @Furniel on your github?

[anestisb]

@ale8530 yes I can to a fully statically compiled binary (if that that is what you mean by insert), although I usually avoid it (mostly due to issues with libc, libgcc_s & libc++ libs across systems).

Is there a reason you prefer the statically compiled binary? All linker paths are relative, so as long as you keep the bin & lib64 dirs under the same path, you can execute from anywhere.

[ale8530]

For those who need to load certain scripts on github deodex can not load all the hostTools folder, which is used to run the file compact_dex_converter, with the tool developed by @Furniel I have the ability to load everything on github without libraries

[anestisb]

@ale8530 I see. I don't like sharing binaries over github thus the solution with the matching API-level wrapping over an external URL. If you modify the gitignore you should be ok sharing the files.

At any case I'll compile all targets and add URLs to the statically compiled binaries too in the README file in case someone wants them. Expect them later today.

[ale8530]

@anestisb Thanks for the great support you provide to the community, take it easy is not urgent :)

[cmstew]

I thought I'd share this with you guys. It's just another way to use anestisb's awesome tool if binaries aren't your thing:

http://vdex2dex.com/

[kevinminions9]

Implemented android P deodexing under windows in my tool.

Strategy to convert CompactDex back to StandardDex under windows(All versions):
You need adb, any android device or emulator and compact_dex_converter tool
Execute from cmd:

adb push compact_dex_converter /data/local/tmp/
adb shell chmod 777 /data/local/tmp/bin/compact_dex_converter
adb push your.cdex /data/local/tmp/classes.cdex
adb shell "/data/local/tmp/compact_dex_converter /data/local/tmp/classes.cdex"
adb pull /data/local/tmp/classes.cdex.new your.dex

Personally i recommend use android x86_64 emulator.

Strategy to convert CompactDex back to StandardDex under Windows 10 x64:
Microsoft introduced WSL(Windows Subsystem for Linux) for windows 10, so windows 10 users can run compact_dex_converter linux binary under windows:

1. Open control panel -> programs and features -> features and check if wsl is enabled

2. Open windows store and install one of linux distributions(Ubuntu, CentOS, Kali and few others supported)

3. Run installed distribution and wait until it will be configured, on last step you will be asked for user name and password

4. Thats all, now you can run compact_dex_converter from wsl command promt

Static compact_dex_converter binaries for linux and android:
compact_dex_converter_android_arm64-v8a.zip
compact_dex_converter_android_armeabi-v7a.zip
compact_dex_converter_android_x86.zip
compact_dex_converter_android_x86_64.zip
compact_dex_converter_linux.zip

Adb Instruction updated, thanks to @anestisb
Added info for windows 10 users

cannot deodex with your tools bro