Reproducible Builds #140
Comments
|
@IzzySoft In these recommendations, it seems that some do not concern the application: I have some questions about some parts of these recommendations:
What I can do today (although I'm not sure if applying only this will solve the problem 🤔): |
|
Nice, thanks! Clean builds: if you use CI to build the app from the (tagged) commit, that should take care for it (wouldn't it usually run in a clean container?). If you build locally: yes, that's almost exactly the command we use with "flaky builds" (those with non-deterministic outcome, producing the "matching" result maybe every 3rd or 5th run, so we let the build run in circles until that "matching" result is achieved): making sure it's a clean tree and no artifacts remaining from previous builds are used. I've heard the equivalent in Android Studio would be running "clean" before "build". JDK: your Compressing Images: going by the above diff, you don't seem affected by that (graphics show separately usually – I'm not aware of graphics inside Now let's see for 2.10.11: -rw-r--r-- 0.0 unx 120 b- 118 defN 1981-01-01 01:01:02 8a77e52d META-INF/version-control-info.textproto
- -rw-r--r-- 0.0 unx 2668 b- 2668 stor 1981-01-01 01:01:02 3bb5d7e3 assets/dexopt/baseline.prof
+ -rw-r--r-- 0.0 unx 2668 b- 2668 stor 1981-01-01 01:01:02 abd4a72f assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 189 b- 189 stor 1981-01-01 01:01:02 0445e85e assets/dexopt/baseline.profm
- -rw-r--r-- 0.0 unx 9278084 b- 3465238 defN 1981-01-01 01:01:02 eb20df91 classes.dex
+ -rw-r--r-- 0.0 unx 9278088 b- 3465214 defN 1981-01-01 01:01:02 b82e816d classes.dex
-rw-r--r-- 0.0 unx 2468 b- 999 defN 1981-01-01 01:01:02 afaef08f classes2.dexOK, embedded commit hash this time matches, -|: new-instance v2, Lcom/best/deskclock/widget/EmptyViewController;
-|: iget-object v3, v5, Lcom/best/deskclock/AlarmClockFragment;.mMainLayout:Landroid/view/ViewGroup;
-|: iget-object v4, v5, Lcom/best/deskclock/AlarmClockFragment;.mRecyclerView:Landroidx/recyclerview/widget/RecyclerView;
-|: invoke-direct {v2, v3, v4, v0}, Lcom/best/deskclock/widget/EmptyViewController;.<init>:(Landroid/view/ViewGroup;Landroid/view/View;Landroid/view/View;)V
-|: iput-object v2, v5, Lcom/best/deskclock/AlarmClockFragment;.mEmptyViewController:Lcom/best/deskclock/widget/EmptyViewController;
+|: new-instance v0, Lcom/best/deskclock/widget/EmptyViewController;
+|: iget-object v2, v5, Lcom/best/deskclock/AlarmClockFragment;.mMainLayout:Landroid/view/ViewGroup;
+|: iget-object v3, v5, Lcom/best/deskclock/AlarmClockFragment;.mRecyclerView:Landroidx/recyclerview/widget/RecyclerView;
+|: iget-object v4, v5, Lcom/best/deskclock/AlarmClockFragment;.mAlarmsEmptyView:Landroid/widget/TextView;
+|: invoke-direct {v0, v2, v3, v4}, Lcom/best/deskclock/widget/EmptyViewController;.<init>:(Landroid/view/ViewGroup;Landroid/view/View;Landroid/view/View;)V
+|: iput-object v0, v5, Lcom/best/deskclock/AlarmClockFragment;.mEmptyViewController:Lcom/best/deskclock/widget/EmptyViewController;
|: new-instance v0, Lcom/best/deskclock/alarms/AlarmTimeClickHandler;
|: iget-object v2, v5, Lcom/best/deskclock/AlarmClockFragment;.mAlarmUpdateHandler:Lcom/best/deskclock/alarms/AlarmUpdateHandler;
|: invoke-direct {v0, v5, v8, v2}, Lcom/best/deskclock/alarms/AlarmTimeClickHandler;.<init>:(Landroidx/fragment/app/Fragment;Landroid/os/Bundle;Lcom/best/deskclock/alarms/AlarmUpdateHandler;)V
@@ -891519,10 +891520,10 @@
catches : (none)
positions :
locals :
- 0x0000 - 0x00e7 reg=5 this Lcom/best/deskclock/AlarmClockFragment;
- 0x0000 - 0x00e7 reg=6 (null) Landroid/view/LayoutInflater;
- 0x0000 - 0x00e7 reg=7 (null) Landroid/view/ViewGroup;
- 0x0000 - 0x00e7 reg=8 (null) Landroid/os/Bundle;
+ 0x0000 - 0x00e9 reg=5 this Lcom/best/deskclock/AlarmClockFragment;
+ 0x0000 - 0x00e9 reg=6 (null) Landroid/view/LayoutInflater;
+ 0x0000 - 0x00e9 reg=7 (null) Landroid/view/ViewGroup;
+ 0x0000 - 0x00e9 reg=8 (null) Landroid/os/Bundle;I'm not sure what to make out of that: looks like the same "code" but assigned a different register? Btw: Building with JDK-17, the DexDiff gets much larger again, so JDK-21 already seems the best choice here at our side. Did you build this with Android Studio? Can you choose OpenJDK there instead of Jetbrains (just to check if that makes a difference, as it sometimes does with different "vendors")? |
|
@IzzySoft JDK:
I am very surprised that you say this because I always publish the application after the commit named "Update gradle, translations, version and fastlane" (or something very similar)
Yes I did. I am using version 2024.2.1 Patch 2.
I can't currently do this because I'm using my company computer which prevents me from installing a lot of software. |
Well, that's what the APK said. There's embedded version info (see the diff:
Fair enough, thanks a lot! I cannot even tell if the Jetbrain SDK is causing a difference there. But if we see nothing else, we need to rule that out (sometimes different SDKs cause differences). |
|
@IzzySoft
You're right... I must have been really tired for that to happen 😬 |
I do know exactly what you mean – am there too often myself…
Wonderful, looking forward to that, thanks! And on the risk of repeating myself: Please remember running "clean" before "build" in Studio. Studio tends to reuse artifacts from prior runs otherwise, I've been told. |
|
Hm, seems to be prety much the same. APK diff: -rw-r--r-- 0.0 unx 120 b- 117 defN 1981-01-01 01:01:02 cd3e6b68 META-INF/version-control-info.textproto
- -rw-r--r-- 0.0 unx 2651 b- 2651 stor 1981-01-01 01:01:02 2b9fd0d4 assets/dexopt/baseline.prof
+ -rw-r--r-- 0.0 unx 2651 b- 2651 stor 1981-01-01 01:01:02 35a31afb assets/dexopt/baseline.prof
-rw-r--r-- 0.0 unx 189 b- 189 stor 1981-01-01 01:01:02 19d3f940 assets/dexopt/baseline.profm
- -rw-r--r-- 0.0 unx 9281136 b- 3465405 defN 1981-01-01 01:01:02 45b2cf3e classes.dex
+ -rw-r--r-- 0.0 unx 9281144 b- 3465394 defN 1981-01-01 01:01:02 4236ef57 classes.dex
-rw-r--r-- 0.0 unx 2468 b- 999 defN 1981-01-01 01:01:02 afaef08f classes2.dexAnd the dex: |: iput-object v0, v5, Lcom/best/deskclock/AlarmClockFragment;.mMainLayout:Landroid/view/ViewGroup;
-|: new-instance v2, Lcom/best/deskclock/alarms/AlarmUpdateHandler;
-|: iget-object v3, v5, Lcom/best/deskclock/AlarmClockFragment;.mContext:Landroid/content/Context;
-|: invoke-direct {v2, v3, v5, v0}, Lcom/best/deskclock/alarms/AlarmUpdateHandler;.<init>:(Landroid/content/Context;Lcom/best/deskclock/alarms/ScrollHandler;Landroid/view/ViewGroup;)V
-|: iput-object v2, v5, Lcom/best/deskclock/AlarmClockFragment;.mAlarmUpdateHandler:Lcom/best/deskclock/alarms/AlarmUpdateHandler;
+|: new-instance v0, Lcom/best/deskclock/alarms/AlarmUpdateHandler;
+|: iget-object v2, v5, Lcom/best/deskclock/AlarmClockFragment;.mContext:Landroid/content/Context;
+|: iget-object v3, v5, Lcom/best/deskclock/AlarmClockFragment;.mMainLayout:Landroid/view/ViewGroup;
+|: invoke-direct {v0, v2, v5, v3}, Lcom/best/deskclock/alarms/AlarmUpdateHandler;.<init>:(Landroid/content/Context;Lcom/best/deskclock/alarms/ScrollHandler;Landroid/view/ViewGroup;)V
+|: iput-object v0, v5, Lcom/best/deskclock/AlarmClockFragment;.mAlarmUpdateHandler:Lcom/best/deskclock/alarms/AlarmUpdateHandler;
|: sget v0, Lcom/best/deskclock/R$id;.alarms_empty_view:I
|: invoke-virtual {v7, v0}, Landroid/view/View;.findViewById:(I)Landroid/view/View;
|: move-result-object v0
|: check-cast v0, Landroid/widget/TextView;
|: iput-object v0, v5, Lcom/best/deskclock/AlarmClockFragment;.mAlarmsEmptyView:Landroid/widget/TextView;
-|: new-instance v2, Lcom/best/deskclock/widget/EmptyViewController;
-|: iget-object v3, v5, Lcom/best/deskclock/AlarmClockFragment;.mMainLayout:Landroid/view/ViewGroup;
-|: iget-object v4, v5, Lcom/best/deskclock/AlarmClockFragment;.mRecyclerView:Landroidx/recyclerview/widget/RecyclerView;
-|: invoke-direct {v2, v3, v4, v0}, Lcom/best/deskclock/widget/EmptyViewController;.<init>:(Landroid/view/ViewGroup;Landroid/view/View;Landroid/view/View;)V
-|: iput-object v2, v5, Lcom/best/deskclock/AlarmClockFragment;.mEmptyViewController:Lcom/best/deskclock/widget/EmptyViewController;
+|: new-instance v0, Lcom/best/deskclock/widget/EmptyViewController;
+|: iget-object v2, v5, Lcom/best/deskclock/AlarmClockFragment;.mMainLayout:Landroid/view/ViewGroup;
+|: iget-object v3, v5, Lcom/best/deskclock/AlarmClockFragment;.mRecyclerView:Landroidx/recyclerview/widget/RecyclerView;
+|: iget-object v4, v5, Lcom/best/deskclock/AlarmClockFragment;.mAlarmsEmptyView:Landroid/widget/TextView;
+|: invoke-direct {v0, v2, v3, v4}, Lcom/best/deskclock/widget/EmptyViewController;.<init>:(Landroid/view/ViewGroup;Landroid/view/View;Landroid/view/View;)V
+|: iput-object v0, v5, Lcom/best/deskclock/AlarmClockFragment;.mEmptyViewController:Lcom/best/deskclock/widget/EmptyViewController;
|: new-instance v0, Lcom/best/deskclock/alarms/AlarmTimeClickHandler;
|: iget-object v2, v5, Lcom/best/deskclock/AlarmClockFragment;.mAlarmUpdateHandler:Lcom/best/deskclock/alarms/AlarmUpdateHandler;
|: invoke-direct {v0, v5, v8, v2}, Lcom/best/deskclock/alarms/AlarmTimeClickHandler;.<init>:(Landroidx/fragment/app/Fragment;Landroid/os/Bundle;Lcom/best/deskclock/alarms/AlarmUpdateHandler;)V
@@ -891542,10 +891544,10 @@
catches : (none)
positions :
locals :
- 0x0000 - 0x00e5 reg=5 this Lcom/best/deskclock/AlarmClockFragment;
- 0x0000 - 0x00e5 reg=6 (null) Landroid/view/LayoutInflater;
- 0x0000 - 0x00e5 reg=7 (null) Landroid/view/ViewGroup;
- 0x0000 - 0x00e5 reg=8 (null) Landroid/os/Bundle;
+ 0x0000 - 0x00e9 reg=5 this Lcom/best/deskclock/AlarmClockFragment;
+ 0x0000 - 0x00e9 reg=6 (null) Landroid/view/LayoutInflater;
+ 0x0000 - 0x00e9 reg=7 (null) Landroid/view/ViewGroup;
+ 0x0000 - 0x00e9 reg=8 (null) Landroid/os/Bundle;I'm a bit clueless. Can you try |
|
@IzzySoft 
Edit: Do you think that before running "Clean" and "Build" I should also do "Invalidate caches" in Android Studio? 🤔 |
|
Yeah, according to this it should be the corresponding Windows command. OK, that does NOT match then either. Now I'm out of clues. Very last straw to pull on: do you get the same has on an APK build from a fresh clode (i.e. checkout the repo in a different place, I have never used Android Studio myself, so all I know about it is "second hand", sorry – but cleaning "everything" should not hurt: cached stuff could indeed play in here, and if "clean" does not take care for that… worth a try. |
|
@IzzySoft I must have made a mistake and I apologize for it as I am not used to using this kind of command. Finally, to clarify this topic, what should I get on my side to know if the Reproducible Builds are compliant? I'm a bit lost... 🤔 Edit 1: It seems that NewPipe has managed to do the Reproducible Build. Do you have any idea what is different compared to our tests? Edit 2: In the build.gradle file of the NewPipe app it is indicated as follows: Do you think I should do the same? But again, how do I know if it's ok without me asking you? |
Nonono. OK, let me be more details. Assuming your code currently resides in cd C:\Users\BlackyHawky\Projects
mkdir test
cd test
git clone -b 2.10.1 https://github.com/BlackyHawky/Clock.git
cd Clock
gradlew assembleReleaseI.e. check out a fresh copy of your project into a clean place (so you get exactly the same code I have here) and build from that. The above gives you a clean tree at the To your "Edit 1": unfortunately, no (there it was just that LK built with JDK-17 while they had switched to 21, which our builders followed up to so we didn't have that issue). To "Edit: 2": that should not affect
I cannot tell. In many cases I, too, only know when I see the results… |
|
Question, are there issues for recent Android versions if apps are built with the lowest possible JDK? |
|
To my knowledge, JDK should not be relevant for that (only |
|
repositories {
system: GIT
local_root_path: "$PROJECT_DIR"
revision: "e00a0d711f9a4aa606c0d43c5c72ee24c741d086"
}Yes, built from e00a0d7, right. Mine too. |
|
@BlackyHawky how many CPU cores do you use when building? This could have something to do with it (see https://issuetracker.google.com/issues/366412380). I've now run builds with 16, 8, 6, 4 and 2 cores, each produce a different APK (could as well be a non-deterministic build (aka "shaky build") in general, should maybe try our "unshaker", will report if that did something). Knowing your number-of-cores would increase our unshaking chances. |
|
Thanks to having multiple builders now, I've utilized them both to run one with 4 and one with 8 cores at the same time (we can set the number of cores in our recipes). I've let the build run in a loop. Both produced 10 times the same hash, though none of them matching yours – and not matching each other. So it's definitely something with cores, but probably not only.
Oh, nice – didn't know that! Looking at the official documantation at Gradle Build Environment Documentation:
and
So the latter would make sense to be set to what your "official builds" have. But: Running with 4 cores, the diff here gets even bigger. Additionally to Was woth a try. Luckily I covered the number of cores you really used, so we can say that does not help – at least not that alone. Something else we might have missed? And should you want to give those gradle settings a try: the hash of the unsigned APK for my builds were:
Tried to many, should have taken notes… |
|
PS, if you want to try it in a loop to see if some setting hits that and if it always produces the same hash: set +x
PROF_FILE=app/build/intermediates/binary_art_profile/release/compileReleaseArtProfile/baseline.prof
PROF_SHA1=6dc56cefc57875c10bdafb34c9ba4f4d4384a424
for _ in {1..10}; do
./gradlew clean assembleRelease --no-build-cache --no-configuration-cache --no-daemon
if [ "$( sha1sum "$PROF_FILE" | cut -d' ' -f1 )" = "$PROF_SHA1" ]; then
break
fi
done |
|
@IzzySoft If I apply this without ever opening Android Studio:
The hash you get is identical to the one you found with 4 cores. 
|
|
@IzzySoft Indeed, in my default folder, if I apply from Android Studio In summary, I think we can conclude:
Am I right, do you think? Below is the apk if you need it: |
|
Great to see we've solved the riddle! Full ack to your "marshal plan". May I suggest we do a test run for that, to get the shield up?
If it works out, you'd just need to tag that commit, make it a release, and attach the APK to that. With that in place, I'd then establish Clock in my builder. |
|
@IzzySoft Quick questions:
|
|
Thanks! As for your points:
|
|
@IzzySoft Commit: Hash: I'm keeping my fingers crossed that everything will run properly. |
|
First try, without nailing it to any fixed CPU number: still the old problem, not RB. Second try, nailing it to 4 cores: different hash, but still the same issue (dex diff looks like the one above). And it seems to be always I gave it a last try with JDK-21, but then the dex diff gets huge. 😢 |
|
@IzzySoft
Is there nothing else I can do? |
|
What about building on the commandline ( |
I get the same apk with the same hash value. I really don't understand what's wrong. 😭 |
|
Maybe it's the fact that I'm using Windows? See issue here on issuetracker.google.com. |
|
I'm clueless now. Thought we had nailed it to be the "number of cores" issue. I can try another time if it's the "flaky build" issue as well (running that now)… Nope, each round the same hash.
👀 Sounds like it. And that issue is from Fay, who confirmed it for multiple gradle versions. Well, sounds like it then, doesn't it? Any chance you could build on Linux (if you don't have a Linux machine/VM yourself, maybe using Github CI)? EDIT: Nope, it's not that issue you've linked to. That we can work around (affected files are just plain text metadata inside |
|
Uhm... I've been skimming over the issue and just tried on macOS: Though I'm not sure if changing permissions of gradlew had anything to do with it: @BlackyHawky if you are developing on windows - maybe trying WSL would help/yield different result/help move it forward? (windows has it's quirks…) |
|
@IzzySoft
So I mainly modified this file and the linked xml file; then I did a "force-push" to overwrite the previous commit. So can you do a new test please? Below is the apk of the new version: Commit: Hash: |
|
Oof, that's not the only thing you changed, huh? OK, working around that: - echo -e "storeFile=/dev/null\nstorePassword=storePass\nkeyAlias=alias\nkeyPassword=keyPass" > keystore.properties
- sed -r '/signingConfigs.release/d' -i app/build.gradleOK, build succeeds, but output file name has changed as well (so it failed to be copied out of the container for comparison), another round then… Wow. I didn't limit CPUs, just to check – but it seems this did the trick: That means this was RB 🥳 Congrats! What a ride, though 🙈 Going for the release then? 😃 |
Indeed, to be able to sign the apk using the command lines, I modified the "build.gradle" file by taking inspiration from this link. (Not sure if this is a correct method but it is very simple to implement and works correctly). Finally, I am very happy to see that my modification made in the "AlarmClockFragment.java" file was able to resolve this issue in the Thank you for trusting me and and thank you for persevering with all these tests! 🎉💪👍👌
If I'm not too tired after finishing my job I will do it later, otherwise tomorrow with my personal computer. 😉 |
That enforces signing – so one couldn't build without a valid keystore (unless modifying the
The latter would require making the signingConfig itself optional, or it would yell out when trying to access it. In my above snippet you see I still had to create a "fake signingConfig". The signingConfig can be wrapped in a "conditional". Pseudo-Code: val propFile = rootProject.file("keystore.properties")
if (propFile.exists()) {
// signingConfig here
}Then, calling it via Just to make sure: not saying you must change it, recipe is working now. Just sharing what I've learned on the topic, by having processed 367 successfully RB recipes in my builder alone 😜 So keep yours as it is if it works for you, and maybe stow away these details for… well, something in the future?
Ah! That explains, thanks!
Hehe, 367+ recipes (not counting the 70+ in my backlog that did not (yet) succeed), you see a lot on the way. Guess I can tell stories at the camp fire about it 🙈
Same, same! That was great teamwork, wasn't it? Thanks a lot for your patience there!!! Oh, and I should not forget the… Ah, wait, we didn't yet add the recipe, still waiting for the next release. Oof, good! OK, later then…
Gladly! It was quite a "bitter ride" – but so much sweeter the fruits of success! 🧺 🥝 🍌 🍒 Thanks again for sticking with me there!
Sure, take a break – well earned! Just give me a ping please once it's there. I'd then do a last test run, and then add the recipe to the builder and report back, so we can close this issue. |
If you don't mind, I wish we could fix this topic in the “build.gradle” file; indeed, even if it works for me (because I have the "keystore.properties" file), I am not a fan of this modification. Also, it would require a very small modification that I should have thought of before (but I was pressed for time because of my job) ; I just have to write: Or else I totally remove this modification about When you think about it the 2 main problems were the different number of cores in our processors and the "AlarmClockFragment.java" file.
I think I'll do this tonight because I'm excited that we're finally validating our app as Reproducible Builds. 😉 |
|
Fine with me! If you make that modification, please don't forget to try signingConfig signingConfigs.findByName("release")(if that works there; I always get confused by what works in
Hehe – and so am I! |
|
@IzzySoft We should note that the output name of the file agrees according to the build type:
Like yesterday, I did a force-push to overwrite the old commit. I hope that on your side, you won't have to adapt too much what you've already done and that it will (finally) be perfect. 😉 Below is the apk of the new version: Commit: Hash: |
"upstream_signed_apk_sha256": "596f01dafca8e88b2a7220b55b86173172b1f2ee03fbd97cf9355b1e999a5074",
"built_unsigned_apk_sha256": "41398898f743a1060796b81330936e105dd869e1f36e11c9184a710c15e9cf66",
"signature_copied_apk_sha256": "596f01dafca8e88b2a7220b55b86173172b1f2ee03fbd97cf9355b1e999a5074"with the simplest possible recipe: build:
- chmod +x gradlew # just to make sure, might not even be needed
- ./gradlew assembleReleaseDone & RB. |
|
Great news!!! 🎉🎉 I'll be releasing the new version shortly. I'll let you know when it's done. |
|
@IzzySoft |
It's done here, too – so 2.11 will arrive at the repo with the next sync around 7 pm UTC, and with the green shield up 🥳 🙌 |
Your app was requested for inclusion with the IzzyOnDroid repo.
At IzzyOnDroid we support Reproducible Builds (see: Reproducible Builds, special client support and more at IzzyOnDroid). Trying for yours, I was able to successfully generate the APK using
./gradlew assembleRelease, but the resulting APKs were not identical. The APK was obviously not built from a clean tree: the embedded commit hash suggests a commit before the tag, but the APK hasversionNameandversionCodewith values committed with the tag. So obviously there were some local changes made after the first commit, then the APK was built, and then there might have been additional changes before the next commit:A diff of the
classes.dexseems to confirm that (that diff is rather big, so I did not include it here).We'd appreciate if you could help making your build reproducible. We've prepared some hints on reproducible builds for that.
Looking forward to your reply!
The text was updated successfully, but these errors were encountered: