bnd-maven-plugin: Support reproducible builds

[kwin]

The Bnd-LastModified header currently makes a build non-reproducible. If existing the fixed timestamp from project.build.outputTimestamp should be used instead (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318) in the Maven plugin.

[rotty3000]

couldn't you simply -removeheaders: Bnd-LastModified?

[kwin]

That would be a workaround, this issue is more to make the bnd-maven-plugin in line with the other maven plugins (i.e. evaluate project.build.outputTimestamp) and then use its value instead of the current date.

[rotty3000]

I tried to use that property but maven didn't seem to think it existed.

…

On Sun, Nov 3, 2019, 17:06 Konrad Windszus, ***@***.***> wrote: That would be a workaround, this issue is more to make the bnd-maven-plugin in line with the other maven plugins (i.e. evaluate project.build.outputTimestamp) and then use its value instead of the current date. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3521?email_source=notifications&email_token=AABD2TBRXC3MLGJCPC5QPB3QR5DONA5CNFSM4JINESJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC56D4A#issuecomment-549183984>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABD2TFP4O2MM63F7CYPACTQR5DONANCNFSM4JINESJA> .

[kwin]

You have to explicitly set it. But it is a well known property which is also evaluated by other plugins (m-jar-p, m-source-p). Further details in https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318

[rotty3000]

Got it, then yeah it should be trivial to check for it.

…

On Mon, Nov 4, 2019, 00:10 Konrad Windszus, ***@***.***> wrote: You have to explicitly set it. But it is a well known property which is also evaluated by other plugins (m-jar-p, m-source-p). Further details in https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74682318 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3521?email_source=notifications&email_token=AABD2TC3J27OYL76FN7XKEDQR6VFBA5CNFSM4JINESJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC6IOKA#issuecomment-549226280>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABD2THKVSIT5ZWC4C2IUGLQR6VFBANCNFSM4JINESJA> .

[bjhargrave]

BTW, Bnd has options for reproducible builds which Bnd itself uses

bnd/cnf/build.bnd

Lines 58 to 59 in a6aed53

	-reproducible: true
	-noextraheaders: true

The latter option removes headers which can vary from build to build like Bnd-LastModified. So if you use these 2 options, you should already be able to make reproducible builds with the maven plugin.

[bjhargrave]

See also https://maven.apache.org/guides/mini/guide-reproducible-builds.html which is new to maven-jar-plugin for the just released 3.2.0. But the use of project.build.outputTimestamp documented there is to use a fixed value rather than the actual current build time.

For the bnd-maven-plugin, if we see -reproducible: true in the Bnd configuration, we should then set the maven project.build.outputTimestamp to the fixed time stamp we use for Bnd reproducible builds.

bnd/biz.aQute.bndlib/src/aQute/bnd/osgi/Jar.java

Lines 71 to 87 in a6aed53

	/**
	* Note that setting the January 1st 1980 (or even worse, "0", as time)
	* won't work due to Java 8 doing some interesting time processing: It
	* checks if this date is before January 1st 1980 and if it is it starts
	* setting some extra fields in the zip. Java 7 does not do that - but in
	* the zip not the milliseconds are saved but values for each of the date
	* fields - but no time zone. And 1980 is the first year which can be saved.
	* If you use January 1st 1980 then it is treated as a special flag in Java
	* 8. Moreover, only even seconds can be stored in the zip file. Java 8 uses
	* the upper half of some other long to store the remaining millis while
	* Java 7 doesn't do that. So make sure that your seconds are even.
	* Moreover, parsing happens via `new Date(millis)` in
	* {@link java.util.zip.ZipUtils}#javaToDosTime() so we must use default
	* timezone and locale. The date is 1980 February 1st CET.
	*/
	private static final long ZIP_ENTRY_CONSTANT_TIME = new GregorianCalendar(1980, Calendar.FEBRUARY, 1, 0, 0, 0)
	.getTimeInMillis();

Note: We should not be setting Bnd-LastModified to the value of project.build.outputTimestamp since that would be a lie. If the user does not want that header, they can either -removeheaders it or, better yet, use -noextraheaders: true .

[rotty3000]

The bnd-maven-plugin is a subordinate in the maven build and so I don't think setting project.build.outputTimestamp would have the desired effect. The goal at most here would be for bnd-maven-plugin to read some signature flags from maven and maybe react by turning on it's reproducible build features.

[bjhargrave]

I guess bnd-maven-plugin could see if that project property is set, and if so, enable -reproducible: true and -noextraheaders: true for the Builder.

[rotty3000]

That's what I'm thinking. @kwin since you know more about this, could you confirm that project.build.outputTimestamp is really the best signature for us to look for?

[kwin]

No, you should rather introduce a dedicated parameter similar like what maven-jar-plugin (https://github.com/apache/maven-jar-plugin/blob/e036eb2a511c6771c172b23d9d33c5563f1d7ae2/src/main/java/org/apache/maven/plugins/jar/AbstractJarMojo.java#L151) does. I am not sure if relying on a fixed timestamp is better or whether a boolean flag would actually be enough (to completely leave out the non-fixed headers). Maybe a dedicated boolean Maven plugin parameter which by default is set to true if project.build.outputTimestamp is set to something sounds reasonable to me. WDYT?

[bjhargrave]

Since bnd-maven-plugin does not jar the bundle, there is no point in defining any new configuration for this. We would need to "look" into the configuration of maven-jar-plugin to see if it is has the outputTimeStamp property set to something which seems wrong.

The only thing that really needs to be done for bnd-maven-plugin is to not emit manifest headers which depend upon the build time. -noextraheaders: true should do this.

But it seems wrong to add a configuration property to bnd-maven-plugin which is the duplicate of the maven-jar-plugin outputTimeStamp property especially since bnd-maven-plugin does not even make the jar so it does not control the timestamps of the jar entries.

[bjhargrave]

Closing since there is no change to the plugin. Use Bnd config for -noextraheaders: true if you want reproducible builds with maven.

[rombert]

Testing with -noextraheaders: true shows that reproducible builds do not yet work completely. The single difference for us is the Bundle-Version header. Between two invocations I get a difference such as:

--- 1/META-INF/MANIFEST.MF	2019-10-02 08:04:00.000000000 +0200
+++ 2/META-INF/MANIFEST.MF	2019-10-02 08:04:00.000000000 +0200
@@ -28,7 +28,7 @@
  itbox.apache.org/repos/asf/sling-org-apache-sling-api.git",tag=HEAD
 Bundle-SymbolicName: org.apache.sling.api
 Bundle-Vendor: The Apache Software Foundation
-Bundle-Version: 2.22.1.20200109153228096
+Bundle-Version: 2.22.1.20200109153234609
 Export-Package: org.apache.sling.api;version="2.3.4";uses:="javax.servle
  t,javax.servlet.http,org.apache.sling.api.adapter,org.apache.sling.api.
  request,org.apache.sling.api.resource",org.apache.sling.api.adapter;ver

To reproduce, you can run the test.sh script from https://github.com/apache/sling-org-apache-sling-api/tree/feature/reproducible-builds ( but note that it needs a parent pom that is not yet release, so you may want to wait for a couple of days ). However, the cause of the error should be clear.

[bjhargrave]

This is caused by an odd historical behavior of bnd-maven-plugin. If Bundle-Version is not explicitly set and if the -snapshot instruction is not also set in the Bnd configuration, then the bnd-maven-plugin will set -snapshot: ${tstamp} into the Bnd configuration.

bnd/maven/bnd-maven-plugin/src/main/java/aQute/bnd/maven/plugin/AbstractBndMavenPlugin.java

Lines 305 to 307 in 5a777f9

	if (builder.getProperty(Constants.SNAPSHOT) == null) {
	builder.setProperty(Constants.SNAPSHOT, TSTAMP);
	}

This will cause the SNAPSHOT qualifier in Bundle-Version to be replaced with the timestamp value.

bnd/biz.aQute.bndlib/src/aQute/bnd/osgi/Builder.java

Lines 439 to 465 in 5a777f9

	private String doSnapshot(String version) {
	String snapshot = getProperty(SNAPSHOT);
	if (snapshot == null) {
	return version;
	}
	if (snapshot.isEmpty()) {
	snapshot = null;
	}
	Version v = Version.parseVersion(version);
	String q = v.getQualifier();
	if (q == null) {
	return version;
	}
	if (q.equals("SNAPSHOT")) {
	q = snapshot;
	} else if (q.endsWith("-SNAPSHOT")) {
	int end = q.length() - "SNAPSHOT".length();
	if (snapshot == null) {
	q = q.substring(0, end - 1);
	} else {
	q = q.substring(0, end) + snapshot;
	}
	} else {
	return version;
	}
	return new Version(v.getMajor(), v.getMinor(), v.getMicro(), q).toString();
	}

So you will also need to set -snapshot: SNAPSHOT in the Bnd configuration to prevent the above behavior.

[bjhargrave]

See https://github.com/bndtools/bnd/tree/master/maven/bnd-maven-plugin#reproducible-builds where I added a section to the bnd-maven-plugin readme.

[kwin]

@bjhargrave Thanks a lot for the clarification and for the documentation update.

[rombert]

That makes sense, thanks @bjhargrave

[kwin]

Just for the record: Since 32a6357 reproducible builds are automatically enabled once project.build.outputTimestamp is set