Detect assemblies with too many entries to fail shell script prepending

[lefou]

This is an attempt to fix the issue of invalid assembly files which occurs when the following two conditions are met:

• The JAR file has more than 65535 ZIP entries
• The JAR has a prepended shell script

The issue was reported and analyzed in the following tickets:

• JavaModule.assembly produces invalid or corrupt jar file (was: publishLocal as altenative to jars : which are corrupt often and at random) #528
• assembly error #2650

This issue also hits other build tools, but it seems Mill is the only tools which automatically enables shell script prepending by default.

Since there is no real fix available, we simply try to detect the issue after the fact and fail the assemble task with a actionable error message.

To make the fix binary compatible, I had to deprecated the upstreamAssembly target in favor to the new upstreamAssembly2 target, which returns also the added ZIP entry count. Since this can be a behavioral change when users have overridden the upstreamAssembly target, I also added some warning messages with will detect this at runtime and provide actionable help.

[lihaoyi]

I wonder if it's really worth keeping track of the number of entries incrementally? v.s. creating the assembly first, scanning it, and then failing if it has too many entries. If the performance of scanning the assembly is acceptable (milliseconds to tens of milliseconds?) then that would save us a bunch of book-keeping passing around addedEntries values, and a bunch of churn in replacing upstreamAssembly with upstreamAssembly2

[lefou]

It probably depends. When we scan the jar right after it's written, it should be in the file system cache of any reasonable OS and scanning should be fast. On the other side, just remembering the added count is quasi for free, esp. when we assume, that the upstream-assembly is the larger portion of the assembly and keeps stable.

I think I want to experiment what the result looks like and how it performs.

[lefou]

The issues with scanning afterwards:

• Scanning and counting entries is slow. Scanning the jar of test case noExe.large took almost 3 seconds (75516 entries), although we just wrote it
• Java's ZipInputStream and JarInputStream aren't able to find any ZipEntry in a prependend jar making scanning after packaging non-trivial
• Shelling out to some OS-installed zip tools doesn't seem right

[lihaoyi]

lgtm