Add functionality to create jar in zinc wrapper #6094

ity · 2018-07-11T07:39:03Z

Problem

Invocations of zinc do not support jars. See #6080 for details

Solution

Add functionality to the zinc wrapper to jar up the contents of _classesDirectory when an option is specified.

illicitonion

Looks great :) Didn't manage to plug this into the python code, but will hopefully try this out soon :)

illicitonion · 2018-07-11T08:52:56Z

src/scala/org/pantsbuild/zinc/compiler/InputUtils.scala

+
+      val jarPath = Paths.get(classesDirectory.toString, settings.outputJar.toString)
+      val target = new JarOutputStream(Files.newOutputStream(jarPath))
+      val entryTime = System.currentTimeMillis()


We should probably either have a flag to set the time to use (so that pants can set a fixed value, maybe falling back to the current time if no value is specified), or always set this to a fixed value, for determinism.

absolutely!

illicitonion · 2018-07-11T08:58:09Z

src/scala/org/pantsbuild/zinc/compiler/InputUtils.scala

+        FileVisitResult.CONTINUE
+      }
+
+      Files.walkFileTree(classesDirectory.toPath, FileVisitor)


IIRC walkFileTree makes no guarantees about which order it visits files; we may want to manually list directory contents, sort them, and visit in sorted order, for determinism

illicitonion · 2018-07-11T08:58:32Z

src/scala/org/pantsbuild/zinc/compiler/Settings.scala

@@ -57,12 +58,13 @@ case class Settings(

  lazy val sources: Seq[File] = _sources map normalise

+  if (_classesDirectory.isEmpty && outputJar.isEmpty) {
+    throw new RuntimeException(
+      s"Either ${Settings.DestinationOpt} or ${Settings.JarDestinationOpt} option is required.")


s/Either/At least one of/?

I think either is accurate here.

stuhood · 2018-07-11T17:21:10Z

Didn't manage to plug this into the python code, but will hopefully try this out soon

Unfortunately, doing that requires doing a local publish: there are instructions in the "dry-run" section here: https://www.pantsbuild.org/release_jvm.html#dry-run

stuhood · 2018-07-11T21:43:54Z

Didn't manage to plug this into the python code, but will hopefully try this out soon

Unfortunately, doing that requires doing a local publish: there are instructions in the "dry-run" section here: https://www.pantsbuild.org/release_jvm.html#dry-run

stuhood · 2018-07-11T17:23:17Z

src/scala/org/pantsbuild/zinc/compiler/InputUtils.scala

+  /**
+   * Jar the contents of output classes (settings.classesDirectory) and copy to settings.outputJar
+   */
+  def createClassesJar(settings: Settings, log: Logger) = {


It might be a good idea to put this in a separate object somewhere (could preserve our silly naming convention here and call it OutputUtil or something, heh).

stuhood · 2018-07-11T17:24:49Z

src/scala/org/pantsbuild/zinc/compiler/InputUtils.scala

+        jarPath
+      }
+
+      val jarPath = Paths.get(classesDirectory.toString, settings.outputJar.toString)


...I really can't believe that this is the recommended way to join paths in Java. But it is!

I've found the ammonite ops library very neat for hygienic path operations (it has implicits that let you do e.g. Path(classesDirectory) / "intermediate-folder-i-just-made-up-for-this-example" / RelPath(settings.outputJar)). Its process creation mechanism isn't as useful as scala.sys.process for general use though, imho (although quite concise for synchronous process execution for testing). Not sure if I'm missing any reason that can't be used here.

stuhood · 2018-07-11T17:26:56Z

src/scala/org/pantsbuild/zinc/compiler/InputUtils.scala

+   */
+  def createClassesJar(settings: Settings, log: Logger) = {
+    val classesDirectory = settings.classesDirectory
+    object FileVisitor extends SimpleFileVisitor[Path]() {


Since this has mutable state inside, I'd recommend doing something like:

val jarCaptureVisitier = new SimpleFileVisitor[Path]() { ... }

stuhood · 2018-07-11T21:41:28Z

src/scala/org/pantsbuild/zinc/compiler/InputUtils.scala

+        FileVisitResult.CONTINUE
+      }
+
+      Files.walkFileTree(classesDirectory.toPath, FileVisitor)


stuhood · 2018-07-11T21:42:55Z

src/scala/org/pantsbuild/zinc/compiler/Settings.scala

@@ -57,12 +58,13 @@ case class Settings(

  lazy val sources: Seq[File] = _sources map normalise

+  if (_classesDirectory.isEmpty && outputJar.isEmpty) {
+    throw new RuntimeException(
+      s"Either ${Settings.DestinationOpt} or ${Settings.JarDestinationOpt} option is required.")


I think either is accurate here.

ity · 2018-07-13T09:48:41Z

Didn't manage to plug this into the python code, but will hopefully try this out soon

Unfortunately, doing that requires doing a local publish: there are instructions in the "dry-run" section here: https://www.pantsbuild.org/release_jvm.html#dry-run

Just saw this and realized why my test locally wasnt working. Since we need to publish, is there any way to write an integration test for this?

illicitonion

Looks great!

I definitely defer to @stuhood for how to write tests here :)

illicitonion · 2018-07-13T10:02:05Z

src/scala/org/pantsbuild/zinc/compiler/Main.scala

@@ -121,6 +121,11 @@ object Main {
      }

      log.info("Compile success " + Util.timing(startTime))
+
+      // TODO(ity): if compile successful, jar the contents of classesDirectory and copy to outputJar


stuhood

Thanks Ity. Please add a quick unit test and then this should be good.

stuhood · 2018-07-13T17:53:39Z

src/scala/org/pantsbuild/zinc/compiler/OutputUtils.scala

+      FileVisitResult.CONTINUE
+    }
+    sorted.map(createJar(_))
+  }


This does not explicitly call close on the JarOutputStream, but should.

stuhood · 2018-07-13T17:55:05Z

src/scala/org/pantsbuild/zinc/compiler/OutputUtils.scala

+    // deterministic jar creation
+    Files.walkFileTree(classesDirectory.toPath, fileSortVisitor)
+
+    val jarPath = Paths.get(classesDirectory.toString, settings.outputJar.toString)


It looks like this will make this a relative path under the classesDirectory?

Both of these should probably be (assumed to be) absolute, so that they don't need to be located within one another.

^ as mentioned below, I think that this is still a thing... the test is currently assuming this behaviour.

stuhood · 2018-07-13T17:57:12Z

src/scala/org/pantsbuild/zinc/compiler/OutputUtils.scala

+  /**
+   * Jar the contents of output classes (settings.classesDirectory) and copy to settings.outputJar
+   */
+  def createClassesJar(settings: Settings, log: Logger) = {


I'd recommend creating one test that sanity checks this method: to make that easier, I'd recommend replacing Settings with explicit arguments for the classesDirectory and outputJar (and then maybe make the timestamp optional).

An example unit test for this code is over here: https://github.com/pantsbuild/pants/tree/master/tests/scala/org/pantsbuild/zinc/analysis

stuhood · 2018-07-13T17:58:57Z

src/scala/org/pantsbuild/zinc/compiler/Settings.scala

@@ -47,7 +48,8 @@ case class Settings(
  compileOrder: CompileOrder        = CompileOrder.Mixed,
  sbt: SbtJars                      = SbtJars(),
  _incOptions: IncOptions           = IncOptions(),
-  analysis: AnalysisOptions         = AnalysisOptions()
+  analysis: AnalysisOptions         = AnalysisOptions(),
+  creationTime: Long                = System.currentTimeMillis()


This isn't a good default for this arg: would recommend 0 (the unix epoch) or something...

I think if we want the jars to be deterministic (each JAR entry to have the same creation time), the default needs to be consistent for each JAR created but not the same as every other JAR created. lmk if I misunderstand

but not the same as every other JAR created

Ideally "when" you create the jar does not matter at all. For the same inputs, creating the same jar twice (one+ second apart) should result in the exact same jar. Hence 0 here.

stuhood · 2018-07-13T19:10:09Z

Since we need to publish, is there any way to write an integration test for this?

Only post merge, unfortunately.

cosmicexplorer · 2018-07-14T06:07:39Z

Only post merge, unfortunately.

Is this a "can't be done practically for reasons" or "someone needs to make this functionality"? It seems like publishing a jar is something that would be useful to have the ability to test (I'm thinking of a mixin or something which could then be extended for e.g. internal tasks which need to test publishing a jar for some reason). Is there context I'm missing?

baroquebobcat

I've got one question below.

I'd also like to see a test that asserts things about the resulting jar contents. I'd be ok with deferring adding automated testing to the python side, but I have a strong preference for having a scala test if possible.

baroquebobcat · 2018-07-17T20:50:45Z

src/scala/org/pantsbuild/zinc/compiler/OutputUtils.scala

+
+    val fileSortVisitor = new SimpleFileVisitor[Path]() {
+      override def preVisitDirectory(path: Path, attrs: BasicFileAttributes): FileVisitResult = {
+        sorted.add(path)


Are these paths relative to the classDirectory? If they're absolute, they might create the wrong entries in the jar file because they're used as the path for the JarEntry

stuhood

Thanks @ity !

I think that there is one remaining issue around relative paths. Should update the test to use two separate temp dirs before merging.

stuhood · 2018-07-17T22:17:50Z

src/scala/org/pantsbuild/zinc/compiler/OutputUtils.scala

+   * @return
+   */
+  def existsClass(jarPath: Path, clazz: String): Boolean = {
+    val jis = new JarInputStream(Files.newInputStream(jarPath))


This stream needs to be closed (probably via try { .. } finally { .. }).

stuhood · 2018-07-17T22:20:52Z

tests/scala/org/pantsbuild/zinc/compiler/JarCreationSpec.scala

+  }
+  "JarCreationWithClasses" should {
+    "succeed when input classes are provided" in {
+      IO.withTemporaryDirectory { tempDir =>


I think that the code and this test assume that the jar is always created under the classes directory... but that won't be the case.

Can you change this test to use two temporary directories: one containing the input classes, and another containing the output jar?

I got rid of this assumption in the previous revision - added 2 separate dirs, PTAL when you can

baroquebobcat · 2018-07-17T23:06:16Z

tests/scala/org/pantsbuild/zinc/compiler/JarCreationSpec.scala

+        val jarOutputPath = Paths.get(tempDirPath.toString, "spec-valid-output.jar")
+
+        OutputUtils.createJar(filePaths, jarOutputPath, System.currentTimeMillis())
+        OutputUtils.existsClass(jarOutputPath, tempFile.toString) must be(true)


Thanks for writing the test. Things are a bit clearer to me now.

I think createJar should take a root for the input file paths, that way the entry paths can be relative to that root.

for example, if you had an input dir like temp-dir that looked like this.

temp-dir/org/example/Clazz.class temp-dir/org/example/inner/Clazz2.class

I think you'd want the resulting jar to contain

org/example/Clazz.class org/example/inner/Clazz2.class

I'm not sure it does this right now.

Does that make sense? I might be missing something.

stuhood · 2018-07-17T23:52:42Z

tests/scala/org/pantsbuild/zinc/compiler/JarCreationSpec.scala

+          val jarOutputPath = Paths.get(tempOutputDir.toString, "spec-valid-output.jar")
+
+          OutputUtils.createJar(filePaths, jarOutputPath, System.currentTimeMillis())
+          OutputUtils.existsClass(jarOutputPath, tempFile.toString) must be(true)


So, tempFile.toString is going to be an absolute filename, I think? In this case we're expecting literally Clazz.class.

baroquebobcat

Thanks for updating the test and clearing up the relative path thing. It looks good to me now, apart from a copyright year that might need updating and possibly a method rename.

baroquebobcat · 2018-07-18T16:53:02Z

tests/scala/org/pantsbuild/zinc/compiler/BUILD

@@ -0,0 +1,12 @@
+# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).


baroquebobcat · 2018-07-18T16:53:27Z

tests/scala/org/pantsbuild/zinc/compiler/JarCreationSpec.scala

+
+          OutputUtils.createJar(tempInputDir.toString, filePaths, jarOutputPath, System.currentTimeMillis())
+          OutputUtils.existsClass(jarOutputPath, tempFile.toString) must be(false)
+          OutputUtils.existsClass(jarOutputPath, tempFile.getName) must be(true)


Thanks 👍

getName is only the final component of a filename, so this test would fail if the classfile were more deeply nested... it looks like the underlying code properly relativizes things though.

baroquebobcat · 2018-07-18T17:01:14Z

src/scala/org/pantsbuild/zinc/compiler/OutputUtils.scala

+   * Determines if a Class exists in a JAR provided.
+   *
+   * @param jarPath Absolute Path to the JAR being inspected
+   * @param clazz Name of the Class, the existence of which is to be inspected


I think this function checks the existence of files in the jar and not the existence of classes by name. It might make sense to rename it and the clazz parameter to make that clearer.

stuhood · 2018-07-18T17:46:32Z

src/scala/org/pantsbuild/zinc/compiler/OutputUtils.scala

+      jarEntry.setTime(jarEntryTime)
+
+      target.putNextEntry(jarEntry)
+      Files.copy(source, target)


Hm... it looks like you're properly collecting directories above (ie, collecting directories in pre-visit), but I feel like this method will fail if the source: Path is a directory? It's important to actually store directory entries in the zip file, so I think that this case needs to add the jarEntry for a directory as a directory, and then skip appending the content.

You might consider having the FileVisitor store a tuple of (path: Path, isFile: Boolean) (which can still be sorted) to avoid needing to check whether the path is a directory here.

Between this and my comment in the test, I think the test needs an update to create a classfile under a directory.

thanks for catching this - ptal when you can :)

ity · 2018-07-18T18:42:00Z

Thanks for updating the test and clearing up the relative path thing. It looks good to me now, apart from a copyright year that might need updating and possibly a method rename.

👍

stuhood

I led you astray on one thing, sorry! Once that's fixed, please try out integrating with zinc, and then feel free to merge!

stuhood · 2018-07-18T19:06:31Z

src/scala/org/pantsbuild/zinc/compiler/OutputUtils.scala

+
+    val fileSortVisitor = new SimpleFileVisitor[Path]() {
+      override def preVisitDirectory(path: Path, attrs: BasicFileAttributes): FileVisitResult = {
+        sorted.add(path, false)


Eek. Sorry. Just looked at the javadocs for this: https://docs.oracle.com/javase/7/docs/api/java/util/zip/ZipEntry.html#isDirectory()

So the boolean isn't really necessary as long as you append a / here, and then later check whether it ends with a slash. Sorry!

(...and the slash is required in order to not confuse consumers of the jar.)

ity · 2018-07-19T23:29:14Z

I led you astray on one thing, sorry! Once that's fixed, please try out integrating with zinc, and then feel free to merge!

I should have read that myself really, thanks - been trying to test this locally but hitting some local caching issue with the zinc changes. will push once I have local integration tested.

### Problem Invocations of zinc do not support jars. ### Solution Add functionality to the zinc wrapper to jar up the contents of _classesDirectory when an option is specified.

ity requested review from baroquebobcat, stuhood and illicitonion July 11, 2018 07:39

illicitonion reviewed Jul 11, 2018

View reviewed changes

stuhood reviewed Jul 11, 2018

View reviewed changes

illicitonion approved these changes Jul 13, 2018

View reviewed changes

ity changed the title ~~WIP - Add functionality to create jar in zinc wrapper~~ Add functionality to create jar in zinc wrapper Jul 13, 2018

stuhood reviewed Jul 13, 2018

View reviewed changes

baroquebobcat reviewed Jul 17, 2018

View reviewed changes

stuhood approved these changes Jul 17, 2018

View reviewed changes

baroquebobcat reviewed Jul 17, 2018

View reviewed changes

ity force-pushed the ity/6080 branch from a50fa15 to ff5188f Compare July 17, 2018 23:48

stuhood reviewed Jul 17, 2018

View reviewed changes

ity force-pushed the ity/6080 branch from ff5188f to 5cedd46 Compare July 18, 2018 09:08

baroquebobcat approved these changes Jul 18, 2018

View reviewed changes

stuhood reviewed Jul 18, 2018

View reviewed changes

ity added 9 commits July 18, 2018 11:39

add outputJar option

0bbfb99

add createClassesJar functionality

c44fdff

add a flag for jar creation time

7ed6e08

Added OutputUtils

2cb21f3

Add sorting of classesDirectory for jar creation

06cb12d

fix comments

38a3655

Add unit tests for jar creation

94ba8e9

Addressed comments

3bba4ca

Relativize jar paths

1b8d8b8

Add test for nested dir and classes

f7a8e6f

ity force-pushed the ity/6080 branch from 5cedd46 to f7a8e6f Compare July 18, 2018 18:40

stuhood approved these changes Jul 18, 2018

View reviewed changes

Addressed final comments

98afdd7

ity force-pushed the ity/6080 branch from df624cb to 98afdd7 Compare July 24, 2018 08:49

ity merged commit b95a3a7 into pantsbuild:master Jul 24, 2018

stuhood mentioned this pull request Jul 26, 2018

Create context jars for javac invocations remotely when using the hermetic exec strategy #6045

Closed

illicitonion mentioned this pull request May 3, 2019

Add new --no-use-system-time flag to use a deterministic timestamp in built PEX pex-tool/pex#722

Merged

		@@ -0,0 +1,12 @@
		# Copyright 2017 Pants project contributors (see CONTRIBUTORS.md).

Add functionality to create jar in zinc wrapper #6094

Add functionality to create jar in zinc wrapper #6094

Conversation

ity commented Jul 11, 2018 • edited Loading

Problem

Solution

illicitonion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood commented Jul 11, 2018

stuhood commented Jul 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ity commented Jul 13, 2018

illicitonion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood commented Jul 13, 2018

cosmicexplorer commented Jul 14, 2018

baroquebobcat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood Jul 17, 2018 • edited Loading

Choose a reason for hiding this comment

baroquebobcat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ity commented Jul 18, 2018

stuhood left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ity commented Jul 19, 2018

ity commented Jul 11, 2018 •

edited

Loading

stuhood Jul 17, 2018 •

edited

Loading

stuhood left a comment •

edited

Loading