Write out the image metadata to a file after build. #2227

mbruggmann · 2020-01-11T22:05:38Z

The jib-maven-plugin will now create a json file with image
metadata during the build, like so:

$ mvn jib:build
...
$ cat target/jib-image.json | jq .
{
  "image": "gcr.io/project/image:tag",
  "imageId": "sha256:61bb3ec31a47cb730eb58a38bbfa813761a51dca69d10e39c24c3d00a7b2c7a9",
  "imageDigest": "sha256:3f1be7e19129edb202c071a659a4db35280ab2bb1a16f223bfd5d1948657b6fc",
  "tags": ["latest", "tag"]
}

This is handy if you generate the image name or tag in a dynamic
fashion (eg, including a timstamp) and you have other tools in a
CD pipeline that operate on the generated image. In our case, we
would like to use the reference to the generated image to deploy to
K8S, and our deployment tooling picks it up from a file.

The path to the output file is configurable in
outputPaths.imageJson.

Related issue: #1875

If the maven plugin is configured with an `imageName` output path: ```xml <configuration> <to> <image>gcr.io/some/path/to/image:tag</image> </to> <outputPaths> <imageName>target/jib-image.name</imageName> </outputPaths> </configuration> ``` The jib-maven-plugin will now create a file at that path with the image name during the build, like so: ``` $ mvn jib:build ... $ cat target/jib-image.name gcr.io/some/path/to/image:tag ``` This is handy if you generate the image name or tag in a dynamic fashion (eg, including a timstamp) and you have other tools in a CD pipeline that operate on the generated image. In our case, we would like to use the reference to the generated image to deploy to K8S, and our deployment tooling picks it up from a file. I have only implemented support in the maven plugin for now, since that is what we are using and I'm not really familiar with Gradle. That should be pretty straight-forward to add, if need be. When it comes to the implementation, I imitated how the image id and digests are written out for now. Changing to a json or yaml output format felt like a bigger (breaking) change. Related issue: GoogleContainerTools#1875

loosebazooka

I'm not sure this is something we want to do. I see the usecase, but this can be solved in plugins already - you can write a custom gradle task to do this, or use something like the Maven ant-run plug-in. It just feels strange to write out an input directly without modifying it.

mbruggmann · 2020-01-13T09:40:37Z

@loosebazooka Agree that there are workarounds using other maven/gradle plugins. Two things I'm wary about with that approach:

We need to force the plugins to run on the same lifecycle for it to make sense. Also, issuing mvn jib:build manually doesn't produce the image metadata file.
The image name is duplicated in the pom, once for the jib plugin and once for producing the file. This means that if a consumer of the parent pom overwrites the jib to.image, passes it in through the CLI, by setting a property etc the output won't match.

How about combining the repository, digests, tags etc into a single jib-image.json output file, as per linked issue? Would that make more sense? Otherwise I'm happy to close this and explore other solutions form our side of course.

loosebazooka · 2020-01-13T15:52:28Z

@mbruggmann these are good points. I think we're still working out how we want jib-image.json to work, but this might be a good time to do it.
We will probably leave the other files for backwards compatibility.

loosebazooka · 2020-01-14T18:00:30Z

@mbruggmann the changes look nice, I'll take a deeper look, can you also output the extra tags in the json file?

briandealwis

Looks good! Two little nits that I saw.

...gins-common/src/main/java/com/google/cloud/tools/jib/plugins/common/ImageMetadataOutput.java

briandealwis · 2020-01-14T18:18:20Z

...ugins-common/src/test/java/com/google/cloud/tools/jib/plugins/common/JibBuildRunnerTest.java

+        .thenReturn(mockJibContainer);
+    testJibBuildRunner.writeImageJson(outputPath).runBuild();
+
+    final String outputJson = new String(Files.readAllBytes(outputPath), StandardCharsets.UTF_8);


I think it would be better to compare this output explicitly as a string to ensure we're producing expected output. (For example, to detect a situation where we change JSON libraries that produces slightly-invalid JSON.)

Had a look at this. Beyond putting the full expected json into a string in the testcase (which seems highly specific) I didn't find a good way of doing that.

As of now, the next line of ImageMetadataOutput.fromJson(outputJson) will at least make sure it properly parses into the expected type so it should catch any invalid JSON, serialization/deserialization differences and similar.

mbruggmann · 2020-01-17T09:05:00Z

Added support for tags now if you'd like to take another look @loosebazooka @briandealwis

briandealwis

Looks good!

jib-core/src/test/java/com/google/cloud/tools/jib/api/JibContainerTest.java

...gins-common/src/main/java/com/google/cloud/tools/jib/plugins/common/ImageMetadataOutput.java

loosebazooka · 2020-01-17T17:40:09Z

Probably kind of annyoing, given our integration tests only run on internal machines (and you can't see the output). But can you run the integration tests yourself and fix the issues?
Instructions here: https://github.com/GoogleContainerTools/jib/blob/master/CONTRIBUTING.md#code-reviews

mbruggmann · 2020-01-18T20:13:20Z

The integration-tests caught an actual bug: the code was relying on ImageReference equality that wasn't implemented (added in the latests commit). Now it seems like Travis had what looks like an intermittent error though. Could someone with write access to the repo retrigger the Travis and Kokoro builds? Thanks

briandealwis

Thanks for your contribution!

googlebot added the cla: yes label Jan 11, 2020

mbruggmann added 2 commits January 11, 2020 23:16

Don't use Java11 from tests

0d77fbb

Refactoring: Keep the target image in JibContainer

f9ec62c

loosebazooka reviewed Jan 12, 2020

View reviewed changes

mbruggmann added 2 commits January 14, 2020 16:59

Write out image metadata as json instead

7fa3cd0

Allow copyright from 2020

5bf7aed

mbruggmann changed the title ~~Optionally write out the image name to a file after build.~~ Write out the image metadata to a file after build. Jan 14, 2020

briandealwis reviewed Jan 14, 2020

View reviewed changes

Also output tags in jib-image.json

caeb2fe

briandealwis reviewed Jan 17, 2020

View reviewed changes

jib-core/src/test/java/com/google/cloud/tools/jib/api/JibContainerTest.java Outdated Show resolved Hide resolved

...gins-common/src/main/java/com/google/cloud/tools/jib/plugins/common/ImageMetadataOutput.java Outdated Show resolved Hide resolved

loosebazooka added the kokoro:run label Jan 17, 2020

kokoro-team removed the kokoro:run label Jan 17, 2020

mbruggmann added 2 commits January 18, 2020 18:17

Use Guavas ImmutableList.sortedCopyOf()

8d64255

Implement equality on ImageReference

4fe9a47

loosebazooka added the kokoro:run label Jan 18, 2020

kokoro-team removed the kokoro:run label Jan 18, 2020

briandealwis approved these changes Jan 21, 2020

View reviewed changes

loosebazooka approved these changes Jan 21, 2020

View reviewed changes

briandealwis merged commit 9b194da into GoogleContainerTools:master Jan 21, 2020

loosebazooka mentioned this pull request Jan 21, 2020

Update changelog for json image metadata file #2239

Merged

mbruggmann deleted the image-name-output branch January 23, 2020 09:25

TadCordle mentioned this pull request Jan 29, 2020

Give access to image repository or full image references built or pushed #1875

Closed

chanseokoh mentioned this pull request Mar 16, 2021

Recover generated images in multi-module project #3143

Closed

mpeddada1 mentioned this pull request Apr 11, 2022

Skip image tag info is lost #3636

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Write out the image metadata to a file after build. #2227

Write out the image metadata to a file after build. #2227

mbruggmann commented Jan 11, 2020 •

edited

Loading

loosebazooka left a comment

mbruggmann commented Jan 13, 2020

loosebazooka commented Jan 13, 2020

loosebazooka commented Jan 14, 2020

briandealwis left a comment

briandealwis Jan 14, 2020

mbruggmann Jan 16, 2020

mbruggmann commented Jan 17, 2020

briandealwis left a comment

loosebazooka commented Jan 17, 2020

mbruggmann commented Jan 18, 2020

briandealwis left a comment

Write out the image metadata to a file after build. #2227

Write out the image metadata to a file after build. #2227

Conversation

mbruggmann commented Jan 11, 2020 • edited Loading

loosebazooka left a comment

Choose a reason for hiding this comment

mbruggmann commented Jan 13, 2020

loosebazooka commented Jan 13, 2020

loosebazooka commented Jan 14, 2020

briandealwis left a comment

Choose a reason for hiding this comment

briandealwis Jan 14, 2020

Choose a reason for hiding this comment

mbruggmann Jan 16, 2020

Choose a reason for hiding this comment

mbruggmann commented Jan 17, 2020

briandealwis left a comment

Choose a reason for hiding this comment

loosebazooka commented Jan 17, 2020

mbruggmann commented Jan 18, 2020

briandealwis left a comment

Choose a reason for hiding this comment

mbruggmann commented Jan 11, 2020 •

edited

Loading