Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change gradle copy to sync to clear old jars #609

Conversation

mikaylathompson
Copy link
Collaborator

@mikaylathompson mikaylathompson commented Apr 24, 2024

Description

There is a step in our gradle build that copies jar files from the project (e.g. Replayer) build artifacts directory into a dockerSolution build directory, which is where the generated Dockerfile expects them to be to copy them into the image.

However, this dockerSolution build directory is only "emptied" on a gradle clean, which means that as versions of our dependencies are updated, that build directory is effectively append-only and continues collecting old versions of jars, which are then copied into our images.

This causes issues. In this specific case, conflicting versions of log4j2 were causing the replayer to crash on startup, when the image was being built in a non-clean environment.
We're also probably making our images larger than they need to be.

The same pattern was being used in TrafficCapture and RFS gradle build scripts, so both have been updated.

Issues Resolved

n/a

Testing

Local testing.
Along with functional testing, I did:

# after running with Copy
-> ls ~/code/opensearch-migrations/TrafficCapture/dockerSolution/build/docker/trafficReplayer/jars > copied_jars.txt
-> cat copied_jars.txt | grep 'log4j-'
log4j-api-2.21.1.jar
log4j-api-2.23.1.jar # note the conflicting versions here
log4j-core-2.21.1.jar
log4j-core-2.23.1.jar
log4j-slf4j2-impl-2.21.1.jar
log4j-slf4j2-impl-2.23.1.jar
# after running with Sync
-> ls ~/code/opensearch-migrations/TrafficCapture/dockerSolution/build/docker/trafficReplayer/jars > synced_jars.txt
# compare (output is truncated to the relevant changes in this case)
-> diff copied_jars.txt synced_jars.txt
74d59
< log4j-api-2.21.1.jar
76d60
< log4j-core-2.21.1.jar
78d61
< log4j-slf4j2-impl-2.21.1.jar
81d63

Check List

  • New functionality includes testing
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@mikaylathompson mikaylathompson merged commit d855554 into opensearch-project:main Apr 24, 2024
5 checks passed
@mikaylathompson mikaylathompson deleted the gradle-sync-instead-of-copy branch April 24, 2024 19:23
Copy link
Collaborator

@gregschohn gregschohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome. I had no idea that gradle could be smart like this! I don't recall ever seeing a project with sync, but it's so much better than copy. This should really reduce the need to do a clean!
Thanks!

@@ -27,7 +27,9 @@ class CommonUtils {
copyArtifact(project, projectName, projectName, destDir)
}
static def copyArtifact(Project project, String artifactProjectName, String destProjectName, String destDir) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it posible to rename this to syncArtifact? It would be great to modernize that too!

AndreKurait pushed a commit to AndreKurait/opensearch-migrations that referenced this pull request Apr 25, 2024
AndreKurait pushed a commit to AndreKurait/opensearch-migrations that referenced this pull request Apr 25, 2024
AndreKurait pushed a commit to AndreKurait/opensearch-migrations that referenced this pull request Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants