Publish snapshots to maven #2438

peternied · 2023-02-14T20:50:34Z

Description

Adds a workflow to publish snapshots to the OpenSearch-Project maven repository when updates are pushed to main, 1.* branches, or 2.* branches.

Following the example from https://github.com/opensearch-project/opensearch-sdk-java

Issues Resolved

Related [Bug] Cannot get latest artifacts for CI opensearch-build#3199
Related [FEATURE] Consume latest security zip from snapshots instead of distribution build security-dashboards-plugin#1328

Testing

Manually tested by editing the snapshot repository by updating the Snapshots target repository from maven to a local file system path.

build.gradle

        maven {
            name = "Snapshots" //  optional target repository name
            url = "https://aws.oss.sonatype.org/content/repositories/snapshots"
            credentials {
                username "$System.env.SONATYPE_USERNAME"
                password "$System.env.SONATYPE_PASSWORD"
            }
        }

->

        maven {
            name = "Snapshots" //  optional target repository name
            url = "snapshots"
        }

Run the command ./gradlew publishPluginZipPublicationToSnapshotsRepository which will publish the files in the file system under the 'snapshots' folder, then scan with find or your favor directory scanner to see the zip + pom with sha files

% find snapshots | sort                                                                                                                                                                               ~/git/security
snapshots
snapshots/org
snapshots/org/opensearch
snapshots/org/opensearch/plugin
snapshots/org/opensearch/plugin/opensearch-security
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/maven-metadata.xml
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/maven-metadata.xml.md5
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/maven-metadata.xml.sha1
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/maven-metadata.xml.sha256
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/maven-metadata.xml.sha512
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.pom
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.pom.md5
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.pom.sha1
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.pom.sha256
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.pom.sha512
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.zip
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.zip.md5
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.zip.sha1
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.zip.sha256
snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/opensearch-security-3.0.0.0-20230214.224249-1.zip.sha512
snapshots/org/opensearch/plugin/opensearch-security/maven-metadata.xml
snapshots/org/opensearch/plugin/opensearch-security/maven-metadata.xml.md5
snapshots/org/opensearch/plugin/opensearch-security/maven-metadata.xml.sha1
snapshots/org/opensearch/plugin/opensearch-security/maven-metadata.xml.sha256
snapshots/org/opensearch/plugin/opensearch-security/maven-metadata.xml.sha512

Check List

New functionality includes testing
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Adds a workflow to publish snapshots to the OpenSearch-Project maven repository when updates are pushed to main, 1.* branches, or 2.* branches. Following the example from https://github.com/opensearch-project/opensearch-sdk-java Signed-off-by: Peter Nied <[email protected]>

peternied · 2023-02-14T20:51:48Z

@gaiksaya We need our repository setup so we can retrieve the credentials for access Sonotype, can you help me get that done?

Signed-off-by: Peter Nied <[email protected]>

codecov-commenter · 2023-02-14T21:23:06Z

Codecov Report

Merging #2438 (5b7e070) into main (bae002b) will decrease coverage by 0.08%.
The diff coverage is n/a.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #2438      +/-   ##
============================================
- Coverage     61.28%   61.21%   -0.08%     
+ Complexity     3330     3324       -6     
============================================
  Files           260      260              
  Lines         18487    18487              
  Branches       3267     3267              
============================================
- Hits          11330    11317      -13     
- Misses         5565     5575      +10     
- Partials       1592     1595       +3

Impacted Files	Coverage Δ
...ecurity/configuration/StaticResourceException.java	`0.00% <0.00%> (-25.00%)`	⬇️
...nsearch/security/dlic/rest/api/AuditApiAction.java	`64.58% <0.00%> (-4.17%)`	⬇️
...ecurity/configuration/ConfigurationRepository.java	`72.13% <0.00%> (-2.19%)`	⬇️
.../dlic/auth/ldap2/LDAPConnectionFactoryFactory.java	`57.46% <0.00%> (-1.50%)`	⬇️
...search/security/transport/SecurityInterceptor.java	`75.38% <0.00%> (-0.77%)`	⬇️
...a/org/opensearch/security/tools/SecurityAdmin.java	`36.60% <0.00%> (-0.24%)`	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

prudhvigodithi · 2023-02-14T21:57:18Z

Hey @peternied this PR should also consider the pluginZips to be published under org.opensearch.plugin group.
./gradlew assemble --no-daemon --refresh-dependencies -DskipTests=true and ./gradlew publishPluginZipPublicationToZipStagingRepository would create a folder build/local-staging-repo/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/ locally with maven zips of security plugin

-rw-r--r--  1 pgodithi  staff       796 Feb 14 13:36 maven-metadata.xml
-rw-r--r--  1 pgodithi  staff        40 Feb 14 13:36 maven-metadata.xml.sha1
-rw-r--r--  1 pgodithi  staff        32 Feb 14 13:36 maven-metadata.xml.md5
-rw-r--r--  1 pgodithi  staff        64 Feb 14 13:36 maven-metadata.xml.sha256
-rw-r--r--  1 pgodithi  staff       128 Feb 14 13:36 maven-metadata.xml.sha512
-rw-r--r--  1 pgodithi  staff  44192967 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.zip
-rw-r--r--  1 pgodithi  staff        40 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.zip.sha1
-rw-r--r--  1 pgodithi  staff        32 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.zip.md5
-rw-r--r--  1 pgodithi  staff        64 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.zip.sha256
-rw-r--r--  1 pgodithi  staff       128 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.zip.sha512
-rw-r--r--  1 pgodithi  staff      1037 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.pom
-rw-r--r--  1 pgodithi  staff        40 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.pom.sha1
-rw-r--r--  1 pgodithi  staff        32 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.pom.md5
-rw-r--r--  1 pgodithi  staff        64 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.pom.sha256
-rw-r--r--  1 pgodithi  staff       128 Feb 14 13:36 opensearch-security-3.0.0.0-20230214.213641-1.pom.sha512

we should also ship them to the maven so that the plugin zips are also published via this workflow.
@gaiksaya @bbarani
Thank you

prudhvigodithi

#2438 (comment)

peternied · 2023-02-14T22:08:58Z

@prudhvigodithi I've updated the description with that output. Are there other changes that should be included?

gaiksaya · 2023-02-14T22:18:40Z

So I cloned your fork @peternied and saw that the task differs repo to repo based on how publications are configured. For this repo seems like it is publishMavenPublicationToSnapshotsRepository
The maven artifacts need to include zip (along with associated poms and shas) and jars as well. The current task publishMavenPublicationToSnapshotsRepository I believe will only take care of jars (previous PR output that @peternied posted). We need to find a way to tell that task about the zip location as well. Maybe something like this https://stackoverflow.com/questions/41346425/publish-a-zip-file-to-nexus-maven-with-gradle

.github/workflows/maven-publish.yml

Signed-off-by: Peter Nied <[email protected]>

peternied · 2023-02-14T22:50:31Z

@prudhvigodithi @gaiksaya I've fix the publish task and wrote the test process into the description.

gaiksaya · 2023-02-14T23:10:30Z

@peternied ~~The jars are missing from that output. I believe we want all of these https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/~~

gaiksaya · 2023-02-15T00:39:12Z

Update: We do not publish jars for security plugin. In that case we can also remove https://github.com/opensearch-project/security/blob/main/build.gradle#L64-L67
Adding @prudhvigodithi for more context

prudhvigodithi · 2023-02-15T00:44:40Z

To add more context security plugin at preset is used only via zips (may be this could change in future), example https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch-security/3.0.0.0-SNAPSHOT/, the folder all contains zips. For zips publish we already have a reserved groupID org.opensearch.plugin https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/opensearch-security/3.0.0.0-SNAPSHOT/ which is taken care by the plugin opensearch.pluginzip.

So long story short for security plugin lets only focus on shipping the files part of org/opensearch/plugin folder to maven using GH workflows.
@peternied

gaiksaya

Will comment back once the role is added to the secrets!

gaiksaya · 2023-02-15T22:37:01Z

Will comment back once the role is added to the secrets!
@gaiksaya We need our repository setup so we can retrieve the credentials for access Sonotype, can you help me get that done?

The role has been added to the secrets. Thanks!

peternied · 2023-02-15T23:18:08Z

@prudhvigodithi I noticed that you have requested changes, but it's not entirely clear to me what changes you would like to see. Could you please provide more specific feedback on what needs to be modified or improved so that I can make the necessary adjustments to the code?

prudhvigodithi · 2023-02-15T23:54:31Z

Hey @peternied I have requested changes, before we considered the the task that picks up the files from folder /org/opensearch/plugin/, but not looks good now.
Thank you

DarshitChanpura · 2023-02-16T05:00:47Z

@peternied Please merge this if you aren't making any more changes.

* Publish snapshots to maven Adds a workflow to publish snapshots to the OpenSearch-Project maven repository when updates are pushed to main, 1.* branches, or 2.* branches. Following the example from https://github.com/opensearch-project/opensearch-sdk-java Signed-off-by: Peter Nied <[email protected]> (cherry picked from commit 5bdcb29)

gaiksaya · 2023-02-16T18:05:22Z

@peternied Give it a try? https://github.com/opensearch-project/security/actions/workflows/maven-publish.yml

peternied · 2023-02-16T18:31:52Z

@gaiksaya Nice, it works! I've got some cleanup - the multiplatform build is entirely unneeded.

https://github.com/opensearch-project/security/actions/runs/4196975415/jobs/7278687473

…ect#2441)

* Publish snapshots to maven Adds a workflow to publish snapshots to the OpenSearch-Project maven repository when updates are pushed to main, 1.* branches, or 2.* branches. Following the example from https://github.com/opensearch-project/opensearch-sdk-java Signed-off-by: Peter Nied <[email protected]> Signed-off-by: Maciej Mierzwa <[email protected]>

opensearch-trigger-bot · 2023-11-20T18:24:15Z

The backport to 1.3 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security/backport-1.3 1.3
# Navigate to the new working tree
pushd ../.worktrees/security/backport-1.3
# Create a new branch
git switch --create backport/backport-2438-to-1.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5bdcb29f02fe3a5e1bbd7367d173ee495ceebfa9
# Push it to GitHub
git push --set-upstream origin backport/backport-2438-to-1.3
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security/backport-1.3

Then, create a pull request where the base branch is 1.3 and the compare/head branch is backport/backport-2438-to-1.3.

opensearch-trigger-bot · 2023-11-20T18:24:21Z

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/security/backport-1.x 1.x
# Navigate to the new working tree
pushd ../.worktrees/security/backport-1.x
# Create a new branch
git switch --create backport/backport-2438-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5bdcb29f02fe3a5e1bbd7367d173ee495ceebfa9
# Push it to GitHub
git push --set-upstream origin backport/backport-2438-to-1.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/security/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-2438-to-1.x.

peternied requested a review from a team February 14, 2023 20:50

peternied mentioned this pull request Feb 14, 2023

[FEATURE] Consume latest security zip from snapshots instead of distribution build opensearch-project/security-dashboards-plugin#1328

Closed

Add trailing whitespace

7639640

Signed-off-by: Peter Nied <[email protected]>

DarshitChanpura previously approved these changes Feb 14, 2023

View reviewed changes

prudhvigodithi requested changes Feb 14, 2023

View reviewed changes

peternied commented Feb 14, 2023

View reviewed changes

.github/workflows/maven-publish.yml Outdated Show resolved Hide resolved

Use correct publishing task to upload the plugin zip

5b7e070

Signed-off-by: Peter Nied <[email protected]>

peternied dismissed DarshitChanpura’s stale review via 5b7e070 February 14, 2023 22:45

gaiksaya approved these changes Feb 15, 2023

View reviewed changes

peternied requested a review from prudhvigodithi February 15, 2023 23:14

prudhvigodithi approved these changes Feb 15, 2023

View reviewed changes

DarshitChanpura approved these changes Feb 16, 2023

View reviewed changes

peternied merged commit 5bdcb29 into opensearch-project:main Feb 16, 2023

peternied deleted the publish-to-maven branch February 16, 2023 14:54

peternied added the backport 2.x backport to 2.x branch label Feb 16, 2023

opensearch-trigger-bot bot mentioned this pull request Feb 16, 2023

[Backport 2.x] Publish snapshots to maven #2441

Merged

peternied pushed a commit that referenced this pull request Feb 16, 2023

Publish snapshots to maven (#2438) (#2441)

87990cd

gaiksaya mentioned this pull request Feb 21, 2023

[META] Move snapshot build jobs out of distribution build jobs opensearch-project/opensearch-build#3185

Closed

20 tasks

peternied mentioned this pull request Mar 9, 2023

[BUG] Migrate modern build.gradle into 1.x branches #2535

Closed

wuychn pushed a commit to ochprince/security that referenced this pull request Mar 16, 2023

Publish snapshots to maven (opensearch-project#2438) (opensearch-proj…

1fdf9e6

…ect#2441)

cwperks added backport 1.3 backport to 1.3 branch backport 1.x backport to 1.x branch labels Nov 20, 2023

opensearch-trigger-bot bot added the backport-failed label Nov 20, 2023

This was referenced Apr 23, 2024

[FEATURE] Publish snapshots to maven for 1.3 line #4281

Closed

Add maven publish workflow for 1.3 #4285

Merged

DarshitChanpura removed the backport-failed label Jul 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish snapshots to maven #2438

Publish snapshots to maven #2438

peternied commented Feb 14, 2023 •

edited

Loading

peternied commented Feb 14, 2023

codecov-commenter commented Feb 14, 2023 •

edited

Loading

prudhvigodithi commented Feb 14, 2023

prudhvigodithi left a comment

peternied commented Feb 14, 2023 •

edited

Loading

gaiksaya commented Feb 14, 2023

peternied commented Feb 14, 2023

gaiksaya commented Feb 14, 2023 •

edited

Loading

gaiksaya commented Feb 15, 2023

prudhvigodithi commented Feb 15, 2023

gaiksaya left a comment

gaiksaya commented Feb 15, 2023

peternied commented Feb 15, 2023

prudhvigodithi commented Feb 15, 2023

DarshitChanpura commented Feb 16, 2023

gaiksaya commented Feb 16, 2023

peternied commented Feb 16, 2023

opensearch-trigger-bot bot commented Nov 20, 2023

opensearch-trigger-bot bot commented Nov 20, 2023

Publish snapshots to maven #2438

Publish snapshots to maven #2438

Conversation

peternied commented Feb 14, 2023 • edited Loading

Description

Issues Resolved

Testing

build.gradle

Check List

peternied commented Feb 14, 2023

codecov-commenter commented Feb 14, 2023 • edited Loading

Codecov Report

prudhvigodithi commented Feb 14, 2023

prudhvigodithi left a comment

Choose a reason for hiding this comment

peternied commented Feb 14, 2023 • edited Loading

gaiksaya commented Feb 14, 2023

peternied commented Feb 14, 2023

gaiksaya commented Feb 14, 2023 • edited Loading

gaiksaya commented Feb 15, 2023

prudhvigodithi commented Feb 15, 2023

gaiksaya left a comment

Choose a reason for hiding this comment

gaiksaya commented Feb 15, 2023

peternied commented Feb 15, 2023

prudhvigodithi commented Feb 15, 2023

DarshitChanpura commented Feb 16, 2023

gaiksaya commented Feb 16, 2023

peternied commented Feb 16, 2023

opensearch-trigger-bot bot commented Nov 20, 2023

opensearch-trigger-bot bot commented Nov 20, 2023

peternied commented Feb 14, 2023 •

edited

Loading

codecov-commenter commented Feb 14, 2023 •

edited

Loading

peternied commented Feb 14, 2023 •

edited

Loading

gaiksaya commented Feb 14, 2023 •

edited

Loading