Clean-up step of migrating Jenkins shared libraries to opensearch-build-libraries repo #2679

jordarlu · 2022-09-30T18:22:47Z

Description

This is one of the clean-up steps of moving Jenkins shared lib from opensearch-build to opensearch-build-libraries
#2646

Issues Resolved

In this PR:

Remove/Clean vars + adding lib session on a few of jobs that did not have it previously to prevent the 'lib@20211123 not found' issue from happening again.
Remove/Clean couple of test jobs that have been migrated to opensearch-build-libraries repo, here is the list:
(the following jobs had been moved to opensearch-build-libraries repo previously )
TestArchiveAssembleUpload.groovy
TestAssembleManifest.groovy
TestAssembleUpload.groovy
TestBuildArchive.groovy
TestBuildAssembleUpload.groovy
TestBuildDockerImage.groovy
TestBuildFailureMessage.groovy
TestBuildInfoYaml.groovy
TestBuildManifest.groovy
TestBuildUploadManifestSHA.groovy
TestBuildYumRepo.groovy
TestCopyContainer.groovy
TestCreateGithubIssue.groovy
TestCreateReleaseTag.groovy
TestCreateTestResultsMessage.groovy
TestDetectDockerImage.groovy
TestDetectTestDockerAgent.groovy
TestDownloadFromS3.groovy
TestGetManifestSHA.groovy
TestInputManifest.groovy
TestMessages.groovy
TestParallelMessages.groovy
TestPostCleanup.groovy
TestPrintArtifactDownloadUrlsForStaging.groovy
TestPromoteArtifacts.groovy
TestPromoteYumRepos.groovy
TestPublishNotification.groovy
TestRpmDashboardsDistValidation.groovy
TestRpmMetaValidation.groovy
TestRpmOpenSearchDistValidation.groovy
TestRunBwcTestScript.groovy
TestRunGradleCheck.groovy
TestRunIntegTestScript.groovy
TestRunPerfTestScript.groovy
TestScanDockerImage.groovy
TestSignArtifacts.groovy
TestUploadIndexFile.groovy
TestUploadMinSnapshotsToS3.groovy
TestUploadTestResults.groovy
TestUploadToS3.groovy
TestbuildManifestVar.groovy

(the following jobs are staying at opensearch-build repo)
TestCCRPerfTest.groovy
TestDataPrepperReleaseArtifacts.groovy
TestMavenSignReleaseJob.groovy
TestOpenSearchBwcTest.groovy
TestOpenSearchDashboardsBwcTest.groovy
TestOpenSearchDashboardsIntegTest.groovy
TestOpenSearchIntegTest.groovy
TestPromoteContainer.groovy
TestRunNonSecurityPerfTestScript.groovy
TestSignStandaloneArtifactsJob.groovy
TestReleaseTagJob.groovy ( this job is temporary removed and will be restored at later time)
Remove/Clean lib-testers + Merging some codes from lib-testers onto the remaining test jobs listed above
Remove/Clean src/jenkins + code refactoring
Change build.gradle from multithreads to single one in order to prevent possible future problems
Remove/Clean tests/jenkins/jobs as it had been migrated to opensearch-build-libraries repo previousely

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Jeff Lu <[email protected]>

Signed-off-by: prudhvigodithi <[email protected]>

Signed-off-by: Jeff Lu <[email protected]>

codecov-commenter · 2022-09-30T18:25:47Z

Codecov Report

Merging #2679 (62879e7) into main (9e80f02) will decrease coverage by 0.14%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2679      +/-   ##
==========================================
- Coverage   94.24%   94.10%   -0.15%     
==========================================
  Files         162      156       -6     
  Lines        4313     4208     -105     
  Branches       19        0      -19     
==========================================
- Hits         4065     3960     -105     
  Misses        248      248

Impacted Files	Coverage Δ
deployment/lambdas/cf-url-rewriter/https-get.ts
...loyment/lambdas/cf-url-rewriter/cf-url-rewriter.ts
deployment/lib/buckets.ts
deployment/lib/build-artifact-stack.ts
deployment/lib/identities.ts
deployment/lib/artifacts-public-access.ts

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

prudhvigodithi · 2022-09-30T18:51:44Z

Hey @jordarlu you can clean up some Jenkinsfiles in tests/jenkins/jobs/ folder, example AssembleUpload_Jenkinsfile, these Jenkinsfiles used to test methods.

jordarlu · 2022-09-30T19:08:21Z

Thanks for the comment @prudhvigodithi ,, I am hesitating on which of those can be removed/shall be staying ... would it be possible letting me know? and I will run test at my local and then update in PR. Thanks!!

prudhvigodithi · 2022-10-01T00:00:01Z

Hey @jordarlu yes the files that test the libraries example AssembleUpload_Jenkinsfile in tests/jenkins/jobs/ is not required, as it has been tested in opensearch-build-libraries repo.
file link https://github.com/opensearch-project/opensearch-build-libraries/blob/main/tests/jenkins/jobs/ArchiveAssembleUpload_Jenkinsfile

Signed-off-by: Jeff Lu <[email protected]>

jordarlu · 2022-10-01T00:06:34Z

Thank you, @prudhvigodithi , let me remove that file as well

prudhvigodithi · 2022-10-01T00:07:12Z

Thank you, @prudhvigodithi , let me remove that file as well

I mean it has many like them :)
But i'm ok to merge this to unblock the groovy tests.
@peterzhuamazon @gaiksaya
Thank you

jordarlu · 2022-10-01T00:08:46Z

right, right .. we can clean more few others at alter ... trying to get the Jacoco to be happy again sooner .. :)

gaiksaya · 2022-10-01T00:36:47Z

Many of the files under /jobs needs to be removed as well. @jordarlu Can you compare and just delete them from here too? Should be straight forward. Thanks!

jordarlu · 2022-10-01T04:41:07Z

Thanks for the guidance, Sayali.. I compared /jobs/ between both repo, the build and build-lib, but found they are almost identical.
The build-lib repo has 2 files ( DockerCopy_Jenkinsfile and PerfTest_JenkinsFile) more than the build repo..
Does that mean we want to delete the entire /jobs/ at build repo?

Build Repo	Build-lib Repo
ArchiveAssembleUpload_Jenkinsfile	ArchiveAssembleUpload_Jenkinsfile
AssembleManifest_rpm_Jenkinsfile	AssembleManifest_rpm_Jenkinsfile
AssembleManifest_tar_Jenkinsfile	AssembleManifest_tar_Jenkinsfile
AssembleUpload_Jenkinsfile	AssembleUpload_Jenkinsfile
Build_OpenSearch_Dashboards_Jenkinsfile	Build_OpenSearch_Dashboards_Jenkinsfile
BuildArchive_Jenkinsfile	BuildArchive_Jenkinsfile
BuildAssembleUpload_Jenkinsfile	BuildAssembleUpload_Jenkinsfile
BuildDockerImage_Jenkinsfile	BuildDockerImage_Jenkinsfile
BuildDockerImage_Qualifier_Jenkinsfile	BuildDockerImage_Qualifier_Jenkinsfile
BuildFailureMessage_Jenkinsfile	BuildFailureMessage_Jenkinsfile
BuildInfoYaml_Jenkinsfile	BuildInfoYaml_Jenkinsfile
BuildManifest_Jenkinsfile	BuildManifest_Jenkinsfile
BuildManifest_Jenkinsfile_component_no_artifact	BuildManifest_Jenkinsfile_component_no_artifact
BuildShManifest_Jenkinsfile	BuildShManifest_Jenkinsfile
BuildUploadManifestSHA_Jenkinsfile	BuildUploadManifestSHA_Jenkinsfile
BuildYumRepo_Jenkinsfile	BuildYumRepo_Jenkinsfile
CreateGithubIssue_Jenkinsfile	CreateGithubIssue_Jenkinsfile
CreateGithubIssueExisting_Jenkinsfile	CreateGithubIssueExisting_Jenkinsfile
CreateReleaseTag_Jenkinsfile	CreateReleaseTag_Jenkinsfile
CreateTestResultsMessage_Jenkinsfile	CreateTestResultsMessage_Jenkinsfile
DetectDockerImage_Jenkinsfile	DetectDockerImage_Jenkinsfile
DetectTestDockerAgent_Jenkinsfile	DetectTestDockerAgent_Jenkinsfile
	DockerCopy_Jenkinsfile
DownloadFromS3_Jenkinsfile	DownloadFromS3_Jenkinsfile
GetManifestSHA_Jenkinsfile	GetManifestSHA_Jenkinsfile
Hello_Jenkinsfile	Hello_Jenkinsfile
InputManifest_Jenkinsfile	InputManifest_Jenkinsfile
Messages_Jenkinsfile	Messages_Jenkinsfile
ParallelMessages_Jenkinsfile	ParallelMessages_Jenkinsfile
	PerfTest_JenkinsFile
PostCleanup_Jenkinsfile	PostCleanup_Jenkinsfile
PrintArtifactDownloadUrlsForStaging_Jenkinsfile	PrintArtifactDownloadUrlsForStaging_Jenkinsfile
PromoteArtifacts_actions_Jenkinsfile	PromoteArtifacts_actions_Jenkinsfile
PromoteArtifacts_actions_OpenSearch_Dashboards_Jenkinsfile	PromoteArtifacts_actions_OpenSearch_Dashboards_Jenkinsfile
PromoteArtifacts_Jenkinsfile	PromoteArtifacts_Jenkinsfile
PromoteArtifacts_OpenSearch_Dashboards_Jenkinsfile	PromoteArtifacts_OpenSearch_Dashboards_Jenkinsfile
PromoteArtifactsQualifier_actions_Jenkinsfile	PromoteArtifactsQualifier_actions_Jenkinsfile
PromoteArtifactsQualifier_actions_OpenSearch_Dashboards_Jenkinsfile	PromoteArtifactsQualifier_actions_OpenSearch_Dashboards_Jenkinsfile
PromoteArtifactsQualifier_Jenkinsfile	PromoteArtifactsQualifier_Jenkinsfile
PromoteArtifactsQualifier_OpenSearch_Dashboards_Jenkinsfile	PromoteArtifactsQualifier_OpenSearch_Dashboards_Jenkinsfile
PromoteYumRepos_Jenkinsfile	PromoteYumRepos_Jenkinsfile
PublishNotification_Jenkinsfile	PublishNotification_Jenkinsfile
RpmDashboardsDistValidation_Jenkinsfile	RpmDashboardsDistValidation_Jenkinsfile
RpmMetaValidation_Jenkinsfile	RpmMetaValidation_Jenkinsfile
RpmOpenSearchDistValidation_Jenkinsfile	RpmOpenSearchDistValidation_Jenkinsfile
RunBwcTestScript_Jenkinsfile	RunBwcTestScript_Jenkinsfile
RunBwcTestScript_OpenSearch_Dashboards_Jenkinsfile	RunBwcTestScript_OpenSearch_Dashboards_Jenkinsfile
RunGradleCheck_Jenkinsfile	RunGradleCheck_Jenkinsfile
RunIntegTestScript_Jenkinsfile	RunIntegTestScript_Jenkinsfile
RunIntegTestScript_LocalPath_Jenkinsfile	RunIntegTestScript_LocalPath_Jenkinsfile
RunIntegTestScript_OpenSearch_Dashboards_Jenkinsfile	RunIntegTestScript_OpenSearch_Dashboards_Jenkinsfile
ScanDockerImage_Jenkinsfile	ScanDockerImage_Jenkinsfile
SignArtifacts_Jenkinsfile	SignArtifacts_Jenkinsfile
UploadIndexFile_Jenkinsfile	UploadIndexFile_Jenkinsfile
uploadMinSnapshotsToS3_Jenkinsfile	uploadMinSnapshotsToS3_Jenkinsfile
UploadTestResults_Jenkinsfile	UploadTestResults_Jenkinsfile
UploadToS3_Jenkinsfile	UploadToS3_Jenkinsfile

gaiksaya · 2022-10-03T16:23:37Z

Yes I believe so. The /jobs contained all the jenkinsfile to call the libraries to be tested. Since the build-lib-repo has those I believe we can delete from here. Thanks!

jordarlu · 2022-10-03T16:27:11Z

Yes I believe so. The /jobs contained all the jenkinsfile to call the libraries to be tested. Since the build-lib-repo has those I believe we can delete from here. Thanks

Thanks again, @gaiksaya , I will remove /jobs as well..

Signed-off-by: Jeff Lu <[email protected]>

prudhvigodithi · 2022-10-03T17:30:57Z

LGTM, @peterzhuamazon @gaiksaya @zelinh can you please take a look?

…ld-libraries repo (opensearch-project#2679) * update lib clean-up p1 Signed-off-by: Jeff Lu <[email protected]> * clean-up p2 Signed-off-by: Jeff Lu <[email protected]> * clean-up p3 Signed-off-by: Jeff Lu <[email protected]> * update parameters on TestRunNonSecurityPerfTestScript.groovy Signed-off-by: Jeff Lu <[email protected]> * Fix tests Signed-off-by: prudhvigodithi <[email protected]> * Fix tests Signed-off-by: prudhvigodithi <[email protected]> * update fix on TestReleaseTagJob Signed-off-by: Jeff Lu <[email protected]> * updating TestReleaseTagJob one more time Signed-off-by: Jeff Lu <[email protected]> * remove TestReleaseTagJob for now to unblock Jacoco error Signed-off-by: Jeff Lu <[email protected]> * remove AssembleUpload_Jenkinsfile Signed-off-by: Jeff Lu <[email protected]> * remove /jobs/ as it has been migrated to build-lib repo Signed-off-by: Jeff Lu <[email protected]> Signed-off-by: Jeff Lu <[email protected]> Signed-off-by: prudhvigodithi <[email protected]> Co-authored-by: prudhvigodithi <[email protected]> Signed-off-by: Monu Singh <[email protected]>

jordarlu and others added 9 commits September 29, 2022 17:16

update lib clean-up p1

65c568b

Signed-off-by: Jeff Lu <[email protected]>

clean-up p2

5ce0835

Signed-off-by: Jeff Lu <[email protected]>

clean-up p3

0bd0004

Signed-off-by: Jeff Lu <[email protected]>

update parameters on TestRunNonSecurityPerfTestScript.groovy

c3d4cf7

Signed-off-by: Jeff Lu <[email protected]>

Fix tests

c91af90

Signed-off-by: prudhvigodithi <[email protected]>

Fix tests

2c9662c

Signed-off-by: prudhvigodithi <[email protected]>

update fix on TestReleaseTagJob

7f92aa4

Signed-off-by: Jeff Lu <[email protected]>

updating TestReleaseTagJob one more time

3ac74a5

Signed-off-by: Jeff Lu <[email protected]>

remove TestReleaseTagJob for now to unblock Jacoco error

fcd1929

Signed-off-by: Jeff Lu <[email protected]>

jordarlu changed the title ~~Clean new lib~~ Clean-up step of relocating Jenkins shared libraries Sep 30, 2022

jordarlu marked this pull request as ready for review September 30, 2022 18:31

jordarlu requested a review from a team as a code owner September 30, 2022 18:31

jordarlu mentioned this pull request Sep 30, 2022

[Bug]: Following up #2679 - Updating TestReleaseTagJob and restore it #2680

Closed

jordarlu mentioned this pull request Sep 30, 2022

Add security plugin to 2.3.1 #2677

Merged

peterzhuamazon mentioned this pull request Sep 30, 2022

Remove deployment directory as it is migrated to opensearch-ci repo #2682

Merged

jordarlu changed the title ~~Clean-up step of relocating Jenkins shared libraries~~ Clean-up step of migrating Jenkins shared libraries to opensearch-build-libraries repo Sep 30, 2022

remove AssembleUpload_Jenkinsfile

f35a9de

Signed-off-by: Jeff Lu <[email protected]>

remove /jobs/ as it has been migrated to build-lib repo

62879e7

Signed-off-by: Jeff Lu <[email protected]>

prudhvigodithi approved these changes Oct 3, 2022

View reviewed changes

jordarlu merged commit a8f72cb into opensearch-project:main Oct 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean-up step of migrating Jenkins shared libraries to opensearch-build-libraries repo #2679

Clean-up step of migrating Jenkins shared libraries to opensearch-build-libraries repo #2679

jordarlu commented Sep 30, 2022 •

edited

Loading

codecov-commenter commented Sep 30, 2022 •

edited

Loading

prudhvigodithi commented Sep 30, 2022

jordarlu commented Sep 30, 2022

prudhvigodithi commented Oct 1, 2022

jordarlu commented Oct 1, 2022

prudhvigodithi commented Oct 1, 2022 •

edited

Loading

jordarlu commented Oct 1, 2022

gaiksaya commented Oct 1, 2022

jordarlu commented Oct 1, 2022

gaiksaya commented Oct 3, 2022

jordarlu commented Oct 3, 2022

prudhvigodithi commented Oct 3, 2022

Clean-up step of migrating Jenkins shared libraries to opensearch-build-libraries repo #2679

Clean-up step of migrating Jenkins shared libraries to opensearch-build-libraries repo #2679

Conversation

jordarlu commented Sep 30, 2022 • edited Loading

Description

Issues Resolved

codecov-commenter commented Sep 30, 2022 • edited Loading

Codecov Report

prudhvigodithi commented Sep 30, 2022

jordarlu commented Sep 30, 2022

prudhvigodithi commented Oct 1, 2022

jordarlu commented Oct 1, 2022

prudhvigodithi commented Oct 1, 2022 • edited Loading

jordarlu commented Oct 1, 2022

gaiksaya commented Oct 1, 2022

jordarlu commented Oct 1, 2022

gaiksaya commented Oct 3, 2022

jordarlu commented Oct 3, 2022

prudhvigodithi commented Oct 3, 2022

jordarlu commented Sep 30, 2022 •

edited

Loading

codecov-commenter commented Sep 30, 2022 •

edited

Loading

prudhvigodithi commented Oct 1, 2022 •

edited

Loading