Remove BouncyCastle dependency from runtime #32193

jkakavas · 2018-07-19T12:24:54Z

This commit introduces a new project that builds a Jar of the
classes that have a dependency on BouncyCastle and puts it in a
subdirectory of lib (/tools/security-cli) along with the
BouncyCastle jars. This directory is then passed in the
ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools that use
these classes.

BouncyCastle is removed as a runtime dependency (remains
as a compileOnly one) from x-pack core and x-pack security.

This commit introduces a new project that builds a Jar of the classes that have a dependency on BouncyCastle and puts it in a subdirectory of lib (/tools/security-cli) along with the BouncyCastle jars. This directory is then passed in the ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools that use these classes. BouncyCastle is removed as a runtime dependency (remains as a compileOnly one) from x-pack core and x-pack security.

elasticmachine · 2018-07-19T12:24:56Z

Pinging @elastic/es-core-infra

jaymode · 2018-07-19T16:09:57Z

distribution/build.gradle

@@ -238,6 +238,10 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
        from { project(':distribution:tools:plugin-cli').jar }
        from { project(':distribution:tools:plugin-cli').configurations.runtime }
      }
+      into('tools/security-cli') {


I am pretty sure this is not where we want to add this since the comment makes me think this section also applies to the oss distribution which does not have xpack.

Thanks, that makes sense. I can take advantage of the oss boolean and call with libFiles(oss) where applicable so that it only includes the jars in tools/security-cli is oss is false

Also, I think this can probably just go under tools now that I see where we are putting the jar. My suggestion for this directory yesterday was due to my outdated view of how files are laid out within the distribution.

Hmm...I think that tools/security-cli is the right place? It mimics what we did with the plugin-cli.

derp. Ignore me and leave it as is

jaymode · 2018-07-19T16:12:08Z

x-pack/plugin/security/cli/build.gradle

+}
+
+jar {
+    from(project(':x-pack:plugin:core').sourceSets.main.output.classesDirs) {


can we not move the source files to this project? At the very least I think we should exclude these from the x-pack core jar

Can you elaborate please? We're not moving source files to this project now. Am i reading your comment wrong or maybe the not in can we not move isn't meant to be there ?

I - probably falsely - assumed we do not want to move the source files to this project since we preferred this approach to the new plugin approach I tried out yesterday. It definitely makes sense to not have the classes both in this jar and in x-pack-core though - thanks.

Sorry, my comment was not clear. I'd like for us to move the source files to this project for a clearer separation.

The distinction between a separate plugin vs a jar is that the certificate generation code doesn't require a plugin and just needs to be packaged in a way that it can be loaded as a CLI tool. I suggested this way as an alternative to a completely separate plugin to avoid any confusion about seeing a security-cli plugin/module deployed.

Great thanks, all clear now. I'll move the source files in this project

jasontedor

I agree with @jaymode's comments and left a question.

jasontedor · 2018-07-19T16:14:31Z

x-pack/plugin/security/cli/build.gradle

+    mapping from: /bc.*/, to: 'bouncycastle'
+}
+
+licenseHeaders.enabled = false


Can you explain what is happening here?

I can certainly try. I started off without the licenseHeaders.enabled = false and precommit checks failed with:

> Task :x-pack:plugin:security:cli:licenseHeaders FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':x-pack:plugin:security:cli:licenseHeaders'. > You must specify at least one file to create the report for.

I interpreted this to be an error caused by the licenseHeaders task looking for Java files in order to check the license heades in security-cli and failing because there aren't any. Depending on whether I'll bring in the source files (#32193 (comment)) , I'll remove this but what should have been the correct approach in this case?

Sorry, I was unclear. I meant can you leave a comment in the build.gradle about this. Of course, this changes if we do bring in the source files.

- Move source for CertificateTool, CertificateGenerateTool and CertGenUtils and associated tests to the new project - Ensure that the genrated jar and bc dependencies are not bundled in oss

jkakavas · 2018-07-19T22:00:15Z

@jaymode @jasontedor this is ready for another round

jaymode

LGTM. I like this better and am so happy to see BC removed from the security runtime dependencies

jaymode · 2018-07-19T22:27:22Z

x-pack/plugin/security/cli/build.gradle

+    testCompile "junit:junit:${versions.junit}"
+    testCompile "org.hamcrest:hamcrest-all:${versions.hamcrest}"
+    testCompile 'org.elasticsearch:securemock:1.2'
+    testCompile("org.elasticsearch.test:framework:${version}")


Nit: can you change this to the same style used above for the elasticsearch compile dependency

jasontedor

Looks great! I left one comment so this is approved contingent on adding a test.

jasontedor · 2018-07-19T22:40:12Z

distribution/build.gradle

+          from { project(':distribution:tools:plugin-cli').jar }
+          from { project(':distribution:tools:plugin-cli').configurations.runtime }
+        }
+        if (oss == false) {


My only ask is a packaging test for this, that these are not in the OSS distribution, and are in the default distribution. In the new style. I am happy to guide, let me know.

I'll take a swing at it and will ping for help if I find myself stumbling in the dark

jkakavas · 2018-07-20T11:38:16Z

@jasontedor I added the a package test, please take a look and see if this resembles what you had in mind.

Also, can you please comment on the use of testImplementation for the dependencies closure of security-cli. It looks like the correct one for this use case (and I do get related errors running the tests with only testCompile) but we are not using it anywhere else in our build scripts so I was wondering if we're solving this in some other manner.

BouncyCastle returns as a compileOnly dependency for security as Opensaml has a compile dependency on cryptacular which hash a compile dependency on bouncycastle.

jasontedor · 2018-07-20T20:07:22Z

Thanks @jkakavas, everything looks great, including the use of testImplementation. Thanks a lot for adding the test.

* es/master: (23 commits) Switch full-cluster-restart to new style Requests (#32140) [DOCS] Clarified that you must remove X-Pack plugin when upgrading from pre-6.3. (#32016) Remove BouncyCastle dependency from runtime (#32193) INGEST: Extend KV Processor (#31789) (#32232) INGEST: Make a few Processors callable by Painless (#32170) Add region ISO code to GeoIP Ingest plugin (#31669) [Tests] Remove QueryStringQueryBuilderTests#toQuery class assertions (#32236) Make sure that field aliases count towards the total fields limit. (#32222) Switch rolling restart to new style Requests (#32147) muting failing test for internal auto date histogram to avoid failure before fix is merged MINOR: Remove unused `IndexDynamicSettings` (#32237) Fix multi level nested sort (#32204) Enhance Parent circuit breaker error message (#32056) [ML] Use default request durability for .ml-state index (#32233) Remove indices stats timeout from monitoring docs Rename ranking evaluation response section (#32166) Dependencies: Upgrade to joda time 2.10 (#32160) Remove aliases resolution limitations when security is enabled (#31952) Ensure that field aliases cannot be used in multi-fields. (#32219) TESTS: Check for Netty resource leaks (#31861) ...

* Remove BouncyCastle dependency from runtime This commit introduces a new gradle project that contains the classes that have a dependency on BouncyCastle. For the default distribution, It builds a jar from those and in puts it in a subdirectory of lib (/tools/security-cli) along with the BouncyCastle jars. This directory is then passed in the ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools that use these classes. BouncyCastle is removed as a runtime dependency (remains as a compileOnly one) from x-pack core and x-pack security.

* Remove BouncyCastle dependency from runtime This commit introduces a new gradle project that contains the classes that have a dependency on BouncyCastle. For the default distribution, It builds a jar from those and in puts it in a subdirectory of lib (/tools/security-cli) along with the BouncyCastle jars. This directory is then passed in the ES_ADDITIONAL_CLASSPATH_DIRECTORIES of the CLI tools that use these classes. BouncyCastle is removed as a runtime dependency (remains as a compileOnly one) from x-pack core and x-pack security. This is a backport of #32193

jkakavas added >enhancement review :Delivery/Build Build or test infrastructure v7.0.0 v6.4.0 labels Jul 19, 2018

jkakavas requested review from jaymode and jasontedor July 19, 2018 12:24

elasticmachine mentioned this pull request Jul 19, 2018

Security: FIPS 140-2 Compliance roadmap #29851

Closed

12 tasks

jaymode requested changes Jul 19, 2018

View reviewed changes

jasontedor requested changes Jul 19, 2018

View reviewed changes

Address feedback

15f8c15

- Move source for CertificateTool, CertificateGenerateTool and CertGenUtils and associated tests to the new project - Ensure that the genrated jar and bc dependencies are not bundled in oss

jaymode approved these changes Jul 19, 2018

View reviewed changes

jasontedor approved these changes Jul 19, 2018

View reviewed changes

jkakavas added 6 commits July 20, 2018 01:42

Update class package in scripts

27286ac

address feedback

aca04e2

Add packaging test for security cli

c40137f

Reintroduce compileOnly dependency for security

af5745a

ensure jimfs is available in test runtime

defa749

Fix security-cli tests

c2e55c2

jkakavas merged commit aaa8f84 into elastic:master Jul 20, 2018

jkakavas added backport pending and removed review labels Jul 20, 2018

jkakavas mentioned this pull request Jul 26, 2018

Remove BouncyCastle dependency from runtime (#32193) #32402

Merged

jkakavas deleted the cli-project branch September 14, 2018 06:55

danielmitterdorfer removed the backport pending label Jan 31, 2019

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove BouncyCastle dependency from runtime #32193

Remove BouncyCastle dependency from runtime #32193

jkakavas commented Jul 19, 2018

elasticmachine commented Jul 19, 2018

jaymode Jul 19, 2018

jkakavas Jul 19, 2018

jaymode Jul 19, 2018

jasontedor Jul 19, 2018

jaymode Jul 19, 2018

jaymode Jul 19, 2018

jkakavas Jul 19, 2018

jaymode Jul 19, 2018

jkakavas Jul 19, 2018

jasontedor left a comment

jasontedor Jul 19, 2018

jkakavas Jul 19, 2018

jasontedor Jul 19, 2018

jkakavas commented Jul 19, 2018

jaymode left a comment

jaymode Jul 19, 2018

jasontedor left a comment

jasontedor Jul 19, 2018

jkakavas Jul 19, 2018

jkakavas commented Jul 20, 2018

jasontedor commented Jul 20, 2018

Remove BouncyCastle dependency from runtime #32193

Remove BouncyCastle dependency from runtime #32193

Conversation

jkakavas commented Jul 19, 2018

elasticmachine commented Jul 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasontedor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkakavas commented Jul 19, 2018

jaymode left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasontedor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkakavas commented Jul 20, 2018

jasontedor commented Jul 20, 2018