Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDFS Repository fails when over-the-wire encryption is enabled #76734

Closed
masseyke opened this issue Aug 19, 2021 · 1 comment · Fixed by #76897
Closed

HDFS Repository fails when over-the-wire encryption is enabled #76734

masseyke opened this issue Aug 19, 2021 · 1 comment · Fixed by #76897
Labels
>bug Team:Data Management Meta label for data/management team

Comments

@masseyke
Copy link
Member

masseyke commented Aug 19, 2021 •
edited
Loading

Elasticsearch version (bin/elasticsearch --version): 7.11.1

Plugins installed: [repository-hdfs]

JVM version (8 or 15):

Description of the problem including expected versus actual behavior:

Steps to reproduce:
If HDFS is configured to use over-the-wire encryption (i.e. dfs.encrypt.data.transfer is set to true), then the repository-hdfs plugin fails when creating a snapshot. The reason appears to be that this setting results in the HDFS client using CryptoOutputStream, which requires some additional runtime permissions that this plugin is not already granted.
To make matters worse, the version of CryptoOutputStream in the hadoop 2 client is not compatible with java 9 and later. It looks like this has only been addressed in the hadoop 3 client. Here is a stack trace on java 8:

java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.nio.ch")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_275]
at java.security.AccessController.checkPermission(AccessController.java:886) ~[?:1.8.0_275]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_275]
at java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1564) ~[?:1.8.0_275]
at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814) ~[?:1.8.0_275]
at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_275]
at org.apache.hadoop.crypto.CryptoStreamUtils.freeDB(CryptoStreamUtils.java:39) ~[?:?]
at org.apache.hadoop.crypto.CryptoInputStream.freeBuffers(CryptoInputStream.java:683) ~[?:?]
at org.apache.hadoop.crypto.CryptoInputStream.close(CryptoInputStream.java:317) ~[?:?]
at java.io.FilterInputStream.close(FilterInputStream.java:181) ~[?:1.8.0_275]
at org.apache.hadoop.hdfs.DataStreamer.closeStream(DataStreamer.java:996) ~[?:?]
at org.apache.hadoop.hdfs.DataStreamer.closeInternal(DataStreamer.java:839) ~[?:?]
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:834) ~[?:?]
@masseyke masseyke added >bug needs:triage Requires assignment of a team area label labels Aug 19, 2021
@masseyke masseyke linked a pull request Aug 19, 2021 that will close this issue
Updating permissions so that CryptoOutputStream works #76585
Closed
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Aug 19, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (Team:Core/Features)

@jtibshirani jtibshirani removed the needs:triage Requires assignment of a team area label label Aug 20, 2021
@masseyke masseyke mentioned this issue Aug 24, 2021
Closed
@masseyke masseyke linked a pull request Aug 24, 2021 that will close this issue
Upgrade repository-hdfs to Hadoop 3 #76897
Merged
@masseyke masseyke mentioned this issue Sep 28, 2021
Merged
masseyke added a commit that referenced this issue Oct 6, 2021
@masseyke
Adding config so that some tests will break if over-the-wire encrypti…
1445638 
…on fails (#78409)

Until recently, if a user configured over-the-wire encryption for repository-hdfs they would get an exception. That was fixed in an upgraded ticket in two ways: (1) jvm permissions were opened up for haddop2, and (2) support for the hadoop 3 hdfs client was added. This commit adds configuration to a couple of integration tests so that they fail if over-the-wire encryption is not working.
Relates #76897 #76734
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug Team:Data Management Meta label for data/management team
Projects
None yet
Milestone
No milestone
3 participants
@masseyke @jtibshirani @elasticmachine