HADOOP-17982. OpensslCipher initialization error should log a WARN message.

[jojochuang]

Description of PR

log a WARN message if OpensslCipher fails to initialize.

How was this patch tested?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[iwasakims]

LGTM.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 34s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 26s		trunk passed
+1 💚	compile	27m 41s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	23m 55s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	1m 10s		trunk passed
+1 💚	mvnsite	1m 54s		trunk passed
+1 💚	javadoc	1m 19s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 52s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 48s		trunk passed
+1 💚	shadedclient	28m 58s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 5s		the patch passed
+1 💚	compile	26m 29s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	26m 29s		the patch passed
+1 💚	compile	24m 42s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	24m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 17s		the patch passed
+1 💚	mvnsite	1m 42s		the patch passed
+1 💚	javadoc	1m 7s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	1m 44s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 56s		the patch passed
+1 💚	shadedclient	28m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	18m 35s		hadoop-common in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		238m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3599/1/artifact/out/Dockerfile
GITHUB PR	#3599
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 0752c1b0b83f 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 886014d
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3599/1/testReport/
Max. process+thread count	2548 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3599/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[steveloughran]

1. is this going to warn on all commands run in envs without ssl on the path, or is it just from when there's a hadoop build without openssl?
2. would it make sense to have a new log for the warning which you could turn off if you don't want to see it.

side issue: if you have a PR for my cloudstore module which adds an "openssl" command that probes for this ,and returns an error code if not satisfied, happy to merge it. good for support calls

[jojochuang]

1. is this going to warn on all commands run in envs without ssl on the path, or is it just from when there's a hadoop build without openssl?

not on all. openssl is initiated only when accessing encrypted file/ if the transport is encrypted.

2. would it make sense to have a new log for the warning which you could turn off if you don't want to see it.

side issue: if you have a PR for my cloudstore module which adds an "openssl" command that probes for this ,and returns an error code if not satisfied, happy to merge it. good for support calls
The 'hadoop checknative' should be sufficient already..

[steveloughran]

thanks for the feedback

+1 from me

[iwasakims]

@jojochuang I'm going to merge this shortly if there is no additional comment.

[iwasakims]

On CentOS 8, OpensslCipher#initIDs raises exception since the provided openssl-1.1.1 is built with no-sm4 option as I reported in HADOOP-17609(#3019). Message like following appears after this fix.

$ bin/hdfs dfs -cat /zone1/README.txt
2021-12-10 09:35:42,766 WARN crypto.OpensslCipher: Failed to load OpenSSL Cipher.
java.lang.UnsatisfiedLinkError: Cannot find AES-CTR/SM4-CTR support, is your version of Openssl new enough?
        at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
        at org.apache.hadoop.crypto.OpensslCipher.<clinit>(OpensslCipher.java:90)
        at org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec.<init>(OpensslAesCtrCryptoCodec.java:36)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:155)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:127)
        at org.apache.hadoop.crypto.CryptoCodec.getInstance(CryptoCodec.java:69)
        at org.apache.hadoop.hdfs.HdfsKMSUtil.getCryptoCodec(HdfsKMSUtil.java:110)
        at org.apache.hadoop.hdfs.HdfsKMSUtil.createWrappedInputStream(HdfsKMSUtil.java:193)
        at org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:972)
        at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:340)
        at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:334)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:351)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:974)
        at org.apache.hadoop.fs.shell.Display$Cat.getInputStream(Display.java:108)
        at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:96)
        at org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:370)
        at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:333)
        at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:306)
        at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:288)
        at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:272)
        at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121)
        at org.apache.hadoop.fs.shell.Command.run(Command.java:179)
        at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)
        at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
...

[steveloughran]

What do you recommend we do here?

[iwasakims]

@steveloughran I submitted a patch to enable OpensslCipher even the OpenSSL is built without SM4 support on #3019, for existing users of AES/CTR/NoPadding on REEL/CentOS 8.