Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

presto access hive OCI object external table failed. return Unable to extract password #51

Closed
zhengwanbo opened this issue Jul 1, 2021 · 6 comments

Comments

@zhengwanbo
Copy link

env:
1、presto-server-0.251
2、HDP 3.1.4.0-315
3、hdfs-connector : oci-hdfs-full-3.3.0.7.0.1.jar

logs:

[root@bigdata-hadoop-1 presto-server-0.251]# presto-cli --server bigdata-hadoop-1:8089 --catalog hive --schema ssb100_poc
presto:ssb100_poc> show tables;
Table

customer
customer_orc
dates
dates_orc
lineorder
lineorder_flat
lineorder_flat_orc
lineorder_orc
lineorder_partion_orc
part
part_orc
ssb_customer_orc
ssb_customer_txt_obj
ssb_date_txt_obj
ssb_dates_orc
ssb_lineorder_orc
ssb_part_orc
ssb_supplier_orc
supplier
supplier_orc
(20 rows)

Query 20210701_091214_00001_c45wy, FINISHED, 3 nodes
Splits: 53 total, 53 done (100.00%)
0:19 [20 rows, 651B] [1 rows/s, 35B/s]

presto:ssb100_poc> show create table ssb_customer_txt_obj;
Create Table

CREATE TABLE hive.ssb100_poc.ssb_customer_txt_obj (
c_custkey integer,
c_name varchar,
c_address varchar,
c_city varchar,
c_nation varchar,
c_region varchar,
c_phone varchar,
c_mktsegment varchar
)
WITH (
external_location = 'oci://bigdata@ocichina001/ssb100_data/customer',
format = 'TEXTFILE'
)
(1 row)

Query 20210701_091239_00002_c45wy, FINISHED, 1 node
Splits: 1 total, 1 done (100.00%)
0:04 [0 rows, 0B] [0 rows/s, 0B/s]

presto:ssb100_poc> select * from ssb_customer_txt_obj limit 2;

Query 20210701_091255_00004_c45wy, FAILED, 1 node
Splits: 17 total, 0 done (0.00%)
0:07 [0 rows, 0B] [0 rows/s, 0B/s]

Query 20210701_091255_00004_c45wy failed: Unable to extract password

presto:ssb100_poc> select count(*) from ssb_customer_txt_obj;
Query 20210701_091315_00005_c45wy failed: Unable to extract password

presto:ssb100_poc>

===========================================================================

vi server.log

2021-07-01T17:17:40.396+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is QUEUED
2021-07-01T17:17:40.409+0800 DEBUG dispatcher-query-13 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is WAITING_FOR_RESOURCES
2021-07-01T17:17:40.884+0800 DEBUG dispatcher-query-14 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is PLANNING
2021-07-01T17:17:40.891+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is DISPATCHING
2021-07-01T17:17:40.981+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is PLANNED
2021-07-01T17:17:40.989+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is PLANNED
2021-07-01T17:17:41.001+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is STARTING
2021-07-01T17:17:41.001+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is SCHEDULING
2021-07-01T17:17:41.031+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is SCHEDULING
2021-07-01T17:17:41.037+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is SCHEDULED
2021-07-01T17:17:41.037+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is RUNNING
2021-07-01T17:17:41.095+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is RUNNING
2021-07-01T17:17:41.138+0800 INFO hive-hive-3 java.io.serialization ObjectInputFilter {0}: {1}, array length: {2}, nRefs: {3}, depth: {4}, bytes: {5}, ex: {6}
2021-07-01T17:17:41.156+0800 DEBUG dispatcher-query-13 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is FAILED
2021-07-01T17:17:41.156+0800 DEBUG query-execution-5 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy failed
com.facebook.presto.spi.PrestoException: Unable to extract password
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:128)
at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Unable to extract password
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:169)
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:132)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createAuthenticator(BmcDataStoreFactory.java:362)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.buildClient(BmcDataStoreFactory.java:193)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createClient(BmcDataStoreFactory.java:101)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createDataStore(BmcDataStoreFactory.java:88)
at com.oracle.bmc.hdfs.BmcFilesystem.initialize(BmcFilesystem.java:143)
at org.apache.hadoop.fs.PrestoFileSystemCache.createFileSystem(PrestoFileSystemCache.java:149)
at org.apache.hadoop.fs.PrestoFileSystemCache.getInternal(PrestoFileSystemCache.java:89)
at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:62)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at com.facebook.presto.hive.cache.HiveCachingHdfsConfiguration.lambda$getConfiguration$0(HiveCachingHdfsConfiguration.java:68)
at com.facebook.presto.hive.cache.HiveCachingHdfsConfiguration$CachingJobConf.createFileSystem(HiveCachingHdfsConfiguration.java:104)
at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:59)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at com.facebook.presto.hive.HdfsEnvironment.lambda$getFileSystem$0(HdfsEnvironment.java:71)
at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:70)
at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:64)
at com.facebook.presto.hive.StoragePartitionLoader.loadPartition(StoragePartitionLoader.java:163)
at com.facebook.presto.hive.DelegatingPartitionLoader.loadPartition(DelegatingPartitionLoader.java:78)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:192)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:40)
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:121)
... 7 more
Caused by: java.io.IOException: Configuration problem with provider path.
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1966)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1926)
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:140)
... 32 more
Caused by: java.io.IOException: Invalid secret key format
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:850)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:106)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:49)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:41)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100)
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:71)
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1947)
... 34 more

2021-07-01T17:17:41.157+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is ABORTED
2021-07-01T17:17:41.157+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is ABORTED
2021-07-01T17:17:41.160+0800 DEBUG page-buffer-client-callback-14 com.facebook.presto.operator.PageBufferClient Request to http://10.0.0.4:8089/v1/task/async/20210701_091740_00007_c45wy.0.0.0/results/0/0 failed java.util.concurrent.CancellationException: Task was cancelled.
2021-07-01T17:17:41.348+0800 INFO dispatcher-query-13 com.facebook.presto.event.QueryMonitor TIMELINE: Query 20210701_091740_00007_c45wy :: Transaction:[43bc0f84-e357-4636-8e2a-bab7eb6ed6ae] :: elapsed 751ms :: planning 117ms :: scheduling 634ms :: running 0ms :: finishing 634ms :: begin 2021-07-01T17:17:40.396+08:00 :: end 2021-07-01T17:17:41.147+08:00

@tooptoop4
Copy link

@zhengwanbo
Copy link
Author

yes, i depoly the hdfs-connect on HDP3.1.4 cluster. and it can run select * from table on hive, it's success, but run select count(*) failed.
I check the hive running by Tez engine, failed. and i also set to MR engine, it's still failed.

@jodoglevy
Copy link
Member

@zhengwanbo thanks for filing this issue - we'll take a look and get back to you

@omkar07
Copy link
Member

omkar07 commented Jul 21, 2021

hi @zhengwanbo, It doesn’t seem to be an issue with the HDFS connector but rather with the way you are passing passphrase using JCEKS generated credential files.

  1. From logs above: Unable to extract password error. [Caused by: java.io.IOException: Invalid secret key format]:
  • It seems like you're providing the passphrase through JCEKS credential file and I guess you are setting it as 'fs.oci.client.auth.passphrase' which is not correct this property is meant to accept only plain text password i.e: String.
  • In order for HDFS connector to pick up JCEKS file you need to set 'JCEKS file path' ie: hadoop.security.credential.provider.path in the (core-site.xml) as below eg:
    <property>   <name>hadoop.security.credential.provider.path</name>   <value>localjceks://file/home/lmccay/oci.jceks</value>   <description>Path to interrogate for protected credentials.</description> </property>
  • For more info on Credential API refer: https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html Or set plain text value for property ‘fs.oci.client.auth.passphrase' in core-site.xml https://docs.oracle.com/en-us/iaas/Content/API/SDKDocs/hdfsconnector.htm
  1. For run count(*) issue please refer to the solution posted on this issue hive external table of OCI object storage , run select count(*) return error : No FileSystem for scheme "oci" #50

Please let us know if you any more questions

@jodoglevy
Copy link
Member

@zhengwanbo - checking in here since we haven't heard back from you in a while. Did @omkar07's response resolve your issue?

@omkar07
Copy link
Member

omkar07 commented Aug 11, 2021

Hi @zhengwanbo - since we haven't heard back from you, we are resolving this issue. But feel free to reopen this if you are still experiencing problems.

@omkar07 omkar07 closed this as completed Aug 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants