-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
presto access hive OCI object external table failed. return Unable to extract password #51
Comments
yes, i depoly the hdfs-connect on HDP3.1.4 cluster. and it can run select * from table on hive, it's success, but run select count(*) failed. |
@zhengwanbo thanks for filing this issue - we'll take a look and get back to you |
hi @zhengwanbo, It doesn’t seem to be an issue with the HDFS connector but rather with the way you are passing passphrase using JCEKS generated credential files.
Please let us know if you any more questions |
@zhengwanbo - checking in here since we haven't heard back from you in a while. Did @omkar07's response resolve your issue? |
Hi @zhengwanbo - since we haven't heard back from you, we are resolving this issue. But feel free to reopen this if you are still experiencing problems. |
env:
1、presto-server-0.251
2、HDP 3.1.4.0-315
3、hdfs-connector : oci-hdfs-full-3.3.0.7.0.1.jar
logs:
[root@bigdata-hadoop-1 presto-server-0.251]# presto-cli --server bigdata-hadoop-1:8089 --catalog hive --schema ssb100_poc
presto:ssb100_poc> show tables;
Table
customer
customer_orc
dates
dates_orc
lineorder
lineorder_flat
lineorder_flat_orc
lineorder_orc
lineorder_partion_orc
part
part_orc
ssb_customer_orc
ssb_customer_txt_obj
ssb_date_txt_obj
ssb_dates_orc
ssb_lineorder_orc
ssb_part_orc
ssb_supplier_orc
supplier
supplier_orc
(20 rows)
Query 20210701_091214_00001_c45wy, FINISHED, 3 nodes
Splits: 53 total, 53 done (100.00%)
0:19 [20 rows, 651B] [1 rows/s, 35B/s]
presto:ssb100_poc> show create table ssb_customer_txt_obj;
Create Table
CREATE TABLE hive.ssb100_poc.ssb_customer_txt_obj (
c_custkey integer,
c_name varchar,
c_address varchar,
c_city varchar,
c_nation varchar,
c_region varchar,
c_phone varchar,
c_mktsegment varchar
)
WITH (
external_location = 'oci://bigdata@ocichina001/ssb100_data/customer',
format = 'TEXTFILE'
)
(1 row)
Query 20210701_091239_00002_c45wy, FINISHED, 1 node
Splits: 1 total, 1 done (100.00%)
0:04 [0 rows, 0B] [0 rows/s, 0B/s]
presto:ssb100_poc> select * from ssb_customer_txt_obj limit 2;
Query 20210701_091255_00004_c45wy, FAILED, 1 node
Splits: 17 total, 0 done (0.00%)
0:07 [0 rows, 0B] [0 rows/s, 0B/s]
Query 20210701_091255_00004_c45wy failed: Unable to extract password
presto:ssb100_poc> select count(*) from ssb_customer_txt_obj;
Query 20210701_091315_00005_c45wy failed: Unable to extract password
presto:ssb100_poc>
===========================================================================
vi server.log
2021-07-01T17:17:40.396+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is QUEUED
2021-07-01T17:17:40.409+0800 DEBUG dispatcher-query-13 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is WAITING_FOR_RESOURCES
2021-07-01T17:17:40.884+0800 DEBUG dispatcher-query-14 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is PLANNING
2021-07-01T17:17:40.891+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is DISPATCHING
2021-07-01T17:17:40.981+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is PLANNED
2021-07-01T17:17:40.989+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is PLANNED
2021-07-01T17:17:41.001+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is STARTING
2021-07-01T17:17:41.001+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is SCHEDULING
2021-07-01T17:17:41.031+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is SCHEDULING
2021-07-01T17:17:41.037+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is SCHEDULED
2021-07-01T17:17:41.037+0800 DEBUG dispatcher-query-11 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is RUNNING
2021-07-01T17:17:41.095+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is RUNNING
2021-07-01T17:17:41.138+0800 INFO hive-hive-3 java.io.serialization ObjectInputFilter {0}: {1}, array length: {2}, nRefs: {3}, depth: {4}, bytes: {5}, ex: {6}
2021-07-01T17:17:41.156+0800 DEBUG dispatcher-query-13 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy is FAILED
2021-07-01T17:17:41.156+0800 DEBUG query-execution-5 com.facebook.presto.execution.QueryStateMachine Query 20210701_091740_00007_c45wy failed
com.facebook.presto.spi.PrestoException: Unable to extract password
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:128)
at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Unable to extract password
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:169)
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:132)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createAuthenticator(BmcDataStoreFactory.java:362)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.buildClient(BmcDataStoreFactory.java:193)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createClient(BmcDataStoreFactory.java:101)
at com.oracle.bmc.hdfs.store.BmcDataStoreFactory.createDataStore(BmcDataStoreFactory.java:88)
at com.oracle.bmc.hdfs.BmcFilesystem.initialize(BmcFilesystem.java:143)
at org.apache.hadoop.fs.PrestoFileSystemCache.createFileSystem(PrestoFileSystemCache.java:149)
at org.apache.hadoop.fs.PrestoFileSystemCache.getInternal(PrestoFileSystemCache.java:89)
at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:62)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at com.facebook.presto.hive.cache.HiveCachingHdfsConfiguration.lambda$getConfiguration$0(HiveCachingHdfsConfiguration.java:68)
at com.facebook.presto.hive.cache.HiveCachingHdfsConfiguration$CachingJobConf.createFileSystem(HiveCachingHdfsConfiguration.java:104)
at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:59)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at com.facebook.presto.hive.HdfsEnvironment.lambda$getFileSystem$0(HdfsEnvironment.java:71)
at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:70)
at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:64)
at com.facebook.presto.hive.StoragePartitionLoader.loadPartition(StoragePartitionLoader.java:163)
at com.facebook.presto.hive.DelegatingPartitionLoader.loadPartition(DelegatingPartitionLoader.java:78)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:192)
at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:40)
at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:121)
... 7 more
Caused by: java.io.IOException: Configuration problem with provider path.
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1966)
at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:1926)
at com.oracle.bmc.hdfs.store.BmcPropertyAccessor$4.get(BmcPropertyAccessor.java:140)
... 32 more
Caused by: java.io.IOException: Invalid secret key format
at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:850)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.apache.hadoop.security.alias.AbstractJavaKeyStoreProvider.(AbstractJavaKeyStoreProvider.java:106)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:49)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:41)
at org.apache.hadoop.security.alias.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:100)
at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:71)
at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:1947)
... 34 more
2021-07-01T17:17:41.157+0800 DEBUG query-execution-4 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.1.0 is ABORTED
2021-07-01T17:17:41.157+0800 DEBUG query-execution-7 com.facebook.presto.execution.StageExecutionStateMachine Stage Execution 20210701_091740_00007_c45wy.0.0 is ABORTED
2021-07-01T17:17:41.160+0800 DEBUG page-buffer-client-callback-14 com.facebook.presto.operator.PageBufferClient Request to http://10.0.0.4:8089/v1/task/async/20210701_091740_00007_c45wy.0.0.0/results/0/0 failed java.util.concurrent.CancellationException: Task was cancelled.
2021-07-01T17:17:41.348+0800 INFO dispatcher-query-13 com.facebook.presto.event.QueryMonitor TIMELINE: Query 20210701_091740_00007_c45wy :: Transaction:[43bc0f84-e357-4636-8e2a-bab7eb6ed6ae] :: elapsed 751ms :: planning 117ms :: scheduling 634ms :: running 0ms :: finishing 634ms :: begin 2021-07-01T17:17:40.396+08:00 :: end 2021-07-01T17:17:41.147+08:00
The text was updated successfully, but these errors were encountered: