Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update Azure & AWS SDKs in exchange filesystem #17100

Merged
merged 1 commit into from
Apr 25, 2023
Merged

Update Azure & AWS SDKs in exchange filesystem #17100

merged 1 commit into from
Apr 25, 2023

Conversation

wendigo
Copy link
Contributor

@wendigo wendigo commented Apr 18, 2023

No description provided.

@cla-bot cla-bot bot added the cla-signed label Apr 18, 2023
@electrum
Copy link
Member

Lots of Netty conflicts. We should update Netty to 4.1.91.Final

@wendigo
Copy link
Contributor Author

wendigo commented Apr 18, 2023

@electrum I'll do it

@wendigo wendigo force-pushed the serafin/update-aws-azure-sdks branch from 43d5f19 to 6ef8d1c Compare April 19, 2023 08:12
@wendigo wendigo force-pushed the serafin/update-aws-azure-sdks branch 3 times, most recently from 70f1f01 to d9ebe60 Compare April 19, 2023 15:08
@github-actions github-actions bot added the delta-lake Delta Lake connector label Apr 19, 2023
@electrum
Copy link
Member

Looks like the AWS client has a bug or incompatibility with MinIO:

software.amazon.awssdk.services.s3.model.S3Exception: An error occurred when parsing the HTTP request PUT at '/' (Service: S3, Status Code: 400, Request ID: )
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95)
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:270)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
	at software.amazon.awssdk.services.s3.DefaultS3Client.createBucket(DefaultS3Client.java:1176)
	at io.trino.plugin.exchange.filesystem.containers.MinioStorage.start(MinioStorage.java:65)
	at io.trino.plugin.deltalake.TestDeltaTaskFailureRecoveryTest.createQueryRunner(TestDeltaTaskFailureRecoveryTest.java:55)
	at io.trino.testing.BaseFailureRecoveryTest.createQueryRunner(BaseFailureRecoveryTest.java:109)
	at io.trino.testing.AbstractTestQueryFramework.init(AbstractTestQueryFramework.java:104)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:515)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:217)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:144)
	at org.testng.internal.TestMethodWorker.invokeBeforeClassMethods(TestMethodWorker.java:169)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

@electrum
Copy link
Member

Do you want to split out the Azure updates? I think we also need to test with secrets

@wendigo wendigo force-pushed the serafin/update-aws-azure-sdks branch from d9ebe60 to 5ada112 Compare April 20, 2023 18:45
@wendigo
Copy link
Contributor Author

wendigo commented Apr 20, 2023

@electrum I've fixed the configuration and it should work with the Minio (the problem was the lack of the s3 path style access)

@wendigo
Copy link
Contributor Author

wendigo commented Apr 21, 2023 •
edited
Loading

@electrum Can you run this with secrets?

@hashhar
Copy link
Member

hashhar commented Apr 21, 2023

/test-with-secrets sha=5ada11295297abd6ae8d1fdc337596ab0f13ec79

@github-actions
Copy link

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4763998549

@wendigo
Copy link
Contributor Author

wendigo commented Apr 21, 2023

Failures are unrelated. PTAL @electrum

@losipiuk
Copy link
Member

/test-with-secrets sha=5ada11295297abd6ae8d1fdc337596ab0f13ec79

@losipiuk
Copy link
Member

@wendigo is this related?

@github-actions
Copy link

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4777554151

@wendigo wendigo force-pushed the serafin/update-aws-azure-sdks branch from 5ada112 to 33f73cd Compare April 24, 2023 13:20
@wendigo
Copy link
Contributor Author

wendigo commented Apr 24, 2023

I've rebased this PR. Can you run it with secrets @ebyhr @hashhar ?

@hashhar
Copy link
Member

hashhar commented Apr 24, 2023

/test-with-secrets sha=33f73cd71edd3e27326b48479d4db23b4bd5b398

@github-actions
Copy link

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/4789114677

@wendigo
Copy link
Contributor Author

wendigo commented Apr 25, 2023

This is good to go. Failures are unrelated to this change.

@ebyhr @hashhar ptal

@kokosing kokosing merged commit 1ed247e into trinodb:master Apr 25, 2023
@github-actions github-actions bot added this to the 415 milestone Apr 25, 2023
@colebow colebow mentioned this pull request Apr 26, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@losipiuk losipiuk losipiuk approved these changes

@hashhar hashhar hashhar approved these changes

@linzebing linzebing linzebing approved these changes

@kokosing kokosing Awaiting requested review from kokosing

Assignees
No one assigned
Labels
cla-signed delta-lake Delta Lake connector
Milestone
415
Development

Successfully merging this pull request may close these issues.
6 participants
@wendigo @electrum @hashhar @losipiuk @linzebing @kokosing