Upgrade Azure SDK and Jackson (#72833) #72995

rjernst · 2021-05-12T16:07:49Z

This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The
Jackson upgrade must happen at the same time due to Azure depending on
this new version of Jackson.

closes #66555
closes #67214

original PR #72833

This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The Jackson upgrade must happen at the same time due to Azure depending on this new version of Jackson. closes elastic#66555 closes elastic#67214

elasticmachine · 2021-05-12T16:07:53Z

Pinging @elastic/es-distributed (Team:Distributed)

elasticmachine · 2021-05-12T16:07:53Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

rjernst · 2021-05-13T21:26:50Z

This PR is currently blocked on finding a way to not use the netty dns resolver. That (statically) tries to access /etc/hosts, which we do not allow (file reads outside of ES).

fcofdez · 2021-05-25T08:15:35Z

We were hitting this issue Azure/azure-sdk-for-java#21163 in our tests.

diff --git a/plugins/repository-azure/build.gradle b/plugins/repository-azure/build.gradle
index be1e164cb54..6fca701f359 100644
--- a/plugins/repository-azure/build.gradle
+++ b/plugins/repository-azure/build.gradle
@@ -22,8 +22,8 @@ esplugin {
 }

 versions << [
-  'azure': '12.11.0',
-  'azureCore': '1.15.0',
+  'azure': '12.11.1',
+  'azureCore': '1.16.0',
   'azureCoreHttpNetty': '1.9.1',
   'azureAvro': '12.0.3',

With these versions we should be good to go.
Could you add me as a reviewer once the PR is ready?

rjernst · 2021-05-26T15:16:25Z

@fcofdez I updated as you suggested. There is still a failure in encrypted repository.

fcofdez · 2021-05-26T17:36:20Z

@rjernst I've pushed a fix in fcofdez@a6b70eb

Before this change we were creating a new netty http client instance per invocation as the client is stateless but with the new async DNS resolvers it was creating a new one each time. That lead to hit the limit imposed in https://github.com/AdoptOpenJDK/openjdk-jdk/blob/ff4997014fe5462dca2b313f3f483400ffee5b62/src/java.base/share/classes/sun/net/ResourceManager.java#L68
Now we only create a netty http client per azure repository client, this solves the problem but we should document this limitation and how to solve it for cases where more than 25 clients are used.

rjernst · 2021-05-27T02:33:19Z

Thanks @fcofdez, that worked. CI is passing now, can you take another look at the whole PR?

fcofdez · 2021-05-27T13:03:11Z

plugins/repository-azure/src/main/plugin-metadata/plugin-security.policy

@@ -14,4 +14,6 @@ grant {
  // Used by jackson bean deserialization
  permission java.lang.RuntimePermission "accessDeclaredMembers";
  permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
+  // needed by netty dns resolver even though we don't use it (!)
+  permission java.io.FilePermission "/etc/hosts", "read";


I think we need to add permissions to list and read files within /etc/resolver. See https://github.com/netty/netty/blob/d34212439068091bcec29a8fad4df82f0a82c638/resolver-dns/src/main/java/io/netty/resolver/dns/UnixResolverDnsServerAddressStreamProvider.java#L73 @rjernst

I added the persmissions and opened a followup (#73479) to see how we might avoid using the netty dns resolver altogether.

fcofdez

LGTM 👍

This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The Jackson upgrade must happen at the same time due to Azure depending on this new version of Jackson. closes elastic#66555 closes elastic#67214 Co-authored-by: Francisco Fernández Castaño <[email protected]>

This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The Jackson upgrade must happen at the same time due to Azure depending on this new version of Jackson. closes #66555 closes #67214 backport #72995 backport #73011 Co-authored-by: Francisco Fernández Castaño <[email protected]> Co-authored-by: Mark Vieira <[email protected]>

* master: (1643 commits) Make DataStreamsSnapshotsIT resilient to failures because of local time. (elastic#73516) Upgrade netty to 4.1.63 (elastic#73011) [DOCS] Create a new page for dissect content in scripting docs (elastic#73437) Deprecate freeze index API (elastic#72618) [DOCS] Remove 'closed data stream' reference [DOCS] Update alias references (elastic#73427) [DOCS] Create a new page for grok content in scripting docs (elastic#73118) Remove dependency on azure shadowjar since it's no longer required [DOCS] Update backport policy for known issues (elastic#73489) Shadowed dependencies should be hidden from pom dependencies (elastic#73467) Disable transitive dependencies when resolving bwc JDBC driver artifact (elastic#73448) Print full JVM implementation version at start of build (elastic#73439) [DOCS] Update snapshot/restore for data stream aliases (elastic#73438) Upgrade Azure SDK and Jackson (elastic#72833) (elastic#72995) [DOCS] Fix typo (elastic#73337) (elastic#73474) [DOCS] Fix typo (elastic#73444) (elastic#73472) [DOCS] Update alias security for data stream aliases (elastic#73436) Fix Bug with Concurrent Snapshot and Index Delete (elastic#73456) [DOCS] Move common scripting use cases up a level (elastic#73445) Add more validation for data stream aliases. (elastic#73416) ...

The recent upgrade of the Azure SDK has caused a few test failures that have been difficult to debug and do not yet have a fix. In particular, a change to the netty reactor resolving (reactor/reactor-netty#1655). We need to wait for a fix for that issue, so this reverts commit 6c4c4a0. relates elastic#73493

The recent upgrade of the Azure SDK has caused a few test failures that have been difficult to debug and do not yet have a fix. In particular, a change to the netty reactor resolving (reactor/reactor-netty#1655). We need to wait for a fix for that issue, so this reverts commit 6c4c4a0. relates #73493

…elastic#73493)" The recent upgrade of the Azure SDK has caused a few test failures that have been difficult to debug and do not yet have a fix. In particular, a change to the netty reactor resolving (reactor/reactor-netty#1655). We need to wait for a fix for that issue, so this reverts commit f454cef. relates elastic#73493

…3861) The recent upgrade of the Azure SDK has caused a few test failures that have been difficult to debug and do not yet have a fix. In particular, a change to the netty reactor resolving (reactor/reactor-netty#1655). We need to wait for a fix for that issue, so this reverts commit f454cef. relates #73493

Upgrade Azure SDK and Jackson (elastic#72833)

645ca16

This commit upgrades the Azure SDK to 12.11.0 and Jackson to 2.12.2. The Jackson upgrade must happen at the same time due to Azure depending on this new version of Jackson. closes elastic#66555 closes elastic#67214

rjernst added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs :Core/Infra/Core Core issues without another label >upgrade v8.0.0 v7.14.0 labels May 12, 2021

elasticmachine added Team:Core/Infra Meta label for core/infra team Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. labels May 12, 2021

rjernst added 4 commits May 12, 2021 10:45

remove unnecessary deps

3742694

Merge branch 'master' into azure_upgrade2

44bc496

adding back dns jar

a893970

add sha

3dfe9d3

rjernst added 2 commits May 13, 2021 15:30

grant permission for /etc/hosts

04745a6

shas

8fef6fa

rjernst added 5 commits May 25, 2021 10:08

Merge branch 'master' into azure_upgrade2

ce293b4

update to 12.11.1

b787a1a

add shas

3d23c3d

Merge branch 'master' into azure_upgrade2

7df1e14

force init hack

6b9a6da

Initialize netty http client only once per azure repository client

a6b70eb

fcofdez self-requested a review May 27, 2021 07:41

fcofdez reviewed May 27, 2021

View reviewed changes

add more files needed by netty dns resolver

f857da2

rjernst added 2 commits May 27, 2021 06:59

Merge branch 'master' into azure_upgrade2

9d13d8d

allow listing /etc/resolver too

5d48e0e

fcofdez approved these changes May 27, 2021

View reviewed changes

rjernst merged commit 6c4c4a0 into elastic:master May 27, 2021

rjernst deleted the azure_upgrade2 branch May 27, 2021 14:55

rjernst mentioned this pull request May 27, 2021

Upgrade Azure SDK and Jackson (#72833) (#72995) #73493

Merged

williamrandolph mentioned this pull request May 28, 2021

[CI] Windows azure repository tests fail with AccessControlException in AzureRepositoryPlugin #73539

Closed

rjernst mentioned this pull request Jun 7, 2021

Hex numbers in YAML requests are treated as strings #66555

Open

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade Azure SDK and Jackson (#72833) #72995

Upgrade Azure SDK and Jackson (#72833) #72995

rjernst commented May 12, 2021

elasticmachine commented May 12, 2021

elasticmachine commented May 12, 2021

rjernst commented May 13, 2021

fcofdez commented May 25, 2021

rjernst commented May 26, 2021

fcofdez commented May 26, 2021

rjernst commented May 27, 2021

fcofdez May 27, 2021 •

edited

Loading

rjernst May 27, 2021

fcofdez left a comment

Upgrade Azure SDK and Jackson (#72833) #72995

Upgrade Azure SDK and Jackson (#72833) #72995

Conversation

rjernst commented May 12, 2021

elasticmachine commented May 12, 2021

elasticmachine commented May 12, 2021

rjernst commented May 13, 2021

fcofdez commented May 25, 2021

rjernst commented May 26, 2021

fcofdez commented May 26, 2021

rjernst commented May 27, 2021

fcofdez May 27, 2021 • edited Loading

Choose a reason for hiding this comment

rjernst May 27, 2021

Choose a reason for hiding this comment

fcofdez left a comment

Choose a reason for hiding this comment

fcofdez May 27, 2021 •

edited

Loading