Azure secp bulk loading #850

jframe · 2023-07-19T01:28:20Z

PR Description

Add bulk loading of secp keys to the eth1 mode of Web3Signer.

This adds the following new command line options to web3signer eth1 subcommand:

--azure-vault-enabled
--azure-vault-auth-mode
--azure-vault-name
--azure-client-id
--azure-tenant-id
--azure-client-secret
--azure-tags

Changes

Add mapKeyProperties to AzureKeyVault to enable processing of Azure keys
Created Azure bulk loader to iterate over the keys in the vault and create EthSecpArtifactSigner's
Remove ForkJoinPool hack and use an ExecutorService as the with problem blocking ForkJoin pool is also an issue with iterating over the Azure keys
Change usage of Azure code to use the AzureKeyVaultFactory instead of the static constructors of the AzureKeyVault so the ExecutorService can be reused and shutdown correctly
Add Azure bulk loading paremeters to the Eth1 config

Fixed Issue(s)

fixes #832

Documentation

I thought about documentation and added the doc-change-required label to this PR if updates are required.

Changelog

I thought about adding a changelog entry, and added one if I deemed necessary.

Testing

I thought about testing these changes in a realistic/non-local environment.

…oading

… used in the test. And remove multiple signing test as this doesn't add any value

…e key in BLS mode

…oading

…xecutorService if doesn't already exist

keystorage/src/test/java/tech/pegasys/web3signer/keystorage/azure/AzureKeyVaultTest.java

    final Optional<String> hexKey = azureKeyVault.fetchSecret(SECRET_NAME);
    Assertions.assertThat(hexKey).isNotEmpty().get().isEqualTo(EXPECTED_KEY);
  }

  @Test
  void connectingWithInvalidClientSecretThrowsException() {
    final AzureKeyVault azureKeyVault =
-        createUsingClientSecretCredentials(CLIENT_ID, "invalid", TENANT_ID, VAULT_NAME);
+        createUsingClientSecretCredentials(
+            CLIENT_ID, "invalid", TENANT_ID, VAULT_NAME, azureExecutor);


…st works with AT ThreadRunner

…oading

…ll values

signing/src/test/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactoryTest.java

+  void createsExecutorWhenUsingClientSecretMode() {
+    azureKeyVaultFactory.createAzureKeyVault(
+        "clientId",
+        "clientSecret",


signing/src/test/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactoryTest.java

+  void reusesExecutorWhenUsingClientSecretMode() {
+    azureKeyVaultFactory.createAzureKeyVault(
+        "clientId",
+        "clientSecret",


signing/src/test/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactoryTest.java

+
+    azureKeyVaultFactory.createAzureKeyVault(
+        "clientId",
+        "clientSecret",


signing/src/test/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactoryTest.java

+  void doesNotCreateExecutorWhenUsingUserAssignedMode() {
+    azureKeyVaultFactory.createAzureKeyVault(
+        "clientId",
+        "clientSecret",


signing/src/test/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactoryTest.java

+  void doesNotCreateExecutorWhenUsingSystemAssignedMode() {
+    azureKeyVaultFactory.createAzureKeyVault(
+        "clientId",
+        "clientSecret",


signing/src/test/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactoryTest.java

+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+
+class AzureKeyVaultFactoryTest {


siladu

Good test coverage 👍
Just some nits really.

CHANGELOG.md

...ests/src/test/java/tech/pegasys/web3signer/tests/slashing/SlashingPruningAcceptanceTest.java

commandline/src/main/java/tech/pegasys/web3signer/commandline/subcommands/ModeSubCommand.java

core/src/main/java/tech/pegasys/web3signer/core/Eth1Runner.java

core/src/main/java/tech/pegasys/web3signer/core/Runner.java

signing/src/main/java/tech/pegasys/web3signer/signing/config/SignerLoader.java

signing/src/main/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactory.java

siladu · 2023-07-26T05:46:56Z

signing/src/main/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactory.java

+    return executorServiceCache.updateAndGet(
+        e ->
+            Objects.requireNonNullElseGet(
+                e, () -> Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors())));


nit/musing: could potentially simplify by using a ConcurrentHashMap, but with the downside of a redundant key name. Don't think it's worth it, was just an interesting pattern to review...

return executorServiceCache.computeIfAbsent( "key", e -> Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors()));

core/src/main/java/tech/pegasys/web3signer/core/Eth2Runner.java

signing/src/main/java/tech/pegasys/web3signer/signing/secp256k1/azure/AzureKeyVaultSigner.java

signing/src/test/java/tech/pegasys/web3signer/signing/config/AzureKeyVaultFactoryTest.java

+  void closeShutdownsExecutor() {
+    azureKeyVaultFactory.createAzureKeyVault(
+        "clientId",
+        "clientSecret",


...sts/src/test/java/tech/pegasys/web3signer/dsl/signer/runner/CmdLineParamsConfigFileImpl.java

...sts/src/test/java/tech/pegasys/web3signer/tests/bulkloading/AzureKeyVaultAcceptanceTest.java

core/src/main/java/tech/pegasys/web3signer/core/Eth1Runner.java

gfukushima

some nits, otherwise lgtm

signing/src/main/java/tech/pegasys/web3signer/signing/secp256k1/azure/AzureConfig.java

...g/src/test/java/tech/pegasys/web3signer/signing/secp256k1/azure/AzureKeyVaultSignerTest.java

This reverts commit 801e3bc.

siladu

The Runners are much more comprehensible now 👍

core/src/main/java/tech/pegasys/web3signer/core/Eth1Runner.java

jframe added 17 commits July 7, 2023 11:28

Azure secp bulk loading

8ed4052

Load keys in parallel using mapKeyProperties

c504fbf

AzureKeyVault tests

b397719

spotless

6960fff

remove forkjoin pool hack

a32247a

add additional eth1 secp tests to AzureKeyVaultAcceptanceTest

5858e75

add support for azure bulk loading using file configuration in AT DSL

be9e45c

separate tagged and non-tagged keys in azure vault AT

63795da

Merge remote-tracking branch 'upstream/master' into azure_secp_bulk_l…

cee246e

…oading

fix integration test

52f6da5

Merge remote-tracking branch 'upstream/master' into azure_secp_bulk_l…

ea18172

…oading

Remove AzureKeyVaultSignerFactory needToHash constructor as this only…

1d1d0ae

… used in the test. And remove multiple signing test as this doesn't add any value

Set the expected number of keys to be loaded to account for multi-lin…

297aa64

…e key in BLS mode

Merge remote-tracking branch 'upstream/master' into azure_secp_bulk_l…

a5e38ec

…oading

Update AzureKeyVaultTest to use keys configured in Azure test env

f387f00

Change all Azure code to use the AzureKeyVaultFactory and create an E…

fb28ee8

…xecutorService if doesn't already exist

Add shutdown hook for AzureKeyVaultFactory

be2f509

github-advanced-security bot found potential problems Jul 21, 2023

View reviewed changes

jframe added 6 commits July 21, 2023 16:06

Re-enable process runner

aac3092

Use separate key dirs for signers so that SlashingPruningAcceptanceTe…

b4937fb

…st works with AT ThreadRunner

Cleanup after review

e15c332

Merge remote-tracking branch 'upstream/master' into azure_secp_bulk_l…

7125afc

…oading

Additional tests for Azure key mapping for handling exceptions and nu…

8f7c28c

…ll values

AzureKeyVaultFactory tests

dba365e

jframe added the doc-change-required Indicates an issue or PR that requires doc to be updated label Jul 24, 2023

github-advanced-security bot found potential problems Jul 24, 2023

View reviewed changes

changelog

17730b2

jframe marked this pull request as ready for review July 24, 2023 06:43

siladu approved these changes Jul 26, 2023

View reviewed changes

changes after group review

741c7b1

github-advanced-security bot found potential problems Jul 27, 2023

View reviewed changes

update dependencyCheck plugin

801e3bc

gfukushima reviewed Jul 27, 2023

View reviewed changes

signing/src/main/java/tech/pegasys/web3signer/signing/secp256k1/azure/AzureConfig.java Outdated Show resolved Hide resolved

...g/src/test/java/tech/pegasys/web3signer/signing/secp256k1/azure/AzureKeyVaultSignerTest.java Outdated Show resolved Hide resolved

jframe added 4 commits July 27, 2023 15:51

Merge branch 'master' into azure_secp_bulk_loading

7a9f1de

Revert "update dependencyCheck plugin"

027b4f1

This reverts commit 801e3bc.

After PR review

35c6dec

After PR review

e139f26

gfukushima approved these changes Jul 28, 2023

View reviewed changes

After PR review

5a655c4

siladu reviewed Jul 28, 2023

View reviewed changes

core/src/main/java/tech/pegasys/web3signer/core/Eth1Runner.java Outdated Show resolved Hide resolved

core/src/main/java/tech/pegasys/web3signer/core/Eth1Runner.java Outdated Show resolved Hide resolved

After PR review

8a1a964

jframe enabled auto-merge (squash) July 28, 2023 04:02

jframe merged commit 09ca080 into Consensys:master Jul 28, 2023

jframe mentioned this pull request Jul 28, 2023

feat: add eth1 azure bulk loading Consensys/doc.web3signer#194

Merged

jframe deleted the azure_secp_bulk_loading branch July 28, 2023 04:21

bgravenorst mentioned this pull request Aug 2, 2023

Add bulk loading of secp keys to the eth1 mode of Web3Signer. Consensys/doc.web3signer#195

Closed

bgravenorst removed the doc-change-required Indicates an issue or PR that requires doc to be updated label Aug 2, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure secp bulk loading #850

Azure secp bulk loading #850

jframe commented Jul 19, 2023 •

edited

Loading

siladu left a comment

siladu Jul 26, 2023

gfukushima left a comment

siladu left a comment

Azure secp bulk loading #850

Azure secp bulk loading #850

Conversation

jframe commented Jul 19, 2023 • edited Loading

PR Description

Fixed Issue(s)

Documentation

Changelog

Testing

siladu left a comment

Choose a reason for hiding this comment

siladu Jul 26, 2023

Choose a reason for hiding this comment

gfukushima left a comment

Choose a reason for hiding this comment

siladu left a comment

Choose a reason for hiding this comment

jframe commented Jul 19, 2023 •

edited

Loading