Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Merged by Bors] - Add listener that allows remote connections without mTLS #5418

Closed
wants to merge 28 commits into from

Conversation

fasmat
Copy link
Member

@fasmat fasmat commented Jan 9, 2024

Motivation

Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

Changes

  • add new grpcPostServer that allows remote connection to a post service without setting up mTLS
    • Post is now available on grpcPostServer (default: 0.0.0.0:9094) to allow exposing it without mTLS and
    • on grpcTLSServer (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
  • GRPC services are now true singletons - simplified code to instantiate and register in node.go during startup
  • updated GRPC PostService to allow connections from multiple post services. (prep. work for multiple smeshers)
    • nipostBuilder / atxBuilder can select the required client based on the node Id the post service reports after connecting
    • only one post service can connect per Node ID
  • Merged similar functionality of grpc.NewPublic and grpc.NewPrivate into one grpc.NewWithServices
    • function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
  • Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
    • By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
    • The node running the post service has an init container where postcli is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the smesherConfig that is deployed with every node and translated into parameters for postcli / post-service
    • systest Makefile has been updated with image names and tags for post-service and postcli
    • systest Dockerfile has been expanded to include libpost. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
  • Fixed a few minor issues with existing system tests
    • tests now generate the same number of keys as nodes are used during the test (1 per node)
    • AddBootnodes didn't keep track of the correct number of bootnodes after deployment

Test Plan

  • Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

TODO

  • Explain motivation or link existing issue(s)
  • Test changes and document test plan
  • Update documentation as needed
  • Update changelog as needed

@fasmat fasmat self-assigned this Jan 9, 2024
Copy link

codecov bot commented Jan 9, 2024

Codecov Report

Attention: 42 lines in your changes are missing coverage. Please review.

Comparison is base (4b0d356) 77.6% compared to head (b3eaede) 77.6%.

Files Patch % Lines
node/node.go 77.3% 15 Missing and 11 partials ⚠️
api/grpcserver/post_service.go 58.8% 8 Missing and 6 partials ⚠️
api/grpcserver/grpc.go 75.0% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##           develop   #5418     +/-   ##
=========================================
- Coverage     77.6%   77.6%   -0.1%     
=========================================
  Files          267     267             
  Lines        30976   31059     +83     
=========================================
+ Hits         24067   24109     +42     
- Misses        5393    5421     +28     
- Partials      1516    1529     +13     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@fasmat fasmat force-pushed the allow-remote-post-without-tls branch 2 times, most recently from e281bdd to 0110842 Compare January 11, 2024 13:30
@fasmat fasmat marked this pull request as ready for review January 11, 2024 15:47
@fasmat fasmat force-pushed the allow-remote-post-without-tls branch 4 times, most recently from a130f9f to 2e17094 Compare January 15, 2024 16:07
api/grpcserver/config.go Show resolved Hide resolved
cmd/root.go Outdated Show resolved Hide resolved
@fasmat fasmat force-pushed the allow-remote-post-without-tls branch 2 times, most recently from 787fabb to 9c8fdac Compare January 17, 2024 09:49
spacemesh-bors bot pushed a commit that referenced this pull request Jan 22, 2024
## Motivation
Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

## Changes
- add new `grpcPostServer` that allows remote connection to a post service without setting up mTLS
  - Post is now available on `grpcPostServer` (default: 0.0.0.0:9094) to allow exposing it without mTLS and
  - on `grpcTLSServer` (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
- GRPC services are now true singletons - simplified code to instantiate and register in `node.go` during startup
- updated GRPC `PostService` to allow connections from multiple post services. (prep. work for multiple smeshers)
  - `nipostBuilder` / `atxBuilder` can select the required client based on the node Id the post service reports after connecting
  - only one post service can connect per Node ID
- Merged similar functionality of `grpc.NewPublic` and `grpc.NewPrivate` into one `grpc.NewWithServices`
  - function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
- Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
  - By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
  - The node running the post service has an init container where `postcli` is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the `smesherConfig` that is deployed with every node and translated into parameters for `postcli` / `post-service`
  - systest Makefile has been updated with image names and tags for `post-service` and `postcli`
  - systest Dockerfile has been expanded to include `libpost`. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
- Fixed a few minor issues with existing system tests
  - tests now generate the same number of keys as nodes are used during the test (1 per node)
  - `AddBootnodes` didn't keep track of the correct number of bootnodes after deployment

## Test Plan
- Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [x] Update documentation as needed
- [x] Update [changelog](../CHANGELOG.md) as needed
@spacemesh-bors
Copy link

Build failed:

  • ci-status

@fasmat
Copy link
Member Author

fasmat commented Jan 22, 2024

bors merge

@spacemesh-bors
Copy link

Merge conflict.

@fasmat fasmat force-pushed the allow-remote-post-without-tls branch from 0d153e9 to 92a1eaa Compare January 22, 2024 13:11
@fasmat
Copy link
Member Author

fasmat commented Jan 22, 2024

bors merge

spacemesh-bors bot pushed a commit that referenced this pull request Jan 22, 2024
## Motivation
Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

## Changes
- add new `grpcPostServer` that allows remote connection to a post service without setting up mTLS
  - Post is now available on `grpcPostServer` (default: 0.0.0.0:9094) to allow exposing it without mTLS and
  - on `grpcTLSServer` (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
- GRPC services are now true singletons - simplified code to instantiate and register in `node.go` during startup
- updated GRPC `PostService` to allow connections from multiple post services. (prep. work for multiple smeshers)
  - `nipostBuilder` / `atxBuilder` can select the required client based on the node Id the post service reports after connecting
  - only one post service can connect per Node ID
- Merged similar functionality of `grpc.NewPublic` and `grpc.NewPrivate` into one `grpc.NewWithServices`
  - function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
- Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
  - By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
  - The node running the post service has an init container where `postcli` is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the `smesherConfig` that is deployed with every node and translated into parameters for `postcli` / `post-service`
  - systest Makefile has been updated with image names and tags for `post-service` and `postcli`
  - systest Dockerfile has been expanded to include `libpost`. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
- Fixed a few minor issues with existing system tests
  - tests now generate the same number of keys as nodes are used during the test (1 per node)
  - `AddBootnodes` didn't keep track of the correct number of bootnodes after deployment

## Test Plan
- Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [x] Update documentation as needed
- [x] Update [changelog](../CHANGELOG.md) as needed
@spacemesh-bors
Copy link

Build failed (retrying...):

spacemesh-bors bot pushed a commit that referenced this pull request Jan 22, 2024
## Motivation
Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

## Changes
- add new `grpcPostServer` that allows remote connection to a post service without setting up mTLS
  - Post is now available on `grpcPostServer` (default: 0.0.0.0:9094) to allow exposing it without mTLS and
  - on `grpcTLSServer` (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
- GRPC services are now true singletons - simplified code to instantiate and register in `node.go` during startup
- updated GRPC `PostService` to allow connections from multiple post services. (prep. work for multiple smeshers)
  - `nipostBuilder` / `atxBuilder` can select the required client based on the node Id the post service reports after connecting
  - only one post service can connect per Node ID
- Merged similar functionality of `grpc.NewPublic` and `grpc.NewPrivate` into one `grpc.NewWithServices`
  - function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
- Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
  - By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
  - The node running the post service has an init container where `postcli` is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the `smesherConfig` that is deployed with every node and translated into parameters for `postcli` / `post-service`
  - systest Makefile has been updated with image names and tags for `post-service` and `postcli`
  - systest Dockerfile has been expanded to include `libpost`. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
- Fixed a few minor issues with existing system tests
  - tests now generate the same number of keys as nodes are used during the test (1 per node)
  - `AddBootnodes` didn't keep track of the correct number of bootnodes after deployment

## Test Plan
- Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [x] Update documentation as needed
- [x] Update [changelog](../CHANGELOG.md) as needed
@spacemesh-bors
Copy link

Build failed:

@fasmat
Copy link
Member Author

fasmat commented Jan 22, 2024

grpc disconnects in TestPartition_30_70

bors merge

spacemesh-bors bot pushed a commit that referenced this pull request Jan 22, 2024
## Motivation
Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

## Changes
- add new `grpcPostServer` that allows remote connection to a post service without setting up mTLS
  - Post is now available on `grpcPostServer` (default: 0.0.0.0:9094) to allow exposing it without mTLS and
  - on `grpcTLSServer` (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
- GRPC services are now true singletons - simplified code to instantiate and register in `node.go` during startup
- updated GRPC `PostService` to allow connections from multiple post services. (prep. work for multiple smeshers)
  - `nipostBuilder` / `atxBuilder` can select the required client based on the node Id the post service reports after connecting
  - only one post service can connect per Node ID
- Merged similar functionality of `grpc.NewPublic` and `grpc.NewPrivate` into one `grpc.NewWithServices`
  - function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
- Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
  - By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
  - The node running the post service has an init container where `postcli` is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the `smesherConfig` that is deployed with every node and translated into parameters for `postcli` / `post-service`
  - systest Makefile has been updated with image names and tags for `post-service` and `postcli`
  - systest Dockerfile has been expanded to include `libpost`. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
- Fixed a few minor issues with existing system tests
  - tests now generate the same number of keys as nodes are used during the test (1 per node)
  - `AddBootnodes` didn't keep track of the correct number of bootnodes after deployment

## Test Plan
- Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [x] Update documentation as needed
- [x] Update [changelog](../CHANGELOG.md) as needed
@spacemesh-bors
Copy link

Build failed:

@fasmat
Copy link
Member Author

fasmat commented Jan 22, 2024

bors merge

spacemesh-bors bot pushed a commit that referenced this pull request Jan 22, 2024
## Motivation
Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

## Changes
- add new `grpcPostServer` that allows remote connection to a post service without setting up mTLS
  - Post is now available on `grpcPostServer` (default: 0.0.0.0:9094) to allow exposing it without mTLS and
  - on `grpcTLSServer` (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
- GRPC services are now true singletons - simplified code to instantiate and register in `node.go` during startup
- updated GRPC `PostService` to allow connections from multiple post services. (prep. work for multiple smeshers)
  - `nipostBuilder` / `atxBuilder` can select the required client based on the node Id the post service reports after connecting
  - only one post service can connect per Node ID
- Merged similar functionality of `grpc.NewPublic` and `grpc.NewPrivate` into one `grpc.NewWithServices`
  - function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
- Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
  - By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
  - The node running the post service has an init container where `postcli` is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the `smesherConfig` that is deployed with every node and translated into parameters for `postcli` / `post-service`
  - systest Makefile has been updated with image names and tags for `post-service` and `postcli`
  - systest Dockerfile has been expanded to include `libpost`. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
- Fixed a few minor issues with existing system tests
  - tests now generate the same number of keys as nodes are used during the test (1 per node)
  - `AddBootnodes` didn't keep track of the correct number of bootnodes after deployment

## Test Plan
- Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [x] Update documentation as needed
- [x] Update [changelog](../CHANGELOG.md) as needed
@spacemesh-bors
Copy link

Build failed:

@fasmat
Copy link
Member Author

fasmat commented Jan 22, 2024

bors merge

spacemesh-bors bot pushed a commit that referenced this pull request Jan 22, 2024
## Motivation
Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

## Changes
- add new `grpcPostServer` that allows remote connection to a post service without setting up mTLS
  - Post is now available on `grpcPostServer` (default: 0.0.0.0:9094) to allow exposing it without mTLS and
  - on `grpcTLSServer` (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
- GRPC services are now true singletons - simplified code to instantiate and register in `node.go` during startup
- updated GRPC `PostService` to allow connections from multiple post services. (prep. work for multiple smeshers)
  - `nipostBuilder` / `atxBuilder` can select the required client based on the node Id the post service reports after connecting
  - only one post service can connect per Node ID
- Merged similar functionality of `grpc.NewPublic` and `grpc.NewPrivate` into one `grpc.NewWithServices`
  - function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
- Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
  - By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
  - The node running the post service has an init container where `postcli` is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the `smesherConfig` that is deployed with every node and translated into parameters for `postcli` / `post-service`
  - systest Makefile has been updated with image names and tags for `post-service` and `postcli`
  - systest Dockerfile has been expanded to include `libpost`. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
- Fixed a few minor issues with existing system tests
  - tests now generate the same number of keys as nodes are used during the test (1 per node)
  - `AddBootnodes` didn't keep track of the correct number of bootnodes after deployment

## Test Plan
- Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [x] Update documentation as needed
- [x] Update [changelog](../CHANGELOG.md) as needed
@spacemesh-bors
Copy link

Build failed:

@fasmat
Copy link
Member Author

fasmat commented Jan 22, 2024

bors merge

spacemesh-bors bot pushed a commit that referenced this pull request Jan 22, 2024
## Motivation
Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

## Changes
- add new `grpcPostServer` that allows remote connection to a post service without setting up mTLS
  - Post is now available on `grpcPostServer` (default: 0.0.0.0:9094) to allow exposing it without mTLS and
  - on `grpcTLSServer` (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
- GRPC services are now true singletons - simplified code to instantiate and register in `node.go` during startup
- updated GRPC `PostService` to allow connections from multiple post services. (prep. work for multiple smeshers)
  - `nipostBuilder` / `atxBuilder` can select the required client based on the node Id the post service reports after connecting
  - only one post service can connect per Node ID
- Merged similar functionality of `grpc.NewPublic` and `grpc.NewPrivate` into one `grpc.NewWithServices`
  - function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
- Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
  - By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
  - The node running the post service has an init container where `postcli` is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the `smesherConfig` that is deployed with every node and translated into parameters for `postcli` / `post-service`
  - systest Makefile has been updated with image names and tags for `post-service` and `postcli`
  - systest Dockerfile has been expanded to include `libpost`. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
- Fixed a few minor issues with existing system tests
  - tests now generate the same number of keys as nodes are used during the test (1 per node)
  - `AddBootnodes` didn't keep track of the correct number of bootnodes after deployment

## Test Plan
- Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [x] Update documentation as needed
- [x] Update [changelog](../CHANGELOG.md) as needed
@spacemesh-bors
Copy link

Build failed:

@fasmat
Copy link
Member Author

fasmat commented Jan 23, 2024

bors merge

spacemesh-bors bot pushed a commit that referenced this pull request Jan 23, 2024
## Motivation
Adds a new listener to allow remote post services to connect if they are within the same private network without having to setup mTLS.

## Changes
- add new `grpcPostServer` that allows remote connection to a post service without setting up mTLS
  - Post is now available on `grpcPostServer` (default: 0.0.0.0:9094) to allow exposing it without mTLS and
  - on `grpcTLSServer` (no default listening address) to allow exposing over insecure networks (to allow transmitting keys securely between node and post service in the future)
- GRPC services are now true singletons - simplified code to instantiate and register in `node.go` during startup
- updated GRPC `PostService` to allow connections from multiple post services. (prep. work for multiple smeshers)
  - `nipostBuilder` / `atxBuilder` can select the required client based on the node Id the post service reports after connecting
  - only one post service can connect per Node ID
- Merged similar functionality of `grpc.NewPublic` and `grpc.NewPrivate` into one `grpc.NewWithServices`
  - function checks if listening IP is in a public network and prints a waring (displayed during startup) to ensure to be careful not to expose it over public networks
- Expanded system tests to automatically add nodes with remote smeshing setups to all tests.
  - By default 25% of all nodes will be deployed in a remote setup (i.e. with a separate pod running the post service remotely and connecting to the node)
  - The node running the post service has an init container where `postcli` is used to create the post data with the same setting as the node would use in a supervised setup. Those settings are picked from the `smesherConfig` that is deployed with every node and translated into parameters for `postcli` / `post-service`
  - systest Makefile has been updated with image names and tags for `post-service` and `postcli`
  - systest Dockerfile has been expanded to include `libpost`. This has unfortunately become necessary to parse the config used during system tests with the same code as we use during the startup of the node (to ensure compatibility with changing configs)
- Fixed a few minor issues with existing system tests
  - tests now generate the same number of keys as nodes are used during the test (1 per node)
  - `AddBootnodes` didn't keep track of the correct number of bootnodes after deployment

## Test Plan
- Extended existing system tests to deploy 1/4 of the nodes during the test with a remote post service.

## TODO
<!-- This section should be removed when all items are complete -->
- [x] Explain motivation or link existing issue(s)
- [x] Test changes and document test plan
- [x] Update documentation as needed
- [x] Update [changelog](../CHANGELOG.md) as needed
@spacemesh-bors
Copy link

Pull request successfully merged into develop.

Build succeeded:

@spacemesh-bors spacemesh-bors bot changed the title Add listener that allows remote connections without mTLS [Merged by Bors] - Add listener that allows remote connections without mTLS Jan 23, 2024
@spacemesh-bors spacemesh-bors bot closed this Jan 23, 2024
@spacemesh-bors spacemesh-bors bot deleted the allow-remote-post-without-tls branch January 23, 2024 02:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants