Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add proxy mode to RemoteClusterConnection #31840

Closed
s1monw opened this issue Jul 5, 2018 · 1 comment
Closed

Add proxy mode to RemoteClusterConnection #31840

s1monw opened this issue Jul 5, 2018 · 1 comment
Assignees
@Tim-Brooks @s1monw
Labels
:Distributed Coordination/Network Http and internode communication implementations

Comments

@s1monw
Copy link
Contributor

s1monw commented Jul 5, 2018

Today we assume we can use the internal structure of a remote cluster to make additional connections. Yet, this might not work if the remote cluster is inside a cloud-env or behind a proxy. We can't use a seed node and then sniff / discover from there using the cluster state of the remote cluster.

One way to solve this would be a proxy mode for remote connections that doesn't sniff the remote cluster at all but uses N connections to a proxy using dummy discovery nodes internally. Configuration wise this could look like this:

search:
    remote:
        cluster_one: 
            seeds: 127.0.0.1:9300
            proxy_mode: true
            num_connections: 10

This would translate to connecting to 10 dummy DiscoveryNodes using the same seed which can then round robin or whatever to connect to nodes. Each of these dummies would use a single channel profile. It's quite a different way of connecting but should work just fine for CCS/CCR purposes.

/cc @tbrooks8 @bleskes

Relates to #31835
@s1monw s1monw added :Distributed Coordination/Network Http and internode communication implementations team-discuss labels Jul 5, 2018
@s1monw s1monw self-assigned this Jul 5, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

s1monw added a commit to s1monw/elasticsearch that referenced this issue Aug 22, 2018
@s1monw
Add proxy support to RemoteClusterConnection
6f0cfb5 
This adds support for connecting to a remote cluster through
a tcp proxy. A remote cluster can configured with an additional
`search.remote.$clustername.proxy` setting. This proxy will be used
to connect to remote nodes for every node connection established.
We still try to sniff the remote clsuter and connect to nodes directly
through the proxy which has to support some kind of routing to these nodes.
Yet, this routing mechanism requires the handshake request to include some
kind of information where to route to which is not yet implemented. The effort
to use the hostname and an optional node attribute for routing is tracked
in elastic#32517

Closes elastic#31840
@s1monw s1monw mentioned this issue Aug 22, 2018
Merged
s1monw added a commit that referenced this issue Aug 25, 2018
@s1monw
Add proxy support to RemoteClusterConnection (#33062)
3376922 
This adds support for connecting to a remote cluster through
a tcp proxy. A remote cluster can configured with an additional
`search.remote.$clustername.proxy` setting. This proxy will be used
to connect to remote nodes for every node connection established.
We still try to sniff the remote clsuter and connect to nodes directly
through the proxy which has to support some kind of routing to these nodes.
Yet, this routing mechanism requires the handshake request to include some
kind of information where to route to which is not yet implemented. The effort
to use the hostname and an optional node attribute for routing is tracked
in #32517

Closes #31840
s1monw added a commit that referenced this issue Aug 25, 2018
@s1monw
Add proxy support to RemoteClusterConnection (#33062)
3b9965b 
This adds support for connecting to a remote cluster through
a tcp proxy. A remote cluster can configured with an additional
`search.remote.$clustername.proxy` setting. This proxy will be used
to connect to remote nodes for every node connection established.
We still try to sniff the remote clsuter and connect to nodes directly
through the proxy which has to support some kind of routing to these nodes.
Yet, this routing mechanism requires the handshake request to include some
kind of information where to route to which is not yet implemented. The effort
to use the hostname and an optional node attribute for routing is tracked
in #32517

Closes #31840
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Tim-Brooks Tim-Brooks

@s1monw s1monw

Labels
:Distributed Coordination/Network Http and internode communication implementations
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Tim-Brooks @s1monw @jasontedor @elasticmachine