[PoC] Efficient client management to support multiple OpenSearch clusters

[zhongnansu]

Problem Statement

As part of #1388, to support Dashboards connecting to multiple OpenSearch Cluster, the thread discussed adding a new saved object datasource, and referenced to the data model of index-pattern. There's also a proposed POC PR.

There's an issue with the POC code. When it establishes the connection with external datasource, it was using the @opensearch-js client in the search strategy.

OpenSearch-Dashboards/src/plugins/data/server/search/opensearch_search/ext_opensearch_search_strategy.ts

Lines 87 to 93 in beb46f5

	const client = new Client({
	node: url,
	auth: {
	username,
	password,
	}
	});

This is not resource efficient since it creates a connection for each and every query. We need better approach to do client management.

Proposed Solution [WIP]

We can take a look at how clients are managed in Dashboards core. The lifecycle starts when Dashboards-core plugin is spinning up, it creates 2 client and registers to the core context to expose to other modules through http services. It can spawn child clients as needed(AsScopedClient) from auth in request, which returns a new client instance that shares the connection pool with the parent client. If you call close in any of the parent/child clients, every client will be closed. The connections close when Dashboards stops.

We may consider adding one more client only for external data source, to the ClusterClient class. And wire our logic to spawn child clients as needed, from datasource. Similar to asScoped function.

OpenSearch-Dashboards/src/core/server/opensearch/client/cluster_client.ts

Lines 76 to 79 in 4469d17

	export class ClusterClient implements ICustomClusterClient {
	public readonly asInternalUser: Client;
	private readonly rootScopedClient: Client;

OpenSearch-Dashboards/src/core/server/opensearch/client/cluster_client.ts

Lines 91 to 98 in 4469d17

	asScoped(request: ScopeableRequest) {
	const scopedHeaders = this.getScopedHeaders(request);
	const scopedClient = this.rootScopedClient.child({
	headers: scopedHeaders,
	});
	return new ScopedClusterClient(this.asInternalUser, scopedClient);
	}

POC is needed and I am currently working on it

Changes need to make

• 1. Along with asCurrentUser and asInternalUser, define a new interface asDataSourceUser, for context.core.client to call
• 2. Create the third client in cluster_client only for data Source
• 3. Register above client to core context
• 4. import save_object client to the core Context, in order to retrieve data source meta info, that will be used to establish connection
• 5. adjust client config as needed(header, ssl). May need add new parsing logic specific for dataSourceClient
• *code style adjustment. Expose clean interfaces

[zhongnansu]

Results

My poc code PR zengyan-amazon#2. it's based on the poc did by @zengyan-amazon (Issue #1388 / PR #1499), which have the data sources added and use plain Client for each request to data source.

However, step 5 is blocked.

5. adjust client config as needed(header, ssl). May need add new parsing logic specific for dataSourceClient

Basically the connection failed connecting to the test endpoint I give, instead routing the request to localhost. It seems that certain ConnectionPool options is not allowed to configure while creating client.child(), for example Connection. It was documented in Elastic

You can pass to the child every client option you would pass to a normal client, but the connection pool specific options (ssl, agent, pingTimeout, Connection, and resurrectStrategy).

• opensearch-js code reference for ConnectionPool options.
  https://github.com/opensearch-project/opensearch-js/blob/f62a8cd644e6c5c65ea66932fc4b22ea76a57a67/lib/pool/index.d.ts#L38-L45
• opensearch-js code reference for Connection options, auth and you can see auth and node options are there. But auth can actually be passed as base64 encoded field in header. However node(url) is not allowed. https://github.com/opensearch-project/opensearch-js/blob/f62a8cd644e6c5c65ea66932fc4b22ea76a57a67/lib/Connection.d.ts#L44-L54

Next Step(propose new solutions)

1. Implement a client pooling strategy that keeps a number of clients(e.g. 200), maintain LRU cache to manage. Within each client scope, we can still use .child() to handler different user on the same endpoint.
2. Spend some time exploring opensearch-js client lib, to see if by making minor change we can enable node(endpoint) option for child() instances.