[PROPOSAL] OpenSearch JS Version 3.0 and OpenSearch Typescript

[nhtruong]

As a part of our effort to generate all clients from the OpenSearch API Spec, and modernize the Node.js client, we are planning some major updates for this repo.

OpenSearch JS Version 3.0

The next major version of this client is going to be fully generated from the API Spec. The vast majority of the current client is hand-written with a lot of inconsistency in the implementations of the less known and less used features:

• Support of both camelCased and snake_cased query string parameters
• Support of both camelCased and snake_cased api method names
• Overriding the HTTP Method of an API Function

Said features are going away in 3.0 to promote a cleaner and more efficient code base:

• The accepted query string parameters will now be the same as what the OpenSearch cluster expects, which is in snake_case. The client will no longer loop through the list of parameters passed to an API method to convert the camelCased keys into snake_case keys every time an API function supporting this feature is invoked.
• The client will only support camelCased API module and function names as this is a JavaScript convention.
• The client will not allow the HTTP method of an API function to be overridden by passing the method param to the function. That means, usages of this feature will result in error from the server as the method param will be treated as a query string parameter.

Version 3.0 will also introduce other improvements:

• Lazy loading of API modules and functions: Instead of loading every API function as soon as this node package is required, those functions will be loaded on the fly as they are invoked for the first time.
• More readable and maintainable file and folder structure:
  • Each API function will now be in their own file: Currently, only functions of the core API namespace are in their own files, while functions of every other namespaces are all written in the same giant .js file.
  • API functions will be divided into folders: Instead of having all API function and module files in one flat folder, the API function files mentioned above will be put into their respective namespace folders.

OpenSearch Typescript

Once OpenSearch JS 3.0 is released, we will start rewriting this client in Typescript in a different repo. This rewrite will carry over all improvements introduced in OpenSearch JS 3.0, along with some other changes to modernize and streamline the client:

• Dropping support for CommonJS in favor of ES6
• Instead of callback, the API functions will be written using async/await syntax.
• [Tentative] You will also no longer be able to pass the transport options to every API function call. This is a niche feature that can be replaced by instantiating multiple client instances with different transport options. A better option is adding the ability to modify a client instance's transport options before an API call.
• A complete rewrite of the transport layer to get rid of many duct-tape solutions in the current JS client.
• More streamlined integration of AWS Sigv4

The Typescript client will be a completely different client, and we're aiming to do it properly with ample research. OpenSearch JS 3.x will be constantly be updated when OpenSearch Typescript is under construction, and it will continue receive API updates long after OpenSearch Typescript is released to give the user time to switch over.

If you have any concern or suggestion, feel free to comment below.

[dblock]

I love the end goal: an opensearch-js to be rewritten in TypeScript, and to use a code generator.

Once OpenSearch JS 3.0 is released, we will start rewriting this client in Typescript in a different repo.

If I understand correctly the TypeScript client will be completely compatible with opensearch-js 3.0, so it should be a branch and eventually be merged to main as a replacement, then we would be shipping a non-breaking opensearch-js 3.x, but rewritten in TypeScript.

If this is not possible, I don't see why we need opensearch-js 3.0 at all. It's a breaking change for users, then they would have another breaking change switching to a TypeScript client, so why cause the pain twice?

• The accepted query string parameters will now be the same as what the OpenSearch cluster expects, which is in snake_case.

I wonder whether we can implement an optional compatibility mode for this that users would inject in the client to preserve backwards compat.

[nhtruong]

If this is not possible, I don't see why we need opensearch-js 3.0 at all. It's a breaking change for users, then they would have another breaking change switching to a Typescript client, so why cause the pain twice?

The breaking changes in 3.0 will only affect a few applications that use those very obscured features that are implemented sporadically (not all API functions support them):

• Per Javascript convention, module and function names are camelCased. So, most JS apps will not use the snake_cased version.
• Our JS docs only mention snake_cased query string params. The fact that some functions accept the camelCased params is a hidden feature.
• Overriding HTTP method is not documented either, and it's not at all useful (if the user needs to make custom requests not provided by our API functions, they should use client.http instead).

So, the switch to 3.0 will be straight forward for most users.

The Typescript client, however, will affect all users as we're also rewriting the transport layer. The options you provide during client instantiation will be different. Legacy apps will also be affected by the use of modern ES6 and async/await syntax. The API function calls will remain the same (creating an index named book will still be client.indices.create({ index: 'books' }) unless the user made use of legacy callbacks in their app. The switch from 3.x to TS will also only happen in very few places for modern apps.

Having both 3.x and TS clients existing in parallel affords the user some time to modernize their apps. With 3.x released, we can also spend more time to build the TS client (and do it right) as the API updates will be done completely via a client generator.

I wonder whether we can implement an optional compatibility mode for this that users would inject in the client to preserve backwards compat.

It's possible, but it's not straight forward. This feature is also a performance issue since it loops through the list of params to reconstruct the querystring params object for every API call. We will also run into a complication with the notifications namespace where query.string.names.like.this are introduced. It's a lot of work to preserve a lesser used feature that's actually causing more harm than good.

[dblock]

Having both 3.x and TS clients existing in parallel affords the user some time to modernize their apps.

This can be achieved by shipping older versions from a 2.x branch. I think it's a small but important nuance whether we use two separate repos or not.

[nhtruong]

I think it's a small but important nuance whether we use two separate repos or not.

Our release pipeline always uses what's in the main branch. So, having 2 different release configs in the same repo is not possible unfortunately. Moreover, since the TS client is a complete rewrite, it will have its own list of issues, and having these issues in a separate repo is easier to manage.

[nhtruong]

I've just uploaded the JS Client 3.0 Preview into this feature branch. It was completely generated from the OpenSearch API Spec, and it is fully functional (The types are still from 2.x. Working on generating the types from the spec next). It has all the changes and improvements mentioned in the OP. I also refactored a lot of the code:

• Old index.js vs New index.js
• Old CatApi vs New CatApi
• Old snapshot.clone vs New snapshot.clone

[nhtruong]

Another breaking change we should consider for 3.0 is ending support for older Node.js versions. The support of old Node.js (We current support Node 10 and up) is preventing us from upgrading dependency packages with tsd as the newest example, and it also poses a security risk. (Node 16 stopped receiving security updates on Sep 11, 2023)

[dblock]

I'd like support for Node.js 0.4. 🤡

[nhtruong]

Just had a discussion with @timursaurus on the TS client:

• The transport layer, which only receives a handful of update a year, can be its own Node package to reduce wasted bandwidth for the user when most of the client updates happen in the API. Though do note that the transport layer is very small comparing to the API, and the size gap is only growing as more API functions are being added. So, the bandwidth saving might not be worth the extra complexity. We need do to more research into this topic.
• Potentially make the TS client backward compatible with CommonJS with a transpiler (This is of a lower priority)
• See if we can avoid using 3rd-party packages, like axios, to make HTTP requests, and use native Node packages instead. Overall reduce the number of dependencies for the new clients.
• Only support AWS SDK 3 for Sigv4.

@timursaurus is going to research the current transport layer to see what business logic should be carried over to the new client. He'll work on a POC and publish his findings for discussion in a different issue in this repo. I'll modify the API generator (which is still WIP) for JS 3.0 to build the API layer for the TS client.

[u873838]

A goal for 3.X should be allowing in-flight requests to be cancelled using an AbortSignal rather than an abort method on the returned Promise.

[nhtruong]

Updates on type generation:
We've gotten through the hardest part of type generation. Check out the generated type files in the ./types folder. I've divided types into 2 folders:

• namespace type files: Each file contains the input type (Request) and output type (Response) of an API function.
• component type files: Each file contains a collection of component types of a certain category.

Similar to the main code, in 3.0, types are also divided into more readable files instead of a handful of giant files.

Remaining work regarding type generation:

• Handle edge cases like RequestProcessor
• Handle bugs like when tasks.list's group_by query type is explicitly defined instead of referencing GroupBy type.
• Generate Client type (This is the type that actually ties everything together)

[nhtruong]

Happy to announce that we're ready for 3.0.0.beta release on Monday (US Time). Check out the api_3.0 branch if you want to test it out yourself. Note that instead of

const { Client } = require('@opensearch-project/opensearch');

you will need to use

const { Client } = require('./index'); // be mindful of where your test file is relatively to the `index.js` file in the root of the branch

to load the source package. The Client class then can be used nearly as identical as you did in 2.X minus some breaking changes already discussed above.

On top of the issues mentioned in the previous update, the typing for the API have been completely overhauled. There's now only 1 type definition for Client. This will be a huge quality-of-life (QoL) change from 2.X where we had 3 different type definitions for Client. The generated Client type also now correctly reflects whether params is required for each API function, among other QoL updates I added while working on the overhaul.

Would love to have some input from the dashboard team as well especially after the beta version is released.
@AMoo-Miki

[dblock]

@nhtruong Excited to try it! Can we do 3.0.0.beta1 or will the next beta update that tag? I am sure it won't be the first and last beta ;)

[nhtruong]

3.0.0-beta.1 has been released. Looking forward to hearing back from you!

[dblock]

I tried the version in https://github.com/dblock/opensearch-node-client-demo and it worked well for the basic search scenario.

The issues I'm seeing.

1. This is also a problem in 2.x, but in VSCode I don't get auto-completion/types using const { Client } = require('@opensearch-project/opensearch'), but I do get some types using import { Client } from '@opensearch-project/opensearch'.
2. Some response types are missing, so var info = await client.info() is a generic HTTP response with a body. I'd like it to be a strongly typed OpenSearchVersionInfo.

[nhtruong]

Just tested this myself on my IDE (JetBrains WebStorm). Typing works on both 2.X and 3x, and on both ts and js files. The main difference is the 3.X and Typescript combo provides the best code hints and autocomplete:

• 3.x and CommonJS: The IDE handles the requests accurately, but autocomplete for the response body is the combination of all properties of all response bodies defined in 3.X even though it accurately determines the type of the response body (Seems like a bug in the IDE).
• 3.x and Typescript: The IDE handles both the requests and responses accurately.
• 2.x (both js and ts): The IDE handles both the requests accurately (The types are of much lower resolution than 3.x), but all response bodies are the generic Record<string, any>, which is sometimes incorrect.

I've taken some screenshots to compare between 3.x and 2.x (both in Typescript) for this code snippet:

import { Client as Client3 } from 'opensearch_3'; // 3.0.0-beta.1
import { Client as Client2 } from 'opensearch_2'; // 2.11.0

// Uncomment 2 lines below for CommonJS
// const Client3 = require('opensearch_3').Client; // 3.0.0-beta.1
// const Client2 = require('opensearch_2').Client; // 2.11.0

const clientOptions = { node: 'http://localhost:9200' } // simplified
const client3 = new Client3(clientOptions);
const client2 = new Client2(clientOptions);

const start = async function() {
  const nodes_info_3 = (await client3.nodes.info({ metric: [] })).body;
  const search_3 = (await client3.search({})).body;
  const cat_indices_3 = (await client3.cat.indices({ format: 'json' })).body;
  console.log(nodes_info_3);
  console.log(search_3);
  console.log(cat_indices_3);

  const nodes_info_2 = (await client2.nodes.info({ metric: [], })).body;
  const search_2 = (await client2.search({})).body;
  const cat_indices_2 = (await client2.cat.indices({ format: 'json' })).body;
  console.log(nodes_info_2);
  console.log(search_2);
  console.log(cat_indices_2);
}
start();

Both 3.x and 2.x have good coverage for API function params:

---

However, types in 3.x has much higher resolutions. When you use client.nodes.info(), the IDE helps you fill out the metrics supported by the endpoint in 3.x:

but not in 2.x

---

The response bodies in 3.x are also of much higher resolution and more accurate. Response body for search():

Notice how in 3.x above you get all the properties of a search response body, while in 2.x below, it's just a generic Object.

---

Each cat endpoint (when format: 'json' is used) returns an array of objects, however 2.x assumes it's Record<string, any> like all other response bodies,

while 3.x correct determines that it's it's an array:

and the data type of the items are of a very high resolution:

[dblock]

I am with you - VScode is having a separate issue that hasn't changed with 3.x, but we should fix it anyway. #839

[nhtruong]

@dblock The use of generic prevents VScode from properly determine the actual body of a response. While JetBrains can properly determine the response types, its Go to -> Declaration feature also points you to the generic definition instead of the actual response body. I'm going to get rid of generic in beta.2.

While I'm at it, I'm also going to restructure the request/response type definitions so that they are importable, and have more meaningful names instead of Response or ResponseBase like what we see below

[nhtruong]

Releasing beta.2 where the response types are no longer generic parameters for ApiResponse. Instead they are now their own types that extend ApiResponse. This allows Webstorm to correctly Go to -> Declaration.

However, in CommonJS (i.e. using require instead of import), the IDE still can't properly provide correct autocomplete for response body objects: It still shows all possible response properties. All response type declarations, although placed in different modules, still share the same name. I speculated that the IDE assumes a response to be any of the declaration for autocomplete and combines of their properties together in CommonJS (EVEN THOUGH its Go to -> Declaration can pinpoint the correct delcaration). This does not happen in Typescript files thankfully. Manually renaming a tested response declaration to something unique resolved this for Webstorm.

Goals for beta.3:

• Restructure Request/Response type declaration files to allow the user to import them (this is a big lift)
• Give Request/Response type declarations globally unique names

[dblock]

Beta 2 looks better with import!

[dblock]

I had search written like so, this doesn't work anymore because of director: 'miller' (director is a field in the index mapping).

Removing it will resolve.

    var results = await client.search({ body: { query: { match: { } } } })

[nhtruong]

@dblock found the issue in the generator. The fix will be in beta.3

[nhtruong]

releasing beta.3 where all Request and Response objects have unique names, and the fix for this. It resolves the ambiguity for WebStorm when determining which Request and Response objects to used for autocomplete in CommonJS files. @dblock, let me know if you find any issues with VSCode in beta.3

---

There's one last QoL change that I'm working on in beta.4. In beta.3, if you want to import the request types for your IDE to enforce type checking, you have to write out the api name twice (in the imported object name and the file to import) and have to import these objects one by one:

import type { Search_RequestBody } from 'opensearch_3/types/functions/search';
import type { Nodes_Info_Request } from 'opensearch_3/types/functions/nodes.info';
import type { CreatePit_Request} from 'opensearch_3/types/functions/create_pit';

...

const createPit_params: CreatePit_Request = { allow_partial_pit_creation: true, index: [movies] }
const createPit = await client.createPit(createPit_params)
console.log(createPit.body.pit_id);

const nodes_info_params: Nodes_Info_Request = { metric: ['jvm'] }
let nodes_info = await client.nodes.info(nodes_info_params);
console.log(nodes_info.body.nodes[0].jvm.mem.direct_max_in_bytes);

const search_body: Search_RequestBody = { query: { match: { director: 'miller' } } };
const search = await client.search({ index: movies, body: search_body });
console.log(search.body.hits.hits[0]._source);

In beta.4 I'm aiming for the below, which is similar to what 2.x is doing:

import type * as API from 'opensearch_3/types/api';

...

const createPit_params: API.CreatePit_Request = { allow_partial_pit_creation: true, index: [movies] }
const createPit = await client.createPit(createPit_params)
console.log(createPit.body.pit_id);

const nodes_info_params: API.Nodes_Info_Request = { metric: ['jvm'] }
let nodes_info = await client.nodes.info(nodes_info_params);
console.log(nodes_info.body.nodes[0].jvm.mem.direct_max_in_bytes);

const search_body: API.Search_RequestBody = { query: { match: { director: 'miller' } } };
const search = await client.search({ index: movies, body: search_body });
console.log(search.body.hits.hits[0]._source);

---

I also found another issue in the spec while testing beta.3. The create_pit endpoint, for example, only accepts an array of index per the spec, even though it should also accept a string as well. This will become a breaking change for users passing in a string for the index param:

await client.createPit({ index: 'movies' }) // This will cause a typescript error
await client.createPit({ index: ['movies'] }) // This is the only accepted format

@Xtansia is this also a problem for the JAVA client?
@dblock should we hold off the 3.0.0 release until this issue is resolved in the spec?

[dblock]

releasing beta.3 where all Request and Response objects have unique names, and the fix for this. It resolves the ambiguity for WebStorm when determining which Request and Response objects to used for autocomplete in CommonJS files. @dblock, let me know if you find any issues with VSCode in beta.3

Works well with import, search and other APIs look nice. Doesn't with require, but I believe that wasn't expected?

[nhtruong]

Releasing beta.4:

• Importing Request and Response types has been streamlined:

import { Client, type API } from '@opensearch-project/opensearch'

const createPitParams: API.CreatePit_Request = { allow_partial_pit_creation: true, index: [movies] }
const nodesInfoParams: API.Nodes_Info_Request = { metric: ['jvm'] }
const searchBody: API.Search_RequestBody = { query: { match: { director: 'miller' } } }

• Each api function .js file is now accompanied by a .d.ts file containing relevant types for that function. For example, the knn namespace folder now has the following files:

  • _api.js
  • deleteModel.d.ts
  • deleteModel.js
  • getModel.d.ts
  • getModel.js
  • searchModels.d.ts
  • searchModels.js
  • stats.d.ts
  • stats.js
  • trainModel.d.ts
  • trainModel.js
  • warmup.d.ts
  • warmup.js

• Another improvement over 2.x: OpensearchAPI and Client are now in their own .js files with matching .d.ts files. OpensearchAPI is now defined as a class, matching its declared type, instead of a constructor function. Overall, the file/folder structure of 3.0.0 is a lot more coherent than 2.x

[andreafspeziale]

Hello and sorry to jump in with a kind of dumb question/point... I'm wrapping this client in a package for NestJS applications and I'm exporting all your types for convenience. I couldn't find any way to type search responses:

const s = (await this.osClient.search(q)).body.hits.hits.map(
      (el) => el._source,
    );

el._source would be of type Hit._source?: Record<string, any> | undefined

Are you planning to use generics in the new codebase in order to enable such use-case? Or am I already missing something in the current implementation?

[nhtruong]

@andreafspeziale
Do you mean el._source is supposed to be the generic representing the doc structure of an index instead of Record<string, any> like in this?

Unfortunately what see now was generated directly from the OpenSearch API Spec. As of right now, the spec doesn't have any directive to specify that a certain property is a generic parameter. So instead of

const s = (await this.osClient.search<MyDocument>(q)).body.hits.hits.map(
  (el) => el._source,
);

Now you'll do

const s = (await this.osClient.search<MyDocument>(q)).body.hits.hits.map(
  (el) => el._source as MyDocument,
);

For clarification, the generics we got rid of in beta.2 was the body property of the ApiResponse interface. Instead, we define every response as an extension of ApiResponse to replace the body with the response body of the endpoint like in search for example.

[dblock]

As of right now, the spec doesn't have any directive to specify that a certain property is a generic parameter.

There aren't many of those, maybe we can keep a workaround in the generator?

[nhtruong]

There aren't many of those, maybe we can keep a workaround in the generator?

We could but It won't be a quick fix unfortunately. The generic is actually nested in several layers:
client.search<T> -> Search_Response<T> -> body<T> -> hits<T> -> hits<T> -> _source: T. In other words, _source being generic bubbles up all the way to the API function signature. We'll need to rework the JSON Schemas -> Typescript Types component, and add the Generic feature to the client signature renderer too.

If we're to do this, I'd say we should put this info in the spec instead of hardcoding it in the generator because of how complex the Generic chain is already to handle, and adding some logic to recognize the edge-case as a workaround will make things even more complex.

This brings up another question: Should we hold up 3.0.0 to work on this feature (It could take awhile), or save this for 3.1.0 because the user actually have a workaround to tell TS that _source is of a certain type.

[nhtruong]

@andreafspeziale Do you think the work around described here (el._source as MyDocument) is a reasonable workaround? How painful is it to do that in your application?

[andreafspeziale]

This brings up another question: Should we hold up 3.0.0 to work on this feature (It could take awhile), or save this for 3.1.0 because the user actually have a workaround to tell TS that _source is of a certain type.

@nhtruong I would save it for later.

P.S: I moved to 3.0.0-beta.4, is the Client.d.ts broken? Can't see the client methods:

Those are all the options I can see, it seems I can't resolve OpenSearchAPI which comes from declare class Client extends OpenSearchAPI (Client.d.ts)

beta.2 and beta.3 are working properly:

[nhtruong]

Client extends OpensearchAPI which has all the API methods. However, the OpenSearchAPI files (both .js and .d.ts) are not included in beta.4 for some reason:

Even though they exist in the source code:

Not sure how that happened.

Edit: NVM found the issue. 🤦

[nhtruong]

Release beta.5:
Replacing .npmignore with package.json#files that includes: "api/", "lib/", "index.d.ts", "index.js", "index.mjs", "README.md", "LICENSE.txt" to:

• Fix the missing files issue in beta.4
• Reduce memory/bandwidth footprint for the package (since we've already put a lot of work into making all api modules and functions lazy load 😀)

[nhtruong]

@andreafspeziale how did you use generic for search in 2.x? Currently we have 2 type declarations for Client in 2.x:

• in ./index.d.ts, which is the default but doesn't utilize TDocument generic (It has other generics for request and response)
• in ./api/new.d.ts where TDocument generic is actually defined.

So to actually use the client type declaration with TDocument generic in 2.11, it's rather convoluted:

import { Client } from '@opensearch-project/opensearch';
import { type Client as NewClient } from '@opensearch-project/opensearch/api/new';

const client = (new Client({...})) as unknown as NewClient; // have to coerce it to unknown first
const search = (await client.search<MyDocument>({...})).body;
const doc = search.hits.hits[0]._source; // Then TS will see doc as MyDocument

[andreafspeziale]

@nhtruong Sorry for the late response 😞 I'm not using 2.x in my projects, I'm already on ^3.0.0-beta.5.
You can check them out here:

• https://github.com/andreafspeziale/nestjs-search
• https://github.com/andreafspeziale/os-cli

(The second is leveraging the first)

Drop a ⭐ if you found them interesting! Have been developed with ❤️ in my free time

[nhtruong]

@andreafspeziale Wow those are great projects. And thanks for helping us test the beta.
We will release beta.7 soon with even more endpoints added from the spec.

[andreafspeziale]

No problem @nhtruong! Thx for your hard work people!