Add support for security and other API requests

[dagoodma]

I am trying to figure out how to send security and saved_objects API requests via the /_dashboards/api/... routes. It looks like opensearch-js does not support _dashboards API requests. Is there a way to form custom requests to paths not currently implemented in opensearch-js? I would like to get access to routes like: /_dashboards/api/v1/configuration/rolesmapping

[dblock]

Hm, Dashboards is a separate product. @kavilla how do people do this?

[dagoodma]

@dblock @kavilla For anyone else trying to do this, I was able to use the Client and Transport layer to form a custom request to the dashboards API.

let client = await getClient()
let path = '/_dashboards/api/v1/configuration/rolesmapping'
let method = 'GET'

// build request object
const request = {
  method: method,
  path: path,
  body: null,
  querystring: ""
}
// do request
try {
    let resp = await client.transport.request(request)

    if (resp && resp.statusCode == 200) {
      console.log("Success!");
      console.log("Result: ", resp.body)
    } else {
      console.log("Unknown response: ", resp)
    }
} catch (e) {
  console.log("Got an error: ", e)
}

Where getClient() is my async function that returns a new Client by passing in an AWS Connector and the endpoint address.

[dagoodma]

PS. Some dashboard requests may need osd-xsrf: true header to be accepted by the server. Note that headers can be passed into Transport.request() via the options argument:

opts = {
  headers: {
    'osd-xsrf': true
  }
} 
  
let resp = await client.transport.request(request, opts)

@dblock Would it be possible to add these instructions for the opensearch dashboards API to the Example use section of README.md?

[dblock]

@dagoodma Yes, please contribute. I think we should expand both the README and the documentation (https://opensearch.org/docs/latest/clients/javascript/) with more examples and general cleanup.

[dagoodma]

@dblock Although I did get these basic sort of requests working, I was not able to handle importing of saved objects due to issues with sending multipart form data while using the AWS4 signing procedure. The opensearch API requires the file being imported to be sent as a stream and it won't accept a string or a buffer. But sending streams with the form-data library seems to be impossible while using AWS4 due to what looks like an ambiguity with calculating and signing the content-length header field. I have tried using many clients: opensearch-js, https, and axios, and they all struggle with this same issue. I'm out of ideas for now.

Does anyone have any experience with getting streams to work with AWS4? Maybe I'm asking this question in the wrong place, but if someone could advise me on how to make opensearch-js handle sending streams better (especially with AWS4) I would greatly appreciate it!

[dagoodma]

Shortly after I sent that last comment, I was miraculously able to successfully get a multipart form message sent to the API using axios, form-data, aws4-axios, @aws-sdk/credential-provider-node and concat-stream. For anyone else who might get stuck similarly, here's how I was able to import saved_objects:

const { defaultProvider } = require("@aws-sdk/credential-provider-node")
const fs = require('fs')
const FormData = require('form-data')
const axios = require('axios')
const aws4axios = require('aws4-axios')
const concat = require('concat-stream')


const setupAxiosAws4Intercepter = async () => {
  const credentials = await defaultProvider()();

  const interceptor = aws4axios.aws4Interceptor({
    region: 'us-east-2',
    service: 'es'
  }, credentials)

  axios.interceptors.request.use(interceptor)
}

const endpoint = 'myopensearch-dashboard.us-east-2.es.amazonaws.com'
const path = '/_dashboards/api/saved_objects/_import'
const form = new FormData();
form.append('file', fs.createReadStream("my_saved_object.ndjson", "utf8"))

params = {
  url: 'https://' + endpoint + path
  headers: {
    'host': endpoint,
    'osd-xsrf': true,
    'accept': '*/*',
    'user-agent': 'opensearch-js/2.0.0 (linux 5.17.5-76051705-generic-x64; Node.js v14.17.0)',
    'osd-version': '1.2.0',
    ...form.getHeaders() // adds content-type for multipart form
  }
}

await setupAxiosAws4Intercepter()
console.log("Starting new request: ", params)
form.pipe(concat({ encoding: 'buffer' }, data => {
  axios.post(params.url, data, {headers: params.headers})
  .then(function (response) {
    if (response.status >= 200 && response.status < 299) {
      console.log(`Success ${response.status}! ${response.statusText}`)
      console.log(response.data)
    } else {
      console.log(`Endpoint returned error response (${response.status}): ${response.statusText}`)
      console.log(response)
    }
  })
  .catch(function(err) {
    console.log("Error while sending request: ", err)
  })
}))

I have abandoned using opensearch-js for now until full API support is added that includes the dashboards and security plugins. If there are plans to make that happen one day, then I will consider contributing to the project.

Feel free to close this issue for now. But hopefully this will save someone else a lot of time.

[wbeckler]

@dagoodma Thank you for sharing that solution.

I'm happy to say that there are plans to fully support security in opensearch-js in the near term.

Would you be willing to clarify what you mean by full API support for dashboards? Is it a matter of documentation or are there missing features?

[kavilla]

Hm, Dashboards is a separate product. @kavilla how do people do this?

Sorry about the delay, this is correct. The API this issue is trying to hit is an OpenSearch Dashboards API which isn't the intention of this repo. This repo's intention is to interact with the OpenSearch cluster with APIs via JavaScript as indicated by the README.

In this situation it seems like there's not much in the OpenSearch JS is doing as any HTTP Client is able to accomplish this while the example is using the transport layer to make the request. Note: _dashboards is cloud provider set configuration as OpenSearch Dashboards doesn't not come out of the box with the base route. I believe this issue can be closed as this client shouldn't be used to interact with OpenSearch Dashboards APIs. But I would like to understand @dagoodma's goal here. This API can be hit with a curl command but it's also recommended that since this is built out for OpenSearch Dashboards that you let OpenSearch Dashboards do the heavy lifting because it wasn't built with this intent so some best practices of an API might not be there for users to build a full client around it.

[dblock]

@dagoodma thanks for hanging on here and for your working samples!

I second what @kavilla says below, but also thought I'd clarify a couple more things. We have separate topics here:

1. For Sigv4 signing we want first class support (not an interceptor that users have to roll out), see [FEATURE] Add support for AWS Sigv4 request signer #252
2. For multipart, that works today, and for Sigv4 signing and multipart see (1) + you have worked around it as above, thank you for the sample!
3. For to as which APIs belong in this library, this is the issue!

The opensearch-js library will expose all APIs in OpenSearch (http://github.com/opensearch-project/OpenSearch), and plugins (e.g. http://github.com/opensearch-project/alerting which exposes APIs to create monitors). For now those two ship together, so it makes sense to just include support for those APIs, similar to other clients.

When it comes to the Amazon OpenSearch Service (_dashboards), that would belong in the AWS SDK, see https://docs.aws.amazon.com/opensearch-service/latest/developerguide/dashboards.html.

I know it may seem confusing: why does one need to use 2 SDKs when they use the Amazon OpenSearch Managed Service (a control plane for OpenSearch and OpenSearch Dashboards)? That's a great question! I do think that maybe the AWS SDK could include all the OpenSearch and OpenSearch Dashboards APIs (I certainly will bring this up with those folks) in it. However vice-versa we wouldn't want to include all the service APIs into this client because that's very vendor specific, and there's no boundary: if I need to create an IAM user for OpenSearch, why shouldn't I have that API in this client, then?

WDYT?

[nhtruong]

Closing this since this is a Dashboard issue and if the user needs to perform a custom request, they can now use the client.http namespace.