[FEATURE] Implement register model group for Model access control in os-py-ml

[dhrubo-os]

In 2.8, we introduces model access control in ml-commons. But we didn't extend this functionality in opensearch-py-ml yet.

We want to add register_model_group:

request parameters: https://opensearch.org/docs/latest/ml-commons-plugin/model-access-control/#request-fields

example request: https://opensearch.org/docs/latest/ml-commons-plugin/model-access-control/#example-request

example response: https://opensearch.org/docs/latest/ml-commons-plugin/model-access-control/#example-response

[AkshathRaghav]

@dhrubo-os Hi! I'm working on this right now on this branch .

[dhrubo-os]

Sounds good. I assigned this issue to you. Thanks.

[AkshathRaghav]

Hi @dhrubo-os. I've spent quite a while trying to get the dashboards set up, and finally got it working.

Following the DEVELOPER_GUIDE.md, I am trying to update cluster settings, but am getting the following error:

{
  "error": {
    "root_cause": [
      {
        "type": "settings_exception",
        "reason": "persistent setting [plugins.ml_commons.enable_inhouse_python_model], not recognized"
      }
    ],
    "type": "settings_exception",
    "reason": "persistent setting [plugins.ml_commons.enable_inhouse_python_model], not recognized"
  },
  "status": 400
}

Could you give me some directions on how to solve this issue?

I've set up opensearch and opensearch dashboards by cloning the repositories from Github. Could that be the issue?

[dhrubo-os]

1. Which OS version are you using?
2. https://opensearch.org/docs/2.9/ml-commons-plugin/api/ --> take a look here for registering model.

[AkshathRaghav]

1. I'm on Windows 11, and am building OpenSearch and OpenSearch-Dashboards locally (cloning and following DEVELOPER_GUIDE)
2. I tried those commands as well. It returns that it is unable to find the URI for method [POST]

[dhrubo-os]

Sorry for the confusion, by OS version I meant Opensearch version

How did you try those commands?

[AkshathRaghav]

I cloned the repo 2 days ago, and on that day github-actions had released Version 1.3.13. Do I consider that to be the version I am on? On the docs page I see it says OpenSearch 2.9.x is the latest..

And I tried those commands through Dev Tool on the Dashboards UI.

[dhrubo-os]

Did you setup Opensearch cluster in your end? If yes, which version is it?

[dhrubo-os]

Did you go through this: https://github.com/opensearch-project/opensearch-py-ml/blob/main/DEVELOPER_GUIDE.md

[AkshathRaghav]

I had previously not set it up, sorry for the confusion (I had completed the tutorial from the OSCI Workshop)
In response to which version -> I am doing 2.9.0

I went through the steps outlined in (https://opensearch.org/docs/2.5/install-and-configure/install-opensearch/tar/), and am stuck when I am trying to apply my config changes. (step: https://opensearch.org/docs/2.5/install-and-configure/install-opensearch/tar/#apply-changes)
Error:

Populate config from /mnt/c/AA-Codebench/aws/opensearch-2.9.0/config/opensearch-security/
Will update '/config' with /mnt/c/AA-Codebench/aws/opensearch-2.9.0/config/opensearch-security/config.yml
   FAIL: Configuration for 'config' failed because of OpenSearchStatusException[OpenSearch exception [type=cluster_block_exception, reason=index [.opendistro_security] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]
Will update '/roles' with /mnt/c/AA-Codebench/aws/opensearch-2.9.0/config/opensearch-security/roles.yml
   FAIL: Configuration for 'roles' failed because of OpenSearchStatusException[OpenSearch exception [type=cluster_block_exception, reason=index [.opendistro_security] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]
Will update '/rolesmapping' with /mnt/c/AA-Codebench/aws/opensearch-2.9.0/config/opensearch-security/roles_mapping.yml
   FAIL: Configuration for 'rolesmapping' failed because of OpenSearchStatusException[OpenSearch exception [type=cluster_block_exception, reason=index [.opendistro_security] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]
Will update '/internalusers' with /mnt/c/AA-Codebench/aws/opensearch-2.9.0/config/opensearch-security/internal_users.yml
   FAIL: Configuration for 'internalusers' failed because of OpenSearchStatusException[OpenSearch exception [type=cluster_block_exception, reason=index [.opendistro_security] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]
Will update '/actiongroups' with /mnt/c/AA-Codebench/aws/opensearch-2.9.0/config/opensearch-security/action_groups.yml
   FAIL: Configuration for 'actiongroups' failed because of OpenSearchStatusException[OpenSearch exception [type=cluster_block_exception, reason=index [.opendistro_security] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]]

I'm on Windows 11 and I tried to change Docker Desktops memory limits in my .wslconfig file (im using WSL2) with the following contents:

[wsl2]
memory=10GB

This shows me that I have a lot of free memory but the 'percentage' of it is low, thus leading it to throw that error

[2023-09-25T13:09:11,623][WARN ][o.o.c.r.a.DiskThresholdMonitor] [Akshaths-PC] flood stage disk watermark [95%] exceeded on [ctE1JnHbQBWbSrfn3K3oqQ][Akshaths-PC][/mnt/c/AA-Codebench/aws/opensearch-2.9.0/data/nodes/0] free: 17.5gb[4.2%], all indices on this node will be marked read-only

Please let me know if you have faced this before and what I could do. Thank you in advance @dhrubo-os :)

[dhrubo-os]

I believe the message clearly indicates that your free storage is free: 17.5gb[4.2%] which is not the correct expectation. In this case, the watermark is set at 95%, meaning that when your disk usage reaches 95% or more, this warning is triggered.

So you need to free up disk space or add more storage.

[dhrubo-os]

@AkshathRaghav Do you have any updates on this issue? Has the issue been resolved?

[rawwar]

@dhrubo-os , can you please assign me this ticket. I will work on this.

[dhrubo-os]

@AkshathRaghav I'm sorry. Due to the priority, I'm assigning this issue to @rawwar. Please let me know if you still want to work on any other issue. Thanks.

[AkshathRaghav]

@dhrubo-os Hello sir, I apologize for missing the original message. I've been working on benchmark and wasn't able to make more progress on this issue.

https://github.com/AkshathRaghav/opensearch-py-ml/blob/model_group_register/AkshathRaghav/opensearch_py_ml/ml_commons/ml_commons_client.py <- I had made some changes in my fork but was not able to finish testing it at the time because I was having cluster issues.

Once OSCI is over, I will try to work on other issues on a proper timeline. Thank you for your patience!