[BUG] creating Anomaly Detector using the API does not work

[zil1]

Describe the bug
The anomaly detection panel in opensearch dashboards fails after creating a detector via the API("Unable to get all detectors").
And it is also impossible to get/update/delete a detector created using the API.("Unable to get user information from detector")

To Reproduce
Steps to reproduce the behavior:

1. Create a detector using the create detector API, not via the plugin UI
2. Open Dashboards and navigate to the anomaly detection dashboards plugin UI

OpenSearch Dashboards logs:

Anomaly detector - Unable to search detectors { Error: [illegal_state_exception] Can't get text on a VALUE_NULL at 1:51
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (events.js:203:15)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  status: 500,
  displayName: 'InternalServerError',
  message:
   '[illegal_state_exception] Can\'t get text on a VALUE_NULL at 1:51',
  path: '/_plugins/_anomaly_detection/detectors/_search',
  query: {},
  body:
   { error:
      { root_cause: [Array],
        type: 'illegal_state_exception',
        reason: 'Can\'t get text on a VALUE_NULL at 1:51' },
     status: 500 },
  statusCode: 500,
  response:
   '{"error":{"root_cause":[{"type":"illegal_state_exception","reason":"Can\'t get text on a VALUE_NULL at 1:51"}],"type":"illegal_state_exception","reason":"Can\'t get text on a VALUE_NULL at 1:51"},"status":500}',
  toString: [Function],
  toJSON: [Function] }

OpenSearch logs(when trying to delete the detector):

[2022-01-17T16:14:43,040][ERROR][o.o.a.u.RestHandlerUtils ] [opensearch-node] Wrap exception before sending back to user
org.opensearch.ad.common.exception.AnomalyDetectionException: Unable to get user information from detector QXycaH4BI3aocvnW0ms2
	at org.opensearch.ad.util.ParseUtils.onGetAdResponse(ParseUtils.java:561) [opensearch-anomaly-detection-1.2.0.0-SNAPSHOT.jar:1.2.0.0-SNAPSHOT]
	at org.opensearch.ad.util.ParseUtils.lambda$getDetector$0(ParseUtils.java:517) [opensearch-anomaly-detection-1.2.0.0-SNAPSHOT.jar:1.2.0.0-SNAPSHOT]
	at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:103) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:97) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:287) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:273) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.0.0.jar:1.2.0.0]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1425) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1405) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:65) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:57) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:40) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:71) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:86) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.2.0.jar:1.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-01-17T16:14:43,042][WARN ][r.suppressed             ] [opensearch-node] path: /_plugins/_anomaly_detection/detectors/QXycaH4BI3aocvnW0ms2, params: {pretty=true, detectorID=QXycaH4BI3aocvnW0ms2}
org.opensearch.OpenSearchStatusException: Unable to get user information from detector QXycaH4BI3aocvnW0ms2
	at org.opensearch.ad.util.RestHandlerUtils.lambda$wrapRestActionListener$2(RestHandlerUtils.java:205) [opensearch-anomaly-detection-1.2.0.0-SNAPSHOT.jar:1.2.0.0-SNAPSHOT]
	at org.opensearch.action.ActionListener$1.onFailure(ActionListener.java:86) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.ad.util.ParseUtils.onGetAdResponse(ParseUtils.java:561) [opensearch-anomaly-detection-1.2.0.0-SNAPSHOT.jar:1.2.0.0-SNAPSHOT]
	at org.opensearch.ad.util.ParseUtils.lambda$getDetector$0(ParseUtils.java:517) [opensearch-anomaly-detection-1.2.0.0-SNAPSHOT.jar:1.2.0.0-SNAPSHOT]
	at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:78) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:103) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:97) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:287) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:273) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:302) [opensearch-security-1.2.0.0.jar:1.2.0.0]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1347) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1425) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1405) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:65) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:57) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:40) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:71) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:86) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792) [opensearch-1.2.0.jar:1.2.0]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [opensearch-1.2.0.jar:1.2.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

Expected behavior
Detectors that were successfully created and exist should be displayed in the UI.
Detectors are removed successfully using the API.

Plugins
OpenSearch Dashboards:

$ /usr/share/opensearch-dashboards/bin/opensearch-dashboards-plugin list
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

OpenSearch:

$ /usr/share/opensearch/bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-sql

Host/Environment (please complete the following information):

• OS: Docker based on official opensearchproject/opensearch:1.2.0 and opensearchproject/opensearch-dashboards:1.2.0 images, Centos7
• Version 1.2.0

Additional context
Installing the AD plugin 1.2.0.1 did not help

[amitgalitz]

Hi @zil1, thank you for taking the time to report this issue to us! However, I haven’t been able to replicate the issue you specify here. I followed the steps you outlined with the same environment (also trying to replicate it with AD 1.2.0.1). I was wondering if you could possibly provide additional insight to the request payload you have for creating the detector. Also, any additional network response information could be helpful along with the logs you provided above. Another thing that could possibly help as I try to replicate this issue, is if you have the exact docker commands, docker-compose file or any other steps you followed to initially set up the environment. Thank you!

[zil1]

we didn't have any important detectors and historical data, so we just deleted the system indexes .opendistro-anomaly*. After that, we do not observe any problems. Thanks

[zil1]

I think I figured out how to repeat the bug. If I don't specify a "description" in the request body, then the detector is created, but the AD plugin breaks and gives an error: "Unable to get all detectors"

OpenSearch version 1.2.4

[ohltyler]

@zil1 thanks for providing more details, and apologize for the late response. I was able to reproduce this bug by not including a description field as well. But, from the API documentation, it does state that description is required.

However, the API should reject the request if the description field is missing, rather than creating a broken configuration. Let me open a separate issue to track this, and thanks for bringing this up.

Edit: I've opened a tracking issue here, with a few different options. Feel free to chime in your ideas there as well: opensearch-project/anomaly-detection#437

[ohltyler]

Closing in favor of the mentioned issue.