Kibana should avoid using .watches index directly

[masseyke]

In Elasticsearch 8.0, direct access to system indices was intentionally made much more difficult by Elasticsearch. Kibana uses system indices (.watches and .watcher_history) for the "Watcher" part of Stack Management. It looks like it makes a request to _security/user/has_privileges for the .watches and .watcher_history system indices, and only shows the Watcher link if the user has read access to them. And then it does a search request directly on the .watches index. The problem is that, due to the changes in Elasticsearch in 8.0, users have to add that read privilege to their role using the "allow_restricted_indices" flag like this:

curl -k -u elastic:password -X POST "localhost:9200/_security/role/my_watcher_admin_role?pretty" -H 'Content-Type: application/json' -d'
{
  "indices": [
    {
      "names": [ ".watches", ".watcher_history" ],
      "privileges": ["read"],
      "allow_restricted_indices": true
    }
  ]
}
'

But I don't think that Kibana will let you use that flag (I could be wrong).
I think this can now be done without accessing system indices at all. There is now an API for fetching all watches (elastic/elasticsearch#64582) that does not require permission to read any system indices directly. If it can't be done without direct access to those indices then the @elastic/es-data-management team probably needs to add whatever is missing.

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[alisonelizabeth]

@ElenaStoeva can you look into the level of effort to address this one?

[ElenaStoeva]

@ElenaStoeva can you look into the level of effort to address this one?

It seems that the Watcher fetches watches by sending a search request on the .watches index in this function, which I think can easily be modified to use the Query Watch API instead as follows:

function fetchWatches(dataClient: IScopedClusterClient) {
  return dataClient.asCurrentUser.watcher.queryWatches();
}

After this, some additional changes are needed to process the response as the Json fields in this response are slightly different from what is returned from the search request to the .watches index.

What concerns me is the part where Watcher fetches a history item by its id. Here, it sends a search request to the .watcher_history index and, since we don't want this, I was thinking what Watcher API we can use instead (@masseyke do you have any suggestion?).
I considered using the Get watch API which we can use to retrieve a watch by its id, but I'm not sure if this is appropriate as the fetch function is supposed to return a response from which we can extract a WatchHistoryItem object. @alisonelizabeth, since I am not really familiar with watch history items and what we use them for, do you know how they are related to watches and whether we can convert information from a watch into a watch history item?

[alisonelizabeth]

Thanks @ElenaStoeva for looking into this.

It seems that the Watcher fetches watches by sending a search request on the .watches index in this function, which I think can easily be modified to use the Query Watch API instead as follows:

👍 let's move forward with this change.

What concerns me is the part where Watcher fetches a history item by its id. Here, it sends a search request to the .watcher_history index and, since we don't want this, I was thinking what Watcher API we can use instead (@masseyke do you have any suggestion?).

Yes, good point. I don't think we'll be able to replace this with the current set of ES APIs. Can you open up a separate issue to track this? @masseyke any guidance on this would be appreciated.

[masseyke]

I'm sorry I completely missed the questions on this. We don't currently have an API that reads from the .watcher_history index. It had originally been thought of as something for expert users to go to for troubleshooting. We would need to add that if it's something you need. Watcher is in a semi-deprecated state though, so probably worth a discussion @tylerperk @dakrone.

[dakrone]

The .watcher_history index is a regular (albeit hidden) index, as far as I can tell. Only the .watches index is a system index which requires the special APIs to utilize directly.

[yuliacech]

After a discussion with @ElenaStoeva, we thought that using the Query Watch API for pagination would be a performance improvement, instead of loading all existing watches into the browser. That is currently not possible because the Query Watch API filters and sorts only by the watch id. @masseyke Do you think it would be possible to add more watch properties like state, when it last met condition etc to filtering and sorting?

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)

[lukewhiting]

@sabarasaba / @alisonelizabeth

Hi folks! I have been tasked to look into this from the Watcher side. Looking at and testing locally the Query Watches API, it does seem that it provides the required functions (Filter, sort, page etc) however it seems the limiting factor is that the underlying .watches index has very few mapped fields, meaning most filtering / querying will not work.

I think the way forward here is if you could get us a list of the fields in .watches that you would like to sort and filter on, then we can update the system index mappings for .watches to open up the existing functions of the Query Watches API.

[sabarasaba]

Hey @lukewhiting, sorry for the delay its been a busy week! Currently sorting and filtering happen on client side. But if the api would support that we could leverage it, currently we allow users to

• sort on: _id metadata.name status.state.active status.last_met_condition status.last_checked
• filter on: name and id

[lukewhiting]

@sabarasaba Fantastic :-) Thanks for that info. I'll get to work making the required mapping changes for those fields.

[lukewhiting]

@sabarasaba I'm afraid we have hit a bit of a speed bump. We can deliver all the requested sort and filter fields requested with the exception of _id. Sorting and filtering (At least on partial matches) on ID is disabled by default as of recent 8.x versions with the following error: Fielddata access on the _id field is disallowed, you can re-enable it by updating the dynamic cluster setting: indices. id_field_data.

Talking to the Search Foundations team, there is deliberately no work around to this due to known performance issues when sorting by _id. I don't believe we can expect all our customers to either enable that setting or reindex their existing .watches index for this change.

How would you like to proceed? Would you like us to continue delivering the other sorts and filters or would you prefer to continue with client side filtering and sorting but using the new API endpoint?

[dakrone]

How would you like to proceed? Would you like us to continue delivering the other sorts and filters or would you prefer to continue with client side filtering and sorting but using the new API endpoint?

I think at the least we can make changes so that we start indexing the name of the watch for newly created .watches documents. That way, for new deployments and re-created watches the name can be sorted via this new field.

[lukewhiting]

I think at the least we can make changes so that we start indexing the name of the watch for newly created .watches documents. That way, for new deployments and re-created watches the name can be sorted via this new field.

Yep that's definitely an option but the client side logic for that may be rather complex? For an upgraded cluster you end up with a mix of some docs that have the new field and some that don't in the .watches index which either gives a rather confusing user experience as some docs come back correctly sorted and others don't or it need some careful handing on the Kibana side to fall back to client side sorting if any document is detected without the new field.

Happy to implement that change but would like to check in with the Kibana team to make sure it's worth it for them.

[lukewhiting]

@sabarasaba (and possibly @mattkime as you seem to be working on this?) Could I get your opinion this this comment please: #152142 (comment)

Would you like us to go ahead with implementing server side sort (Given the limitations and split results) or would you prefer to continue client side sorting just with the new API?

[sabarasaba]

@lukewhiting Hey sorry for the delay, I've been on paternal leave for a while and just now came back. Let me chat with the team today about it and I'll get back to you.

[sabarasaba]

We will try to get some telemetry on the average number of watches customers have in order to determine if we want to pursue a client side or server side implementation, we'll keep you posted

[tylerperk]

@sabarasaba See if you can use this dashboard for watcher telemetry

[lukewhiting]

If nobody objects, i'm going to close this issue as the required work for 9.0 and the original scope of this ticket is done.

I have raised a feature request on the Elasticsearch side for server side sorting and will update the team as and when that's implemented.