Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API for fieldname suggestion/completion #74816

Open
markharwood opened this issue Jul 1, 2021 · 7 comments
Open

API for fieldname suggestion/completion #74816

markharwood opened this issue Jul 1, 2021 · 7 comments
Labels
>enhancement :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch

Comments

@markharwood
Copy link
Contributor

Like the recently added terms_enum api, there is a need for clients like Kibana to help people complete the values they are typing. This requires a high-speed API to look up field names that match parts of what the user has already typed.

It's true to say that clients like Kibana typically already have ready access to lists of the physical fields in an index but the one omission is the potentially millions of "soft" field names introduced by the flattened field type which exist as document data in the index. Previously we had considered creating an API for listing just flattened field names but for practical purposes it would be convenient for clients if there was one elasticsearch API for autocompleting all field names, flattened or otherwise.

Usability concerns

The precise details of the API will depend to a large extent on the user experience clients want to provide and these are being discussed by members of the Kibana team (cc @elastic-jb ).
There are 2 main styles of matching - infix and prefix - either matching user input anywhere in the field name (infix) or completing the end of the field name being typed (prefix). Both have their merits-

  • Infix matching could be useful to find any field name regardless of document structure e.g. searching for bytes might match the field names source.http.bytes_sent and target.dns.bytes. The user only cares about fields that mention bytes.
  • Prefix matching could be useful to navigate using the structure of hierarchical documents e.g. ECS organises properties under several object fields so typing dns would match dns.answers and dns.id and further selecting dns.answers would reveal dns.answers.name and dns.answers.id.

This raises the question do users want to make use of the hierarchical structure of documents or ignore it completely?

Infix matching concerns

Infix matching is more expensive to run on flattened fields with millions of values so we will either not support it at all or curtail exploration after a max number of terms scanned or time taken

Prefix matching concerns

Is this a breadth-first match of matching values or depth-first? So in ECS would prefix completion of c suggest
a) breadth-first client cloud and container OR
b) depth-first client.address, client.bytes, client.domain etc?
With a) would the client only see values for client like client.address after they had typed the ending full stop e.g. client.? This is how completion works in a unix shell when typing field names in the command line - you only see the files in subdirectories when you put the / character at the end of whatever containing folder name e.g. typing D and using tab-completion will not suggest filenames from the Downloads or Documents folder until you type Downloads/ or Documents/. Clearly the client needs to know which fields represent objects or "leaf" fields with no subvalues. In the unix shell these suggestions are displayed differently (folders are always displayed with the trailing /)

Results format

Unlike the terms_enum api, returning simple strings with the matching names is probably not sufficient. We should probably return some type information too. Again, some input from UX teams would help here.

@markharwood markharwood added >enhancement :Search/Search Search-related issues that do not fall into other categories labels Jul 1, 2021
@markharwood markharwood self-assigned this Jul 1, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Jul 1, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@jimczi jimczi changed the title Feature - new API for fieldname suggestion/completion API for fieldname suggestion/completion Jul 1, 2021
markharwood added a commit to markharwood/elasticsearch that referenced this issue Sep 21, 2021
This implementation has the following issues:

* probably many `terms_enum` references that need renaming to `fields_enum` (a lot of copy and paste was used)
* the implementation offers infix matching although I expect we’ll revert to prefix matching only once we’ve got the Kibana requirements,

Closes elastic#74816
markharwood added a commit to markharwood/elasticsearch that referenced this issue Jan 12, 2022
This implementation has the following issues:

* probably many `terms_enum` references that need renaming to `fields_enum` (a lot of copy and paste was used)
* the implementation offers infix matching although I expect we’ll revert to prefix matching only once we’ve got the Kibana requirements,

Closes elastic#74816
@tsullivan
Copy link
Member

Hi @markharwood I'm working on a project definition for field list hierarchies in Kibana, and was led to this issue. It looks like this new API will be useful in Kibana to provide a unified experience to allow users to quickly find a field they're looking for when it is in an object or flattened structure.

  • Would it allow interactions with the *_range field types as well?
  • Can you confirm that the nested field type doesn't intersect with this?

@javanna
Copy link
Member

javanna commented Jul 5, 2022

hey @tsullivan thanks for the ping. The work on this API has been suspended. We need to get back to it but we are not sure when we will get to it. It would be good to sync up and collect all the usecases for this new API, the needs and eventually re-prioritize this effort.

@kertal
Copy link
Member

kertal commented Apr 26, 2023

We discussed this in the team. We currently rely on having all fields available the doing filename suggestions. Here are 2 examples:

Adding filters:

Bildschirmfoto 2023-04-26 um 11 07 55

Selecting breakdown fields:
Bildschirmfoto 2023-04-26 um 11 08 05

The question is, having just a field suggestion/completion would not be sufficient. We would also need to be able to filter by type, and also we would need information about the capabilities of the field. Also helpful it would be to filter out fields without values. All this sounds like something we already have ... the field_caps API. Given we need more input/output options for completion, would there be any advantage in creating a dedicated API fro filename completion/suggestion? Or wouldn't it make more sense to further improve field_caps

@javanna
Copy link
Member

javanna commented Apr 26, 2023

Given we need more input/output options for completion, would there be any advantage in creating a dedicated API fro filename completion/suggestion?

Speed. Field caps is not made to provide real-time suggestions. Yet if we were to create a more responsive API targeted at suggestions, we would have to accept limited functionality to keep it fast.

@kertal
Copy link
Member

kertal commented Apr 26, 2023

Linking more issues with use cases @lukasolson identified that could benefit from this API:
elastic/kibana#24709
elastic/kibana#111642
elastic/kibana#96150

@javanna javanna added :Search Foundations/Search Catch all for Search Foundations and removed :Search/Search Search-related issues that do not fall into other categories labels Jul 17, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch label Jul 17, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search-foundations (Team:Search Foundations)

@elasticsearchmachine elasticsearchmachine removed the Team:Search Meta label for search team label Jul 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Foundations/Search Catch all for Search Foundations Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants