New search bar component

[AlexRuiz7]

Summary

Our current SearchBar component is far from working properly, and it's being harder and harder to maintain over time.

We need to deprecate this component in favor of a better one, and use it consistently along the App.

This Epic will hold the issues of the current component to be used as functional requirements for the new component, and also will cover its design and development.

Functional requirements

1. The search bar must allow any operator defined by the Wazuh Query Language, including parentheses. The search bar removes parentheses from the values #4313
2. The search bar must allow values to contain and and or on their names, including the white spaces. Error searching vulnerabilities by the name indexed in the summary  #4088

Non-functional requirements

To be done.

Related issues

• The search bar removes parentheses from the values #4313
• Error searching vulnerabilities by the name indexed in the summary  #4088
• Ruleset search bars don't sanitize input #3968
• The help/suggestion feature on the search bar is not working #3818
• WzSearchBar component malfunction with suggestions that contain " and " or " or " #3325

Design

There are several designs on the table at the moment. The most supported one is to use the Wazuh Query Language as is, with no translations, as was done on the previous search bar, which translated any and and or in the query with ; and ,, respectively.

Desirable tasks

• Research if it is possible to remove the manual maintenance of the suggestions using the API specification. Feed the suggestion field for the new search bar #5370

[Machi3mfl]

Research about current SearchBar behavior

sequenceDiagram
WzSearchBar->>SuggestHandler: input query in string
SuggestHandler ->> QInterpreter: query string
loop
    QInterpreter->>QInterpreter: PARSE and VALIDATE the query
end
QInterpreter->>SuggestHandler: response is VALID or INVALID
SuggestHandler->>WzSearchBadges: send VALID filters
Note right of WzSearchBadges: Add badges for each VALID filter

Loading

Query Filters route

graph TD;
UISearchBar-->ValidFiltersList-->filtersToObject-->WazuhManager(API)

Loading

[Desvelao]

The current search bar translates the input to the expected Wazuh query syntax. This has some problems related to the and and or words, that translate to AND (;) and OR (,) operators of Wazuh query syntax. This affects to manage of when should suggest a new field.

I think the easier approach is using directly the Wazuh query language without translations. The problem with this approach is that is more complicated for users that don't know the query language.

Idea

I think we could offer multiple query languages.

One of them, should be the Wazuh query language without translations, this would avoid any problems on our side.

Moreover, we could add a user-friendly language/interface to the query that could be based on the current query syntax provided by the WzSearchBar component. It will have the same difficulties as the current query syntax.

This approach is based on the search bar used to query to Elasticseach/Wazuh indexer in the Discover or Dashboard plugins.

[Desvelao]

POC - Search bar - Multiple query languages

I am working around the idea of a search bar that supports multiple query languages. The selection can be changed through a control.

This base component expects the query language to be registered and has a specific interface to interact with.

The base component only manages the input text, then this is passed to the selected query language to analyze the query and returns the properties that should use the base component. This enables to the extension of the functionality.

The new search bar base component is based on a custom EuiSuggest component (the current WzSearchBar wraps it).

I tried to use the EuiSuggest component from the @elastic/eui, but it seems to have some bugs. I guess this is the reason because the EuiSuggest component was recreated as a custom one in the plugin source code to use in WzSearchBar component.

The approach of this POC, let to decouple the business logic of each query language. For example, adding suggestions according to the input query or defining the action when a suggestion item is clicked.

[Desvelao]

POC - Query language - API query language

Taking as the base the POC of the new search bar component that supports multiple query languages, I am working on a compatible interface that is based on a similar approach to the Wazuh API query language https://documentation.wazuh.com/current/user-manual/api/queries.html.

The behavior is similar to the current WzSearchBar. This means displaying suggestions to help the user.

The current implementation analyzes (through a custom analysis) the query and tokenizes it. Then, these tokens are used to get the suggestions that will be displayed in the search bar.

I have added some tests with use cases to test the tokenizer and the suggestions handler.

Features:

• Get suggestions from the input
• Supports dynamic suggestions (let to build the suggestions from any source that requires fetching data)

This query language has not user-friendly or easily understandable syntax by not familiar users. The purpose of adding this query language is to prevent the interpretation of a high-level query syntax as the current WzSearchBar component that causes some queries don't work as expected. The idea of the new search bar that supports multiple query languages is to add a new high-level syntax more user-friendly and allows the user to choose which query language to use.

Screenshots:

Open the query language information

This contains information about the current query language and a selector to change the selected query language.

Suggestions

[davidjiglesias]

@wazuh/framework will complete an investigation on the reports done by @Desvelao and our current implementation of the q parameter.

[Desvelao]

To prevent some differences between the expected queries by the Wazuh API and the implementation of the query language in the search bar, we must know the implementation and decomposition of the query through the q query parameter used in Wazuh API endpoints. This knowledge could help us to replicate the business logic and reduce the discordances.

[Desvelao]

POC - Search bar - Query Language custom behavior

I am working around the idea to allow each query language can define custom behaviors.

As a POC, I added a prepend element to the search bar to indicate that there is a implicit filter. This is useful to display that there is a non-editable query that will be added to the user query. This case is in the search bar located in the Agents section, that negates the data related to the Wazuh managers (id is not 000).

Screenshots

I need to refactor some logic in the POC of new search bar component.

[Desvelao]

POC - Search bar - Multiple query languages

I uploaded a POC of this search bar to the branch https://github.com/wazuh/wazuh-kibana-app/tree/feat/4312-poc-search-bar-component-multiple-query-language-api-query-language-implementation.

This POC contains:

• New search bar component that supports multiple query languages
• Query languages:
  • AQL: custom implementation of a similar approach of Wazuh Query Language
    • Include suggestions
  • UIQL: simple implementation
• The search bar was implemented as an example in the section of Agents

[Desvelao]

The work in the POC will be on hold until the @wazuh/framework can give us details about the implementation and analysis of the query in the q query parameter. This knowledge will let us work on the implementation of the query language and provide suggestions to the user.

[Desvelao]

Discussion

In today's meeting, it was decided the framework team will work to provide knowledge how the query decomposition works in the q query parameter.

[Selutario]

Decomposition of q API parameter

The following is the procedure applied in the framework for those requests that include q. This is a somewhat complex process with multiple particularities depending on the data to be obtained. The explanation will be based on this example request:

GET /agents?q=(status=active;id<3),(status=disconnected);(name~c)

1. Regex

First, it is checked if the value of q matches with the following regex:

        # To correctly turn a query into SQL, a regex is used. This regex will extract all necessary information:
        # For example, the following regex -> (name!=wazuh;id>5),group=webserver <- would return 3 different matches:
        #   (name != wazuh ;
        #    id   > 5      ),
        #    group=webserver
        self.query_regex = re.compile(
            # A ( character.
            r"(\()?" +
            # Field name: name of the field to look on DB.
            r"([\w.]+)" +
            # Operator: looks for '=', '!=', '<', '>' or '~'.
            rf"([{''.join(self.query_operators.keys())}]{{1,2}})" +
            # Value: A string.
            r"((?:(?:\((?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]*)\))*"
            r"(?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]+)"
            r"(?:\((?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]*)\))*)+)" +
            # A ) character.
            r"(\))?" +
            # Separator: looks for ';', ',' or nothing.
            rf"([{''.join(self.query_separators.keys())}])?"
        )

(\()?([\w.]+)([=!=<>~]{1,2})((?:(?:\((?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]*)\))*(?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]+)(?:\((?:\[[\[\]\w _\-.,:?\\/'\"=@%<>{}]*]|[\[\]\w _\-.:?\\/'\"=@%<>{}]*)\))*)+)(\))?([,;])?

Here its parts can be better visualized:

It has undergone many changes, and its current state is the result of this issue:

• Unify 4.3.0 and master's regex used in the API query parameter wazuh#13369

2. Extract matching groups

From the original query ((status=active;id<3),(status=disconnected);(name~c)) tuples are extracted with 6 groups for each of them (1. opening parenthesis, 2. field name, 3. operator, 4. value, 5. closing parenthesis, 6. separator) (link):

[
    ('(', 'status', '=', 'active', '', ';'), 
    ('', 'id', '<', '3', ')', ','), 
    ('(', 'status', '=', 'disconnected', ')', ';'), 
    ('(', 'name', '~', 'c', ')', '')
]

If any of the field names (2.) do not match those predefined for the resource type (in this case, agent), an error message is raised. The same happens with operators:
https://github.com/wazuh/wazuh/blob/36d17b36a732ecc4b4bffda7ddd047d60a3c79ac/framework/wazuh/core/utils.py#L1505-L1506

3. Process matching groups

Each tuple is iterated, processing each field name and value if necessary. For example:

• The characters " and ' are replaced by wildcards (_) (link).
• A subset of valid operators is used if the field name is group. Wildcards are also included. This is an example of specific processing for agent queries. (link).
• If it includes operations with date type data, they are transformed into the appropriate format according to the operation (link).

Result

After applying the above steps (and a few more that are not specific to the q parameter), the example query (status=active;id<3),(status=disconnected);(name~c) is transformed into this part of a SQL statement:

((connection_status = 'active' COLLATE NOCASE) AND (id < '3' COLLATE NOCASE)) OR (connection_status = 'disconnected' COLLATE NOCASE) AND (name LIKE '%c%' COLLATE NOCASE)

The complete SQL query that would be sent to wazuh-db for this GET /agents request would be this:

SELECT disconnection_time as 'disconnection_time',os_arch as 'os.arch',os_major as 'os.major',config_sum as 'configSum',last_keepalive as 'lastKeepAlive',os_minor as 'os.minor',connection_status as 'status',os_uname as 'os.uname',coalesce(ip,register_ip) as 'ip',os_build as 'os.build',os_version as 'os.version',internal_key as 'internal_key',id as 'id',date_add as 'dateAdd',`group` as 'group',os_codename as 'os.codename',name as 'name',merged_sum as 'mergedSum',version as 'version',node_name as 'node_name',register_ip as 'registerIP',os_platform as 'os.platform',group_config_status as 'group_config_status',os_name as 'os.name',manager_host as 'manager' FROM agent WHERE ((connection_status = 'active' COLLATE NOCASE) AND (id < '3' COLLATE NOCASE)) OR (connection_status = 'disconnected' COLLATE NOCASE) AND (name LIKE '%c%' COLLATE NOCASE)   ORDER BY id ASC LIMIT 500 OFFSET 0

[Desvelao]

Thank you so much for the q parameter analysis @Selutario!

I will work to enhance the current API query language implementation in the search bar.

[Desvelao]

I would like to implement a validation of the query in the search bar to avoid or reduce and indicate to the user the query is invalid.

@Selutario, is there some validation that the API is checking according to the schema? or validating if the group operators (opening and closing) are in the correct place or grouping level? Any more consideration? If yes, could you provide the information?

Thank you so much!

[Desvelao]

POC - Query language - API query language

In a previous meeting related to the search query done in #5196, it was mentioned that the new search bar and AQL implementation could use this query when the user introduces a search term, searching in each used (table columns) through a like (~).

The current implementation of AQL doesn't support this functionality so the user should introduce the query manually.

We should discuss if this feature should be allowed by the implementation or not.

[Desvelao]

POC - Query language - API query language - Hidding the suggestions when using Search suggestion.

The search bar displays a suggestion to run the search query. When clicking it, the suggestions popover disappeared for a short period of time and this appeared again.

Digging

I was researching the cause of this. I reviewed the current WzSearchBar implementation that changes the visibility of the suggestions popover and blurs the input. This seems not to work as expected, and the suggestion popover appeared after using the Search suggestion, the same for the Apply filter suggestion of the WzSearchBar.

Clicking in a suggestion item of the popover, ran a handler. This handler called the onSearch function and close the suggestion popover and blurred the input (same logic as the current WzSearchBar), then a focus event was triggered causing the visibility of the suggestion popover is on. I didn't understand the reason for this behavior so I researched the cause of this.

I researched and tried multiple things without success.

Finally, I asked for help and I had a meeting with @asteriscos, where I was explaining him the problem to review if I was missing something. We concluded that something unknown and not intentional by our side is firing the focus event in the input. So I researched the components used. One of them is coming from the opensearch-project/oui that is called EuiInputPopover https://github.com/opensearch-project/oui/blob/1.0.0/src/components/popover/input_popover.ts. The EuiInputPopover uses a focus trap, which could be the cause of the focus event being fired. I found a property to disable this behavior called disableFocusTrap https://github.com/opensearch-project/oui/blob/1.0.0/src/components/popover/input_popover.tsx#L141. Adding the disableFocusTrap property to the EuiInputPopover fixed the problem of the undesired behavior.

[Desvelao]

POC - Query language - API query language

Fixes

• Fixed a minor problem related to input text with undefined value

[Selutario]

Answering these questions:

Is there some validation that the API is checking according to the schema?

Yes, there are multiple validations in addition to the regex. For each match (<opening_parenthesis><field><operator><value><closing_parenthesis><separator>) within the query, the following validations are done:

1. Check whether field is allowed

Code:

            if field not in self.fields.keys():
                raise WazuhError(1408, "Available fields: {}. Field: {}".format(', '.join(self.fields), field))

The allowed fields vary for each type of data, not being the same for agents as for syscollector (for example). The following is a list of all allowed fields for each of the endpoints that access wazuh-db:

• Agents fields:
  • GET /agents
  • PUT /agents/upgrade
  • PUT /agents/upgrade_custom
  • GET /agents/upgrade_result
  • GET /agents/no_group
  • GET /agents/outdated
  • GET /agents/stats/distinct
  • GET /groups/{group_id}/agents
• Ciscat fields:
  • GET /ciscat/{agent_id}/results
• *Decoders fields:
  • GET /decoders
• *Cluster fields:
  • GET /cluster/nodes
  • GET /cluster/{node_id}/logs
• *Manager:
  • GET /manager/logs
• MITRE:
  • *GET /mitre/groups: fields and relation fields and extra fields.
  • *GET /mitre/mitigations: fields and relation fields and extra fields
  • *GET /mitre/references: fields.
  • *GET /mitre/tactics: fields and relation fields and extra fields
  • *GET /mitre/software: fields and relation fields and extra fields
  • *GET /mitre/techniques: fields and relation fields and extra fields
• Rootcheck fields:
  • GET /rootcheck/{agent_id}
• *Rule fields:
  • GET /rules
• SCA:
  • GET /sca/{agent_id}: fields.
  • GET /sca/{agent_id}/checks/{policy_id}: All these fields.
• Syscheck:
  • WIP (it should be possible to use any field returned in the responses)
• Syscollector:
  • WIP (it should be possible to use any field returned in the responses)
• Tasks:
  • WIP (it should be possible to use any field returned in the responses)
• Vulnerability:
  • WIP (it should be possible to use any field returned in the responses)

In addition, those marked with an asterisk do not get the data from the database. This means that they do not follow the process mentioned here. Therefore, although in those cases the q parameter tries to mimic the behavior of the other listed endpoints, its behavior is not identical (for example, it does not verify that the fields used are allowed, simply nothing is returned if the field does not exist).

Is validated whether group operators (opening and closing) are in the correct place or grouping level?

No, it is not validated. However, wazuh-db will return an error response if those groups are wrong.

Any more consideration?

WIP

[Desvelao]

Answering these questions:

Is there some validation that the API is checking according to the schema?

Yes, there are multiple validations in addition to the regex. For each match (<opening_parenthesis><field><operator><value><closing_parenthesis><separator>) within the query, the following validations are done:

1. Check whether field is allowed

Code:

            if field not in self.fields.keys():
                raise WazuhError(1408, "Available fields: {}. Field: {}".format(', '.join(self.fields), field))

The allowed fields vary for each type of data, not being the same for agents as for syscollector (for example). The following is a list of all allowed fields for each of the endpoints that access wazuh-db:

• Agents fields:

  • GET /agents
  • PUT /agents/upgrade
  • PUT /agents/upgrade_custom
  • GET /agents/upgrade_result
  • GET /agents/no_group
  • GET /agents/outdated
  • GET /agents/stats/distinct
  • GET /groups/{group_id}/agents

• Ciscat fields:

  • GET /ciscat/{agent_id}/results

• *Decoders fields:

  • GET /decoders

• *Cluster fields:

  • GET /cluster/nodes
  • GET /cluster/{node_id}/logs

• *Manager:

  • GET /manager/logs

• MITRE:

  • *GET /mitre/groups: fields and relation fields and extra fields.
  • *GET /mitre/mitigations: fields and relation fields and extra fields
  • *GET /mitre/references: fields.
  • *GET /mitre/tactics: fields and relation fields and extra fields
  • *GET /mitre/software: fields and relation fields and extra fields
  • *GET /mitre/techniques: fields and relation fields and extra fields

• Rootcheck fields:

  • GET /rootcheck/{agent_id}

• *Rule fields:

  • GET /rules

• SCA:

  • GET /sca/{agent_id}: fields.
  • GET /sca/{agent_id}/checks/{policy_id}: All these fields.

• Syscheck:

  • WIP (it should be possible to use any field returned in the responses)

• Syscollector:

  • WIP (it should be possible to use any field returned in the responses)

• Tasks:

  • WIP (it should be possible to use any field returned in the responses)

• Vulnerability:

  • WIP (it should be possible to use any field returned in the responses)

In addition, those marked with an asterisk do not get the data from the database. This means that they do not follow the process mentioned here. Therefore, although in those cases the q parameter tries to mimic the behavior of the other listed endpoints, its behavior is not identical (for example, it does not verify that the fields used are allowed, simply nothing is returned if the field does not exist).

Is validated whether group operators (opening and closing) are in the correct place or grouping level?

No, it is not validated. However, wazuh-db will return an error response if those groups are wrong.

Any more consideration?

WIP

This is useful to add minimal validation in the search bar and inform to the user of a problem with the input query.

I will see to add the validation of field to the POC. We thought about what to do with the validation of group operators.

At the moment, I have no more considerations.

Thank you so much for the information @Selutario!

[Desvelao]

POC - Search bar - Display input query error

The new search bar component wraps the custom EuiSuggest component. If we want to be able to display a message about the input validation error, we will have to modify the custom EuiSuggestInput component (public/components/eui-suggest/suggest_input.js) wrapping the EuiFieldText with a EuiFormRow that enables to display the error message.

For the required query language implementations, they should validate the user input and returns the validation error when the input analysis is done. This property could be forwarded to the EuiFormRow component to display the error and inform to the user.

The validation of the input query could be useful to prevent the user can run an invalid query and inform that there is an error.

[Desvelao]

POC - Query language - API query language

In a previous meeting related to the search query done in #5196, it was mentioned that the new search bar and AQL implementation could use this query when the user introduces a search term, searching in each used (table columns) through a like (~).

The current implementation of AQL doesn't support this functionality so the user should introduce the query manually.

We should discuss if this feature should be allowed by the implementation or not.

Some options:

• Manage from the user input. We should differentiate if the input is a valid query for q or is a query that should be searched in each field using the like operator ~.

User-friendly
Complicated implementation and could cause unexpected behaviors.

• When there is some input, display a suggestion that when clicked, runs the search transforming the query to use the like operator in all the expected fields.

Less user-friendly than the previous option
Less complicated of implementing

As I said in the quoted message, we should discuss if we want to implement this feature.

[Desvelao]

POC - Search bar

Changes

• Add to the search bar example implementation the ability to receive filters from the external components to the search bar
• Enhance the search bar component documentation

[Desvelao]

Disccussion

In today's meeting, I was exposing the work done on this issue and the POC.

The action items to do are:

• Implement a new query language, similar to the current one of WzSearchBar that solves the problems of this.

The query language can't be easily ported to the new search bar component.

[Desvelao]

POC - Search bar - HAQL

I was working on a high-level implementation based on the Wazuh API Query Language.

The syntax is similar to the current query language used by the current WzSearchBar component. It uses a human-understandable logical operators (and, or) instead of the used by the Wazuh API Query Langue.

HAQL is the key name of this language. This could be changed. It means Human API Query Language.

I added the ability to define the values wrapped with quotes " and the quote " can be escaped using a \ escape character (\").

Moreover, the usage of whitespace to separate entities is optional.

Features:

• Display suggestions
• Support implicit query
• Support external query

I added documentation related to this query language.

I added tests to test some functionalities.

Screenshots

[Desvelao]

Changes

I replaced the search bar and table component on some sections of the Wazuh plugin.

Pull requests:

• Replace search bar on TableWzAPI component #5442
• Replace search bar on Agent > Inventory data #5443
• Replace search bar on Modules > Integrity monitoring > Inventory #5444
• Replace search bar on Management > Groups #5445
• Replace search bar on Explore agent modal #5447
• Replace search bar on Agents #5452

The pull requests are on draft due they need the enhancement of API endpoints. The work is paused until wazuh/wazuh#16552 is solved and merged, so I can develop with a Wazuh manager and get other potential requirements of all the implementation.

Currently, potential problems to solve:

• The date fields of q query parameter don't support some values wazuh#16902 (comment)

I added the actions items: wazuh/wazuh#16902 (comment)

The tables in the Security section can't be replaced easily due these tables use data from 2 API endpoints and possibly the suggestion or search doesn't work as expected. This should be discussed with the API co-workers. It would be interesting to increase the UX if these tables could use the reusable TableWzAPI component. I will open an issue to discuss this.

#5467
wazuh/wazuh#17171

[Desvelao]

Check

I checked a problem happening in the previous search bar #4313 (comment). This issue could be closed once we replace the current search bar by the new one.

[Desvelao]

Update

The development that was asked here has been completed:

issue: wazuh/wazuh#16552
pull request: wazuh/wazuh#17865

We can continue to implement the missing features in the related pull requests.

[Desvelao]

Changes

• Add the API requests to get the distinct values
• Fix minor bugs found
• Remove specific validation for the token of type value that is causing catastrophic backtracking.

TODO

• Add specific validation for the token of type value

Done in 579d30f

• Remove suggestions for the values we expect that return values that could cause problems if the suggestion is used. Workaround.

[Desvelao]

Problem

Some suggested values can not be used to search due a validation error

Solution

As a workaround, the values provided by the API responses are filtered. They should match the validation regular expression. This ensures the validation done by the search bar will not display a validation error when using a suggested value. This causes some possible value suggestions to be not displayed.

The validation is based on a regular expression used by API. If this is changed, then we should change it too. A more permissive validation by API could reduce this problem.

Current approach:

• Get the distinct values through an API request with a limit (30)
• Use the filter that match the frontend regular expression that validates a token of type value
• Limits the displayed suggestions (10)

Clarification: the API request uses a limit (30) greater that the ones displayed (10) in the UI with the intention to get at least the values that are displayed. This is related to some values retrieved in the API response could not match the regular expression used to filter them.

[JuanGarriuz]

Problem

When using suggested close parenthesis the query into the search bar doesn't work

[Desvelao]

Problem

When using suggested close parenthesis the query into the search bar doesn't work

This was fixed here: 9e121ad

[Desvelao]

Changes

A new mechanism was added to debounce the updating of the search bar state was added. This mitigates some problems related to the state changes that cause the UI to display data that should be incorrect regarding the input query. This lets to reduce the API request count if the user is typing in the search bar within the time interval.

For another hand, the search bar seems to be less reactive due to the debounce time.

[Tostti]

All PRs successfully merged