Search - add case insensitive support for regex queries.

[markharwood]

While Lucene has more complex bitmask flags for regex match settings like case insensitivity the JSON representation we add here is a simple case_sensitive boolean parameter which defaults to true. Currently this case logic only works with ASCII characters.

To implement this feature I have forked a copy of Lucene’s RegexpQuery and RegExp classes from Lucene master.
These classes can be removed when 8.7 Lucene is released which is the first release with the case insensitivity option.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jpountz]

Since we can't make the request case-sensitive if the analyzer/normalizer lowercased the content, I wonder if we should reject setting case_sensitive: true. (I think this question was initially raised by @jimczi.)

[markharwood]

I wonder if we should reject setting case_sensitive: true

We do state in the docs that case sensitivity (or insensitivity) is in relation to the indexed content so we have a justification for the logic. I don't have a strong opinion either way on whether to error or not.

The bit where I come unstuck is thinking about how to describe the default value (something we normally document).

[markharwood]

Following discussion with @jimczi we may want to reconsider the implementation here.

Rather than an explicit flag in the JSON we could consider adding passing the case insensitivity option as part of the regular expression string. There are a number of ways regex engines do this - either globally using /...../i syntax or locally within a section of the regex using ...(?i)...(?-i)

The advantage of expressing the option as part of the regex string is that query-string and KQL based inputs could be extended to make these choices (if the users know to use this syntax).
The alternative is that we stick with Elasticsearch APIs that express options in less cryptic ways and rely more on GUIs to let users select query options.

[jimczi]

Rather than an explicit flag in the JSON we could consider adding passing the case insensitivity option as part of the regular expression string. There are a number of ways regex engines do this - either globally using /...../i syntax or locally within a section of the regex using ...(?i)...(?-i)

I made this comment for the query_string query since this is used by Kibana KQL internally. I did not meant that we should do the same for the regex query where a flag or a boolean is ok to me.

[markharwood]

I made this comment for the query_string query since this is used by Kibana KQL internally. I did not meant that we should do the same for the regex query where a flag or a boolean is ok to me.

Ah - I thought we'd agreed we would want KQL, query_string and RegExpQuery to all support the same string syntax if we were to expand it. If there's two ways to express the option (in-string and in JSON boolean) there's the potential for conflicting arguments.

[markharwood]

run elasticsearch-ci/docs

[jimczi]

The change looks good to me. I left some comments regarding the testing of the new options.

[markharwood]

Thanks for the review, @jimczi
The one remaining concern I have is that user of the HLRC will make the mistake of using the RegExp.ALL flag when creating queries. In Lucene 8.6 that is 0xffff and we now reject that in RegExpQueryBuilder because RegExp87.ALL is only the range up to 0xff.
I suspect HLRC clients are unlikely to pass RegExp.ALL (RegExpQueryBuilder has DEFAULT_FLAGS_VALUE) but there's no type safety to prevent them from doing so.

Otherwise is this good to go?

[jimczi]

I suspect HLRC clients are unlikely to pass RegExp.ALL (RegExpQueryBuilder has DEFAULT_FLAGS_VALUE) but there's no type safety to prevent them from doing so.

Can we be more lenient ? It's a breaking change imo so we could apply 0xff as a mask to clear out the extra bits ?
We pass the flags value in the rest serialization so currently all clients would fail if they don't use the updated version ?

[markharwood]

Good to go @jimczi ?

[jimczi]

I left one question for cleanup, LGTM otherwise

[kevinqfu]

Hi, not sure if caused by this PR. We use regex in query_string and want the search to be case insensitive. It has been working until recently we upgraded our AWS ES cluster from 7.7 to 7.10. The same regex query became case sensitive now (it only works with lower case as in the standard analyzer). I'm unable to set case_insensitive param in query_string query.

Changing regex to wildcard makes it case insensitive again.