Form filter: filter autocomplete list based on previous applied filters

[nboisteault]

This PR changes the behaviour of the form filter. Previously, all unique values from the text field (with the autocompletion activated) were downloaded once at startup. Now, every time the user types letters, a new request is triggered with the given filter and the LIKE and fetches all corresponding data.

Funded by VSB

simplescreenrecorder-2024-12-03_10.51.09.mp4

[Gustry]

Nice addition.

Related to form_filter, is there any chance we can remove the manual test about the "form filter" at the same time, as you moved from Cypress to Playwright ? https://github.com/3liz/lizmap-web-client/blob/master/tests/qgis-projects/tests/form_filter.md (maybe with the help of echoproxy ?)

[nboisteault]

Nice addition.

Related to form_filter, is there any chance we can remove the manual test about the "form filter" at the same time, as you moved from Cypress to Playwright ? https://github.com/3liz/lizmap-web-client/blob/master/tests/qgis-projects/tests/form_filter.md (maybe with the help of echoproxy ?)

Done bae3440

[Gustry]

Thanks for the removal of the manual test ... :)

[mdouchin]

This PR changes the behaviour of the form filter. Previously, all unique values from the text field (with the autocompletion activated) were downloaded once at startup. Now, every time the user types letters, a new request is triggered with the given filter and the LIKE and fetches all corresponding data.

It is more dynamic and better than before, but I think we should test it against a big dataset to evaluate the performance. LIKE clauses are known to have poor performances. We could promote the use of the pg_trgm PostgreSQL extension (and add index as described here https://docs.lizmap.com/next/en/publish/configuration/spatial_search.html#optimisation)

[nboisteault]

This PR changes the behaviour of the form filter. Previously, all unique values from the text field (with the autocompletion activated) were downloaded once at startup. Now, every time the user types letters, a new request is triggered with the given filter and the LIKE and fetches all corresponding data.

It is more dynamic and better than before, but I think we should test it against a big dataset to evaluate the performance. LIKE clauses are known to have poor performances. We could promote the use of the pg_trgm PostgreSQL extension (and add index as described here https://docs.lizmap.com/next/en/publish/configuration/spatial_search.html#optimisation)

I've tested with 1 100 000 lines inserted with:

INSERT INTO tests_projects.form_filter (label, geom)
SELECT
 md5(random()::text),
 null
FROM generate_series(1, 1000000);

Request made to get values when typing 'abc' returns in 800ms. 'abcd' or 'abcde' in 300ms.

[mdouchin]

Request made to get values when typing 'abc' returns in 800ms. 'abcd' or 'abcde' in 300ms.

great. Have you tried by using the following index

CREATE EXTENSION IF NOT EXISTS pg_trgm;
DROP INDEX IF EXISTS test_index;
CREATE INDEX test_index ON tests_projects.form_filter USING GIN (label gin_trgm_ops);

[nboisteault]

Request made to get values when typing 'abc' returns in 800ms. 'abcd' or 'abcde' in 300ms.

great. Have you tried by using the following index

CREATE EXTENSION IF NOT EXISTS pg_trgm;
DROP INDEX IF EXISTS test_index;
CREATE INDEX test_index ON tests_projects.form_filter USING GIN (label gin_trgm_ops);

Now request made to get values when typing 'abc' returns in 500ms. 'abcd' or 'abcde' in 100ms.

[mdouchin]

The generated filter can contain OR terms. We should use parenthesis to ensure the correct logic will be used. Spaces are used around them to please QGIS :

Suggested change

	sdata.filter = globalThis['filterConfigData'].filter + ' AND ' + autocompleteFilter;
	sdata.filter = ' ( ' + globalThis['filterConfigData'].filter + ' ) AND ' + autocompleteFilter;