[Enhancement][Refactor] optimize array_contains_all/array_contains_seq function #51701

silverbullet233 · 2024-10-10T06:55:17Z

Why I'm doing:

What I'm doing:

optimize the performance of array_contains_all and array_contains_seq, and reduce their implementation complexity from O(m*n) to O(m+n)

basic idea

for array_contains_all, choose the side with less data to build a hash table to speed up the search
for array_contains_seq, it is essentially similar to the string search algorithm. We can use the idea of the KMP algorithm to speed up the search.

Test

Here I use two very simple queries to compare the performance

-- query 1
set @arr0 = array_generate(100000);
set @arr1 = array_generate(200000);
select array_contains_all(@arr1, @arr0);

-- query 2
set @arr0=array_append(array_repeat(1,100000),2);
set @arr1=array_append(array_repeat(1,50000),2);
select array_contains_seq(@arr0,@arr1);

	baseline	after optimization
query 1	1.66s	0.05s
query 2	1.08s	0.02s

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

Signed-off-by: silverbullet233 <[email protected]>

sonarcloud · 2024-10-18T06:25:16Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-10-18T08:17:16Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-10-18T08:18:36Z

[FE Incremental Coverage Report]

✅ pass : 5 / 5 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/catalog/FunctionSet.java	2	2	100.00%	[]
🔵	com/starrocks/sql/analyzer/DecimalV3FunctionAnalyzer.java	3	3	100.00%	[]

github-actions · 2024-10-18T08:24:08Z

[BE Incremental Coverage Report]

❌ fail : 0 / 249 (00.00%)

file detail

	path	new_line	coverage	not_covered_line_detail
🔵	be/src/util/bit_mask.h	4	00.00%	[36, 37, 40, 41]
🔵	be/src/exprs/array_functions.h	12	00.00%	[158, 159, 162, 164, 167, 169, 179, 180, 183, 185, 188, 190]
🔵	be/src/exprs/array_functions.tpp	224	00.00%	[1962, 1970, 1971, 1979, 1980, 1983, 1987, 1992, 1993, 1995, 1997, 2000, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2011, 2012, 2014, 2015, 2018, 2020, 2021, 2024, 2025, 2026, 2028, 2029, 2031, 2035, 2038, 2040, 2041, 2043, 2044, 2046, 2047, 2049, 2052, 2056, 2058, 2059, 2060, 2061, 2063, 2065, 2068, 2069, 2070, 2072, 2074, 2075, 2076, 2077, 2078, 2079, 2082, 2083, 2084, 2085, 2086, 2087, 2090, 2092, 2093, 2095, 2096, 2098, 2101, 2103, 2108, 2110, 2111, 2113, 2114, 2115, 2116, 2117, 2119, 2120, 2121, 2124, 2127, 2130, 2131, 2136, 2137, 2138, 2139, 2141, 2146, 2147, 2148, 2151, 2153, 2154, 2159, 2162, 2163, 2164, 2165, 2167, 2168, 2169, 2170, 2171, 2174, 2175, 2177, 2179, 2182, 2185, 2186, 2187, 2190, 2193, 2196, 2198, 2201, 2202, 2203, 2205, 2206, 2207, 2208, 2209, 2211, 2212, 2213, 2215, 2216, 2218, 2219, 2225, 2230, 2233, 2236, 2237, 2238, 2240, 2241, 2242, 2244, 2246, 2247, 2248, 2250, 2252, 2253, 2256, 2257, 2259, 2267, 2270, 2271, 2276, 2278, 2279, 2280, 2282, 2284, 2285, 2286, 2288, 2291, 2292, 2294, 2295, 2297, 2298, 2299, 2302, 2304, 2305, 2306, 2307, 2308, 2312, 2313, 2314, 2315, 2316, 2320, 2321, 2322, 2323, 2324, 2326, 2328, 2329, 2330, 2331, 2333, 2335, 2337, 2338, 2340, 2343, 2344, 2348, 2349, 2355, 2359, 2361, 2362, 2363, 2364, 2365, 2366, 2368, 2373, 2374, 2375, 2376, 2377, 2379, 2382, 2388, 2392, 2394, 2395]
🔵	be/src/column/column_helper.cpp	9	00.00%	[439, 440, 441, 443, 444, 445, 446, 447, 448]

github-actions · 2024-10-22T05:59:49Z

@Mergifyio backport branch-3.3

github-actions · 2024-10-22T05:59:51Z

@Mergifyio backport branch-3.2

mergify · 2024-10-22T06:00:21Z

backport branch-3.3

✅ Backports have been created

#52190 [Enhancement][Refactor] optimize array_contains_all/array_contains_seq function (backport #51701) has been created for branch branch-3.3 but encountered conflicts

mergify · 2024-10-22T06:00:22Z

backport branch-3.2

✅ Backports have been created

#52189 [Enhancement][Refactor] optimize array_contains_all/array_contains_seq function (backport #51701) has been created for branch branch-3.2 but encountered conflicts

…q function (#51701) Signed-off-by: silverbullet233 <[email protected]> (cherry picked from commit ac623b3) # Conflicts: # be/src/column/column_helper.h # be/src/exprs/array_functions.h # be/src/exprs/array_functions.tpp # be/src/util/bit_mask.h # fe/fe-core/src/main/java/com/starrocks/catalog/FunctionSet.java # gensrc/script/functions.py # test/sql/test_array_fn/R/test_array_contains # test/sql/test_array_fn/T/test_array_contains

…q function (#51701) Signed-off-by: silverbullet233 <[email protected]> (cherry picked from commit ac623b3) # Conflicts: # be/src/column/column_helper.h

…q function (backport #51701) (#52190) Signed-off-by: silverbullet233 <[email protected]> Co-authored-by: eyes_on_me <[email protected]> Co-authored-by: silverbullet233 <[email protected]>

…q function (StarRocks#51701) Signed-off-by: silverbullet233 <[email protected]>

…q function (StarRocks#51701) Signed-off-by: silverbullet233 <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

…q function (StarRocks#51701) Signed-off-by: silverbullet233 <[email protected]>

…q function (backport #51701) (#52934) Signed-off-by: silverbullet233 <[email protected]>

silverbullet233 requested review from a team as code owners October 10, 2024 06:55

github-actions bot added the title needs [type] label Oct 10, 2024

mergify bot assigned silverbullet233 Oct 10, 2024

silverbullet233 changed the title ~~[WIP][Enhancement][Refactor] optimize array_contains_all/array_contains_seq function~~ [Enhancement][Refactor] optimize array_contains_all/array_contains_seq function Oct 10, 2024

github-actions bot added 3.3 3.2 and removed title needs [type] labels Oct 10, 2024

Seaven previously approved these changes Oct 10, 2024

View reviewed changes

trueeyu previously approved these changes Oct 10, 2024

View reviewed changes

LiShuMing previously approved these changes Oct 10, 2024

View reviewed changes

silverbullet233 added 6 commits October 14, 2024 15:56

stash

15d1cae

Signed-off-by: silverbullet233 <[email protected]>

stash

1c528c4

Signed-off-by: silverbullet233 <[email protected]>

opt

d91af3e

Signed-off-by: silverbullet233 <[email protected]>

fix format

841fb09

Signed-off-by: silverbullet233 <[email protected]>

fix

990cb25

Signed-off-by: silverbullet233 <[email protected]>

fix

d45b86d

Signed-off-by: silverbullet233 <[email protected]>

silverbullet233 dismissed stale reviews from LiShuMing, trueeyu, and Seaven via d45b86d October 14, 2024 08:34

silverbullet233 force-pushed the opt_array_has branch from ea2b6b2 to d45b86d Compare October 14, 2024 08:34

fix ut

ba14e54

Signed-off-by: silverbullet233 <[email protected]>

silverbullet233 requested review from a team as code owners October 18, 2024 06:17

silverbullet233 enabled auto-merge (squash) October 21, 2024 11:58

Seaven approved these changes Oct 21, 2024

View reviewed changes

satanson approved these changes Oct 21, 2024

View reviewed changes

trueeyu approved these changes Oct 21, 2024

View reviewed changes

HangyuanLiu approved these changes Oct 21, 2024

View reviewed changes

LiShuMing approved these changes Oct 21, 2024

View reviewed changes

andyziye disabled auto-merge October 22, 2024 05:59

andyziye merged commit ac623b3 into StarRocks:main Oct 22, 2024
59 of 60 checks passed

github-actions bot removed the 3.3 label Oct 22, 2024

github-actions bot removed the 3.2 label Oct 22, 2024

This was referenced Oct 22, 2024

[Enhancement][Refactor] optimize array_contains_all/array_contains_seq function (backport #51701) #52189

Closed

[Enhancement][Refactor] optimize array_contains_all/array_contains_seq function (backport #51701) #52190

Merged

github-actions bot added the 3.3-merged label Oct 22, 2024

ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Oct 31, 2024

[Enhancement][Refactor] optimize array_contains_all/array_contains_se…

89b3ed0

…q function (StarRocks#51701) Signed-off-by: silverbullet233 <[email protected]>

silverbullet233 mentioned this pull request Nov 15, 2024

[Enhancement][Refactor] optimize array_contains_all/array_contains_seq function (backport #51701) #52934

Merged

42 tasks

silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request Nov 15, 2024

[Enhancement][Refactor] optimize array_contains_all/array_contains_se…

0c5bcc5

…q function (StarRocks#51701) Signed-off-by: silverbullet233 <[email protected]>

silverbullet233 added a commit to silverbullet233/starrocks that referenced this pull request Nov 15, 2024

[Enhancement][Refactor] optimize array_contains_all/array_contains_se…

755f8d9

…q function (StarRocks#51701) Signed-off-by: silverbullet233 <[email protected]>

wanpengfei-git pushed a commit that referenced this pull request Nov 15, 2024

[Enhancement][Refactor] optimize array_contains_all/array_contains_se…

ac9a824

…q function (backport #51701) (#52934) Signed-off-by: silverbullet233 <[email protected]>

github-actions bot added the 3.2-merged label Nov 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement][Refactor] optimize array_contains_all/array_contains_seq function #51701

[Enhancement][Refactor] optimize array_contains_all/array_contains_seq function #51701

silverbullet233 commented Oct 10, 2024 •

edited

Loading

sonarcloud bot commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

github-actions bot commented Oct 18, 2024

github-actions bot commented Oct 22, 2024

github-actions bot commented Oct 22, 2024

mergify bot commented Oct 22, 2024 •

edited

Loading

mergify bot commented Oct 22, 2024 •

edited

Loading

[Enhancement][Refactor] optimize array_contains_all/array_contains_seq function #51701

[Enhancement][Refactor] optimize array_contains_all/array_contains_seq function #51701

Conversation

silverbullet233 commented Oct 10, 2024 • edited Loading

Why I'm doing:

What I'm doing:

Test

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sonarcloud bot commented Oct 18, 2024

Quality Gate passed

github-actions bot commented Oct 18, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Oct 18, 2024

[FE Incremental Coverage Report]

file detail

github-actions bot commented Oct 18, 2024

[BE Incremental Coverage Report]

file detail

github-actions bot commented Oct 22, 2024

github-actions bot commented Oct 22, 2024

mergify bot commented Oct 22, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Oct 22, 2024 • edited Loading

✅ Backports have been created

silverbullet233 commented Oct 10, 2024 •

edited

Loading

mergify bot commented Oct 22, 2024 •

edited

Loading

mergify bot commented Oct 22, 2024 •

edited

Loading