Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement](inverted index) apply inverted index when has any #41547

Merged
merged 1 commit into from
Oct 8, 2024
Merged

[Enhancement](inverted index) apply inverted index when has any #41547

merged 1 commit into from
Oct 8, 2024

Conversation

airborne12
Copy link
Member

Proposed changes

This PR optimizes the inverted index logic by skipping the execution of inverted index conditions when none are present. This change improves performance by avoiding unnecessary inverted index condition flow.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9631/25834)
Line Coverage: 28.68% (79885/278582)
Region Coverage: 28.10% (41291/146931)
Branch Coverage: 24.73% (21039/85086)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2846d83e2c3ad7045a856d49d6ce0a161db45613_2846d83e2c3ad7045a856d49d6ce0a161db45613/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40477 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2846d83e2c3ad7045a856d49d6ce0a161db45613, data reload: false

------ Round 1 ----------------------------------
q1	17599	8124	7319	7319
q2	2016	281	268	268
q3	12003	1049	1135	1049
q4	10574	757	741	741
q5	7751	2901	2826	2826
q6	235	153	147	147
q7	1003	616	591	591
q8	9354	1889	1909	1889
q9	6897	6480	6399	6399
q10	6984	2263	2303	2263
q11	431	244	237	237
q12	402	216	221	216
q13	17772	2989	2994	2989
q14	244	210	220	210
q15	572	529	529	529
q16	645	568	574	568
q17	973	515	549	515
q18	7175	6624	6689	6624
q19	1364	867	925	867
q20	465	205	197	197
q21	3922	3278	3042	3042
q22	1114	991	1006	991
Total cold run time: 109495 ms
Total hot run time: 40477 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7203	7277	7207	7207
q2	323	231	230	230
q3	3025	2922	2948	2922
q4	2081	1878	1798	1798
q5	5744	5732	5763	5732
q6	229	140	144	140
q7	2228	1858	1856	1856
q8	3337	3549	3463	3463
q9	8924	8931	8848	8848
q10	3588	3554	3541	3541
q11	578	495	487	487
q12	826	651	626	626
q13	10149	3218	3181	3181
q14	308	267	281	267
q15	583	532	505	505
q16	688	653	635	635
q17	1846	1608	1641	1608
q18	8294	7714	7476	7476
q19	1690	1374	1575	1374
q20	2092	1897	1867	1867
q21	5461	5376	5454	5376
q22	1138	1069	1045	1045
Total cold run time: 70335 ms
Total hot run time: 60184 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192540 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2846d83e2c3ad7045a856d49d6ce0a161db45613, data reload: false

query1	914	395	401	395
query2	6265	2145	2069	2069
query3	8687	194	197	194
query4	34429	23654	23337	23337
query5	3493	465	462	462
query6	280	158	163	158
query7	4194	309	308	308
query8	277	214	215	214
query9	9584	2690	2694	2690
query10	468	297	283	283
query11	17878	15200	15167	15167
query12	162	98	104	98
query13	1580	461	439	439
query14	9764	6997	7047	6997
query15	270	165	186	165
query16	7890	461	473	461
query17	1626	615	591	591
query18	1902	310	327	310
query19	266	156	154	154
query20	124	121	113	113
query21	211	104	100	100
query22	4961	4525	4502	4502
query23	35141	34178	34603	34178
query24	11597	2886	2849	2849
query25	637	410	406	406
query26	1619	163	165	163
query27	2583	301	297	297
query28	7693	2443	2414	2414
query29	986	444	424	424
query30	266	151	156	151
query31	1047	825	806	806
query32	96	55	54	54
query33	773	318	309	309
query34	906	509	499	499
query35	862	743	734	734
query36	1108	913	942	913
query37	199	85	87	85
query38	3995	3958	3903	3903
query39	1477	1458	1428	1428
query40	267	99	99	99
query41	48	44	44	44
query42	123	94	93	93
query43	543	494	496	494
query44	1272	798	821	798
query45	194	165	170	165
query46	1156	722	740	722
query47	1983	1850	1870	1850
query48	440	349	347	347
query49	1003	411	408	408
query50	829	425	417	417
query51	7144	7026	7031	7026
query52	101	85	87	85
query53	257	183	183	183
query54	1249	485	489	485
query55	82	76	80	76
query56	288	274	302	274
query57	1266	1158	1135	1135
query58	226	232	235	232
query59	3140	3104	3187	3104
query60	292	266	274	266
query61	108	104	101	101
query62	829	658	698	658
query63	218	185	186	185
query64	4956	649	608	608
query65	3303	3188	3191	3188
query66	1112	307	297	297
query67	15999	15639	15680	15639
query68	4455	570	563	563
query69	482	290	307	290
query70	1185	1129	1118	1118
query71	339	273	268	268
query72	7066	4023	4039	4023
query73	767	345	345	345
query74	10076	8969	8996	8969
query75	3406	2658	2659	2658
query76	3055	901	893	893
query77	584	292	285	285
query78	10590	9754	10054	9754
query79	1667	585	591	585
query80	999	459	458	458
query81	591	239	238	238
query82	661	136	141	136
query83	291	133	133	133
query84	270	79	78	78
query85	1265	312	294	294
query86	392	280	309	280
query87	4423	4340	4291	4291
query88	3230	2572	2434	2434
query89	406	288	289	288
query90	1871	191	188	188
query91	140	110	108	108
query92	61	49	47	47
query93	1834	553	541	541
query94	743	293	300	293
query95	352	249	254	249
query96	617	288	284	284
query97	3316	3161	3181	3161
query98	220	194	197	194
query99	1619	1295	1322	1295
Total cold run time: 301611 ms
Total hot run time: 192540 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2846d83e2c3ad7045a856d49d6ce0a161db45613, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.06	0.07
query4	1.66	0.10	0.10
query5	0.50	0.52	0.52
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.49	0.50
query10	0.55	0.56	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	2.71	2.73	2.75
query15	0.90	0.83	0.83
query16	0.38	0.38	0.38
query17	1.06	1.00	1.08
query18	0.20	0.20	0.19
query19	1.96	1.85	2.07
query20	0.01	0.02	0.01
query21	15.37	0.59	0.57
query22	2.76	2.31	1.94
query23	16.88	1.02	0.78
query24	2.46	0.75	1.14
query25	0.19	0.23	0.04
query26	0.48	0.14	0.13
query27	0.04	0.03	0.03
query28	11.17	1.09	1.06
query29	12.56	3.18	3.18
query30	0.24	0.06	0.06
query31	2.89	0.38	0.38
query32	3.26	0.47	0.46
query33	2.99	3.07	3.01
query34	16.64	4.50	4.53
query35	4.54	4.50	4.52
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.53 s
Total hot run time: 32.52 s

qidaye
qidaye approved these changes Oct 8, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 8, 2024

PR approved by anyone and no changes requested.

csun5285
csun5285 approved these changes Oct 8, 2024
View reviewed changes
Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 9101ed0 into apache:master Oct 8, 2024
24 of 29 checks passed
@airborne12 airborne12 deleted the enhance branch October 8, 2024 12:10
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 9, 2024
@airborne12
[Enhancement](inverted index) apply inverted index when has any (apac…
325cddb 
…he#41547)

## Proposed changes

This PR optimizes the inverted index logic by skipping the execution of
inverted index conditions when none are present. This change improves
performance by avoiding unnecessary inverted index condition flow.
@airborne12 airborne12 mentioned this pull request Oct 9, 2024
Merged
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 9, 2024
@airborne12
[Enhancement](inverted index) apply inverted index when has any (apac…
257ac14 
…he#41547)

## Proposed changes

This PR optimizes the inverted index logic by skipping the execution of
inverted index conditions when none are present. This change improves
performance by avoiding unnecessary inverted index condition flow.
@airborne12 airborne12 mentioned this pull request Oct 9, 2024
Merged
airborne12 added a commit that referenced this pull request Oct 9, 2024
@airborne12
[Enhancement](inverted index) apply inverted index when has any #41547 (
9fe77b3 
#41584)

cherry pick from #41547
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 9, 2024
@airborne12
[Enhancement](inverted index) apply inverted index when has any (apac…
e6e30d5 
…he#41547)

## Proposed changes

This PR optimizes the inverted index logic by skipping the execution of
inverted index conditions when none are present. This change improves
performance by avoiding unnecessary inverted index condition flow.
airborne12 added a commit that referenced this pull request Oct 9, 2024
@airborne12
[Enhancement](inverted index) apply inverted index when has any #41547 (
ba825ef 
#41585)

cherry pick from #41547
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Oct 10, 2024
@airborne12 @eldenmoon
[Enhancement](inverted index) apply inverted index when has any (apac…
dfa4287 
…he#41547)

## Proposed changes

This PR optimizes the inverted index logic by skipping the execution of
inverted index conditions when none are present. This change improves
performance by avoiding unnecessary inverted index condition flow.
cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
@airborne12 @cjj2010
[Enhancement](inverted index) apply inverted index when has any (apac…
fb68520 
…he#41547)

## Proposed changes

This PR optimizes the inverted index logic by skipping the execution of
inverted index conditions when none are present. This change improves
performance by avoiding unnecessary inverted index condition flow.
amorynan pushed a commit to amorynan/doris that referenced this pull request Oct 12, 2024
@airborne12 @amorynan
[Enhancement](inverted index) apply inverted index when has any (apac…
2e96b02 
…he#41547)

## Proposed changes

This PR optimizes the inverted index logic by skipping the execution of
inverted index conditions when none are present. This change improves
performance by avoiding unnecessary inverted index condition flow.
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@csun5285 csun5285 csun5285 approved these changes

@qidaye qidaye qidaye approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@airborne12 @doris-robot @qidaye @csun5285 @wm1581066