Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Chore][inverted index] remove duplicate null bitmap reader in function array index #40907

Merged
merged 1 commit into from
Sep 18, 2024
Merged

[Chore][inverted index] remove duplicate null bitmap reader in function array index #40907

merged 1 commit into from
Sep 18, 2024

Conversation

airborne12
Copy link
Member

Proposed changes

duplicate code in array_contains inverted index evaluate function.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41679 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0bfa400c24b7c520b529a249ddccd9c3657ab901, data reload: false

------ Round 1 ----------------------------------
q1	17667	7786	7301	7301
q2	2040	161	155	155
q3	10679	1091	1140	1091
q4	10484	773	753	753
q5	7735	3154	3101	3101
q6	237	154	150	150
q7	1033	622	604	604
q8	9452	2038	2079	2038
q9	6803	6400	6444	6400
q10	7022	2282	2352	2282
q11	441	252	255	252
q12	412	222	219	219
q13	17793	3041	2984	2984
q14	241	224	236	224
q15	580	515	520	515
q16	687	620	607	607
q17	996	833	820	820
q18	7432	6706	6799	6706
q19	1395	940	992	940
q20	586	313	290	290
q21	4058	3256	3332	3256
q22	1113	1023	991	991
Total cold run time: 108886 ms
Total hot run time: 41679 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7440	7257	7239	7239
q2	326	231	229	229
q3	3065	2984	2976	2976
q4	2050	1832	1829	1829
q5	5636	5616	5707	5616
q6	231	149	154	149
q7	2183	1816	1823	1816
q8	3311	3442	3426	3426
q9	8834	8926	8828	8828
q10	3479	3523	3500	3500
q11	578	492	483	483
q12	834	623	589	589
q13	10033	3209	3185	3185
q14	309	286	273	273
q15	614	538	517	517
q16	718	680	686	680
q17	1856	1578	1581	1578
q18	8120	7962	7711	7711
q19	1745	1624	1576	1576
q20	2121	1892	1880	1880
q21	5497	5333	5391	5333
q22	1115	1089	1067	1067
Total cold run time: 70095 ms
Total hot run time: 60480 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199182 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0bfa400c24b7c520b529a249ddccd9c3657ab901, data reload: false

query1	1279	860	859	859
query2	6399	2083	2112	2083
query3	10807	3942	3935	3935
query4	64725	29364	23425	23425
query5	5078	508	481	481
query6	419	164	170	164
query7	5460	296	299	296
query8	314	226	221	221
query9	8439	2656	2655	2655
query10	449	303	279	279
query11	17292	15165	15589	15165
query12	164	100	108	100
query13	1446	430	415	415
query14	10745	7450	7670	7450
query15	203	173	173	173
query16	6793	505	473	473
query17	1154	636	620	620
query18	1541	314	343	314
query19	213	154	151	151
query20	121	119	113	113
query21	213	103	106	103
query22	4741	4387	4673	4387
query23	34862	34137	33923	33923
query24	6013	2865	2874	2865
query25	470	379	376	376
query26	605	155	151	151
query27	1606	280	277	277
query28	4276	2474	2405	2405
query29	650	413	405	405
query30	216	161	156	156
query31	947	739	752	739
query32	69	59	50	50
query33	382	270	284	270
query34	909	493	471	471
query35	848	710	708	708
query36	1065	916	928	916
query37	143	83	86	83
query38	3998	3929	3929	3929
query39	1449	1408	1402	1402
query40	209	96	92	92
query41	77	42	45	42
query42	118	95	95	95
query43	508	480	473	473
query44	1156	802	789	789
query45	193	162	163	162
query46	1117	775	750	750
query47	1913	1831	1854	1831
query48	463	354	356	354
query49	683	385	420	385
query50	835	403	402	402
query51	7127	6892	7018	6892
query52	96	83	86	83
query53	246	182	185	182
query54	565	445	450	445
query55	77	78	77	77
query56	264	264	246	246
query57	1233	1059	1085	1059
query58	231	227	235	227
query59	3093	3034	3051	3034
query60	299	268	258	258
query61	102	99	98	98
query62	747	678	653	653
query63	217	192	181	181
query64	1320	662	646	646
query65	3223	3193	3193	3193
query66	668	304	305	304
query67	15890	15667	15634	15634
query68	1254	546	545	545
query69	418	277	288	277
query70	1119	1135	1149	1135
query71	315	264	270	264
query72	5243	4015	4064	4015
query73	751	318	326	318
query74	9141	8955	8947	8947
query75	3285	2658	2674	2658
query76	1333	885	874	874
query77	420	316	303	303
query78	9904	9481	9372	9372
query79	1292	874	877	874
query80	836	579	568	568
query81	497	252	252	252
query82	1305	236	228	228
query83	227	163	156	156
query84	276	103	106	103
query85	768	402	358	358
query86	318	315	305	305
query87	4425	4289	4494	4289
query88	4367	4071	4035	4035
query89	383	357	360	357
query90	1651	310	307	307
query91	171	168	179	168
query92	80	74	73	73
query93	937	896	892	892
query94	673	395	403	395
query95	446	420	420	420
query96	497	486	487	486
query97	3166	3106	3243	3106
query98	231	243	235	235
query99	1456	1292	1310	1292
Total cold run time: 306358 ms
Total hot run time: 199182 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0bfa400c24b7c520b529a249ddccd9c3657ab901, data reload: false

query1	0.05	0.05	0.04
query2	0.06	0.03	0.02
query3	0.23	0.06	0.07
query4	1.65	0.09	0.09
query5	0.53	0.50	0.53
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.55	0.50	0.48
query10	0.54	0.54	0.55
query11	0.15	0.11	0.10
query12	0.13	0.10	0.10
query13	0.60	0.59	0.58
query14	3.02	2.99	3.00
query15	0.90	0.82	0.82
query16	0.38	0.37	0.39
query17	1.06	1.04	1.04
query18	0.20	0.20	0.19
query19	1.84	1.81	2.00
query20	0.01	0.01	0.01
query21	15.35	0.60	0.62
query22	2.81	2.67	2.28
query23	17.49	0.78	0.86
query24	2.48	1.47	0.76
query25	0.16	0.15	0.04
query26	0.59	0.14	0.13
query27	0.05	0.04	0.03
query28	10.86	1.08	1.05
query29	12.55	3.24	3.22
query30	0.25	0.06	0.06
query31	2.87	0.40	0.38
query32	3.27	0.46	0.45
query33	3.03	3.01	3.02
query34	16.87	4.40	4.45
query35	4.44	4.41	4.45
query36	0.66	0.49	0.48
query37	0.09	0.05	0.05
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.15	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.27 s
Total hot run time: 32.87 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.33% (9576/25654)
Line Coverage: 28.70% (79126/275704)
Region Coverage: 28.19% (40987/145397)
Branch Coverage: 24.81% (20886/84190)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0bfa400c24b7c520b529a249ddccd9c3657ab901_0bfa400c24b7c520b529a249ddccd9c3657ab901/report/index.html

eldenmoon
eldenmoon approved these changes Sep 18, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

zzzxl1993
zzzxl1993 approved these changes Sep 18, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 merged commit 7cca252 into apache:master Sep 18, 2024
23 of 29 checks passed
@airborne12 airborne12 deleted the remove-useless branch September 18, 2024 07:33
@airborne12 airborne12 mentioned this pull request Sep 18, 2024
Merged
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 18, 2024
@airborne12
[Chore][inverted index] remove duplicate null bitmap reader in functi…
42db126 
…on array index (apache#40907)

## Proposed changes
duplicate code in array_contains inverted index evaluate function.
airborne12 added a commit that referenced this pull request Sep 18, 2024
@airborne12
[Chore][inverted index] remove duplicate null bitmap reader in functi…
609e968 
…on array index #40907 (#40920)

cherry pick from #40907
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Sep 29, 2024
@airborne12
[Chore][inverted index] remove duplicate null bitmap reader in functi…
e321880 
…on array index (apache#40907)

## Proposed changes
duplicate code in array_contains inverted index evaluate function.
@airborne12 airborne12 mentioned this pull request Sep 29, 2024
Merged
airborne12 added a commit that referenced this pull request Sep 29, 2024
@airborne12
[Chore][inverted index] remove duplicate null bitmap reader in functi…
ddbb436 
…on array index #40907 (#41444)

cherry pick from #40907
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eldenmoon eldenmoon eldenmoon approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@airborne12 @doris-robot @zzzxl1993 @eldenmoon