Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cluster key) cluster key support vertical_segment_writer #38538

Merged
merged 4 commits into from
Aug 19, 2024

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Jul 30, 2024

The mow cluster key does not support vertical_segment_writer.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi mymeiyi force-pushed the cluster-key-ver-segment0-writer-2 branch from 2f6dbff to ff8e376 Compare July 30, 2024 11:05
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Jul 30, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41937 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff8e376453cba69a758eab881c658c1912d52f5a, data reload: false

------ Round 1 ----------------------------------
q1	19068	4223	4113	4113
q2	2021	209	211	209
q3	10446	1319	1326	1319
q4	12532	885	917	885
q5	7665	3001	3020	3001
q6	220	140	142	140
q7	1066	646	654	646
q8	9708	1803	1996	1803
q9	9607	6728	6721	6721
q10	8839	3846	3934	3846
q11	1505	261	260	260
q12	1223	243	239	239
q13	22063	2960	2990	2960
q14	275	251	243	243
q15	525	504	500	500
q16	527	399	393	393
q17	973	878	872	872
q18	7967	7249	7205	7205
q19	1535	1219	1217	1217
q20	555	333	350	333
q21	5327	4752	4758	4752
q22	358	280	290	280
Total cold run time: 124005 ms
Total hot run time: 41937 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4086	4028	4229	4028
q2	333	225	239	225
q3	3224	3179	3230	3179
q4	2007	1951	1968	1951
q5	5439	5510	5484	5484
q6	222	134	135	134
q7	2166	1801	1829	1801
q8	3315	3380	3350	3350
q9	9208	9131	8939	8939
q10	3928	4060	4017	4017
q11	539	478	470	470
q12	776	618	586	586
q13	16458	3149	3149	3149
q14	301	273	266	266
q15	522	486	483	483
q16	472	415	416	415
q17	1774	1712	1737	1712
q18	8380	7663	7773	7663
q19	1769	1729	1720	1720
q20	2069	1853	1846	1846
q21	5658	5333	5434	5333
q22	549	459	467	459
Total cold run time: 73195 ms
Total hot run time: 57210 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170600 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff8e376453cba69a758eab881c658c1912d52f5a, data reload: false

query1	908	385	380	380
query2	6375	1641	1672	1641
query3	6645	221	222	221
query4	19937	17518	17060	17060
query5	3541	541	543	541
query6	271	185	165	165
query7	4599	297	310	297
query8	260	204	191	191
query9	8527	2397	2376	2376
query10	451	284	282	282
query11	10577	10049	10111	10049
query12	119	93	89	89
query13	1652	381	367	367
query14	9915	7295	7338	7295
query15	205	169	177	169
query16	7001	456	463	456
query17	937	569	550	550
query18	1931	290	281	281
query19	213	152	144	144
query20	96	90	86	86
query21	205	107	99	99
query22	4273	4121	4072	4072
query23	33962	33734	33543	33543
query24	10480	3130	3050	3050
query25	705	422	409	409
query26	1823	153	155	153
query27	3089	291	288	288
query28	7652	2028	2019	2019
query29	1308	450	443	443
query30	235	160	156	156
query31	964	813	795	795
query32	106	62	64	62
query33	703	338	364	338
query34	934	509	506	506
query35	866	783	807	783
query36	1079	884	863	863
query37	267	79	79	79
query38	2842	2785	2786	2785
query39	858	833	820	820
query40	247	119	114	114
query41	48	43	44	43
query42	128	101	101	101
query43	489	436	408	408
query44	1192	741	750	741
query45	214	177	178	177
query46	1094	817	818	817
query47	1772	1705	1684	1684
query48	371	292	289	289
query49	919	437	439	437
query50	912	451	442	442
query51	6810	6740	6607	6607
query52	107	92	92	92
query53	261	186	184	184
query54	636	449	464	449
query55	79	75	77	75
query56	265	265	254	254
query57	1101	1067	1056	1056
query58	281	284	295	284
query59	2482	2537	2369	2369
query60	292	276	291	276
query61	109	96	99	96
query62	881	667	655	655
query63	213	192	193	192
query64	5655	1940	1925	1925
query65	3221	3139	3101	3101
query66	1328	391	331	331
query67	15204	14892	14854	14854
query68	4264	564	569	564
query69	633	350	329	329
query70	1141	1066	1084	1066
query71	433	293	278	278
query72	7144	2680	2572	2572
query73	781	326	331	326
query74	5996	5713	5656	5656
query75	3390	2768	2764	2764
query76	2247	1343	1404	1343
query77	576	318	309	309
query78	9446	8917	8917	8917
query79	1690	544	542	542
query80	997	525	536	525
query81	556	223	228	223
query82	1030	128	126	126
query83	260	175	175	175
query84	274	81	78	78
query85	1225	321	365	321
query86	404	320	300	300
query87	3241	3115	3124	3115
query88	2986	2444	2432	2432
query89	396	300	298	298
query90	1690	203	202	202
query91	140	116	117	116
query92	63	55	55	55
query93	1451	624	622	622
query94	790	312	305	305
query95	400	277	281	277
query96	597	280	289	280
query97	3197	3031	3067	3031
query98	225	209	206	206
query99	1694	1309	1291	1291
Total cold run time: 262179 ms
Total hot run time: 170600 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff8e376453cba69a758eab881c658c1912d52f5a, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.06
query5	0.49	0.47	0.49
query6	1.14	0.72	0.71
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.59	0.51	0.54
query10	0.58	0.58	0.57
query11	0.16	0.11	0.12
query12	0.15	0.13	0.13
query13	0.62	0.60	0.60
query14	0.78	0.78	0.82
query15	0.92	0.87	0.87
query16	0.34	0.36	0.35
query17	1.00	0.99	1.00
query18	0.21	0.21	0.20
query19	1.84	1.76	1.80
query20	0.02	0.01	0.01
query21	15.41	0.79	0.66
query22	3.97	9.10	1.04
query23	17.95	1.38	1.43
query24	2.29	0.22	0.22
query25	0.19	0.08	0.08
query26	0.31	0.21	0.21
query27	0.46	0.22	0.23
query28	13.19	1.00	0.97
query29	12.51	3.32	3.29
query30	0.26	0.06	0.06
query31	2.86	0.41	0.41
query32	3.24	0.50	0.48
query33	2.97	2.97	2.94
query34	15.40	4.27	4.25
query35	4.29	4.30	4.33
query36	0.68	0.47	0.50
query37	0.18	0.16	0.15
query38	0.16	0.14	0.16
query39	0.04	0.03	0.04
query40	0.16	0.13	0.13
query41	0.10	0.04	0.04
query42	0.06	0.06	0.05
query43	0.05	0.05	0.04
Total cold run time: 107.64 s
Total hot run time: 29.85 s

@mymeiyi mymeiyi force-pushed the cluster-key-ver-segment0-writer-2 branch from ff8e376 to ee89809 Compare August 5, 2024 11:35
@github-actions github-actions bot added the doing label Aug 5, 2024
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Aug 5, 2024

run buildall

Copy link
Contributor

github-actions bot commented Aug 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41437 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee89809b233c0df76ef3aa14d509c1b9bbf454d2, data reload: false

------ Round 1 ----------------------------------
q1	17704	4114	4060	4060
q2	2019	203	217	203
q3	10454	1271	1337	1271
q4	10176	863	960	863
q5	7644	3053	2975	2975
q6	220	141	141	141
q7	1053	630	628	628
q8	9446	1731	1942	1731
q9	8439	6592	6592	6592
q10	8735	3841	3819	3819
q11	427	253	246	246
q12	413	228	223	223
q13	17767	2930	2973	2930
q14	279	245	255	245
q15	532	473	483	473
q16	517	383	392	383
q17	974	932	924	924
q18	8057	7259	7192	7192
q19	1443	1220	1223	1220
q20	568	330	332	330
q21	5382	4782	4710	4710
q22	347	278	287	278
Total cold run time: 112596 ms
Total hot run time: 41437 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4097	4014	4031	4014
q2	327	227	214	214
q3	3002	2987	3130	2987
q4	2036	2047	1980	1980
q5	5601	5530	5458	5458
q6	220	131	140	131
q7	2110	1789	1855	1789
q8	3318	3365	3324	3324
q9	8710	8650	8821	8650
q10	3967	4065	3952	3952
q11	542	463	474	463
q12	783	647	599	599
q13	16120	3151	3161	3151
q14	297	291	293	291
q15	526	481	479	479
q16	471	421	414	414
q17	1777	1721	1735	1721
q18	8252	7820	7643	7643
q19	1709	1702	1697	1697
q20	2095	1867	1839	1839
q21	5815	5596	5270	5270
q22	553	462	456	456
Total cold run time: 72328 ms
Total hot run time: 56522 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169991 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee89809b233c0df76ef3aa14d509c1b9bbf454d2, data reload: false

query1	910	384	368	368
query2	6468	1731	1746	1731
query3	6655	212	225	212
query4	20103	17689	17405	17405
query5	3615	530	533	530
query6	284	194	189	189
query7	4604	305	287	287
query8	251	192	195	192
query9	8521	2412	2413	2412
query10	453	265	259	259
query11	10464	9966	9986	9966
query12	126	87	87	87
query13	1615	376	376	376
query14	8692	7118	6883	6883
query15	194	162	167	162
query16	6889	470	460	460
query17	934	570	535	535
query18	1844	276	288	276
query19	200	144	141	141
query20	98	92	87	87
query21	204	102	96	96
query22	4146	3920	3878	3878
query23	33768	33819	33717	33717
query24	10310	3050	3063	3050
query25	682	395	417	395
query26	1718	155	154	154
query27	2951	280	291	280
query28	7562	2061	2039	2039
query29	1306	439	426	426
query30	237	158	158	158
query31	935	762	766	762
query32	101	53	63	53
query33	673	308	333	308
query34	918	517	511	511
query35	878	762	765	762
query36	1035	914	899	899
query37	280	91	80	80
query38	3022	2836	2845	2836
query39	858	834	820	820
query40	252	116	112	112
query41	47	48	45	45
query42	127	98	102	98
query43	501	440	448	440
query44	1173	743	751	743
query45	223	179	179	179
query46	1090	817	784	784
query47	1811	1716	1718	1716
query48	359	292	300	292
query49	979	420	427	420
query50	884	443	428	428
query51	6892	6579	6618	6579
query52	117	93	94	93
query53	251	190	188	188
query54	622	461	449	449
query55	77	76	76	76
query56	270	257	251	251
query57	1149	1044	1017	1017
query58	274	289	274	274
query59	2557	2492	2585	2492
query60	285	284	277	277
query61	94	94	93	93
query62	879	654	669	654
query63	216	182	181	181
query64	5693	1902	1862	1862
query65	3166	3105	3159	3105
query66	1301	333	338	333
query67	15350	15038	14924	14924
query68	4308	570	572	570
query69	645	357	298	298
query70	1133	1062	1060	1060
query71	432	289	283	283
query72	7159	2698	2486	2486
query73	772	325	329	325
query74	6019	5715	5634	5634
query75	3384	2745	2723	2723
query76	2352	1185	1256	1185
query77	555	360	318	318
query78	9496	8865	8984	8865
query79	2256	533	544	533
query80	926	500	506	500
query81	578	226	225	225
query82	794	134	130	130
query83	256	169	175	169
query84	260	80	79	79
query85	1165	317	305	305
query86	471	275	275	275
query87	3329	3122	3114	3114
query88	3807	2407	2402	2402
query89	386	301	282	282
query90	1759	195	195	195
query91	128	100	101	100
query92	60	106	52	52
query93	2175	618	615	615
query94	789	294	300	294
query95	377	260	268	260
query96	597	280	280	280
query97	3268	3065	3139	3065
query98	232	207	188	188
query99	1625	1297	1294	1294
Total cold run time: 262636 ms
Total hot run time: 169991 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee89809b233c0df76ef3aa14d509c1b9bbf454d2, data reload: false

query1	0.05	0.04	0.05
query2	0.07	0.04	0.04
query3	0.23	0.04	0.04
query4	1.69	0.07	0.07
query5	0.50	0.49	0.47
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.57	0.50	0.51
query10	0.57	0.56	0.57
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.61	0.61	0.59
query14	0.77	0.81	0.79
query15	0.92	0.87	0.85
query16	0.35	0.35	0.34
query17	0.99	1.02	1.04
query18	0.22	0.22	0.22
query19	1.86	1.75	1.77
query20	0.02	0.01	0.02
query21	15.41	0.75	0.66
query22	3.98	7.82	1.14
query23	17.83	1.36	1.35
query24	2.30	0.22	0.21
query25	0.18	0.08	0.07
query26	0.32	0.23	0.21
query27	0.46	0.23	0.23
query28	13.17	0.98	0.98
query29	12.54	3.32	3.30
query30	0.25	0.06	0.05
query31	2.86	0.41	0.41
query32	3.24	0.49	0.49
query33	2.98	2.92	2.94
query34	15.45	4.29	4.27
query35	4.30	4.32	4.32
query36	0.68	0.48	0.46
query37	0.19	0.16	0.16
query38	0.16	0.16	0.16
query39	0.04	0.03	0.04
query40	0.16	0.13	0.14
query41	0.10	0.04	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 107.64 s
Total hot run time: 29.96 s

zhannngchen
zhannngchen previously approved these changes Aug 7, 2024
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 7, 2024
Copy link
Contributor

github-actions bot commented Aug 7, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Aug 7, 2024

PR approved by anyone and no changes requested.

@zhannngchen zhannngchen requested a review from dataroaring August 7, 2024 07:15
@mymeiyi mymeiyi force-pushed the cluster-key-ver-segment0-writer-2 branch from ee89809 to 3f6d8dc Compare August 8, 2024 02:39
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Aug 8, 2024

run buildall

Copy link
Contributor

github-actions bot commented Aug 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39835 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3f6d8dc23c4685fe5c596d866079c4e038359ba8, data reload: false

------ Round 1 ----------------------------------
q1	17599	4404	4294	4294
q2	2011	178	172	172
q3	10493	1170	1126	1126
q4	10144	750	824	750
q5	7495	2551	2480	2480
q6	229	145	142	142
q7	990	624	585	585
q8	9219	1928	1936	1928
q9	8616	6598	6545	6545
q10	7038	2238	2190	2190
q11	471	240	239	239
q12	411	233	229	229
q13	17822	3030	2971	2971
q14	279	236	250	236
q15	513	484	499	484
q16	507	385	398	385
q17	993	685	756	685
q18	8186	7492	7394	7394
q19	5778	1138	1005	1005
q20	665	324	346	324
q21	5363	4640	4640	4640
q22	1134	1031	1032	1031
Total cold run time: 115956 ms
Total hot run time: 39835 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4559	4298	4283	4283
q2	382	267	271	267
q3	2871	2600	2741	2600
q4	2006	1751	1701	1701
q5	5603	5513	5449	5449
q6	219	135	131	131
q7	2114	1765	1772	1765
q8	3317	3458	3511	3458
q9	8733	8836	8793	8793
q10	3525	3300	3173	3173
q11	612	483	499	483
q12	779	667	639	639
q13	15829	3151	3210	3151
q14	311	304	284	284
q15	521	512	467	467
q16	483	439	452	439
q17	1854	1547	1550	1547
q18	8086	7842	7757	7757
q19	1777	1532	1555	1532
q20	2164	1906	1897	1897
q21	5582	5463	5134	5134
q22	1128	1050	1050	1050
Total cold run time: 72455 ms
Total hot run time: 56000 ms

@mymeiyi mymeiyi force-pushed the cluster-key-ver-segment0-writer-2 branch from 3f6d8dc to ecbed95 Compare August 15, 2024 08:27
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 15, 2024
@mymeiyi mymeiyi force-pushed the cluster-key-ver-segment0-writer-2 branch from ecbed95 to 6bab567 Compare August 15, 2024 08:30
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38248 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 664ac9170d5e676a8b6e60310876b0cec25a0d64, data reload: false

------ Round 1 ----------------------------------
q1	18201	4490	4382	4382
q2	2207	238	234	234
q3	10448	1247	1085	1085
q4	10170	725	791	725
q5	7823	2886	2741	2741
q6	263	157	162	157
q7	989	652	658	652
q8	9387	2115	2080	2080
q9	7247	6561	6573	6561
q10	7088	2208	2202	2202
q11	500	273	280	273
q12	470	265	257	257
q13	17781	2997	3018	2997
q14	302	257	264	257
q15	577	541	539	539
q16	530	425	416	416
q17	992	637	800	637
q18	7495	6929	6780	6780
q19	6693	1151	1096	1096
q20	722	355	358	355
q21	3892	2876	2777	2777
q22	1134	1045	1053	1045
Total cold run time: 114911 ms
Total hot run time: 38248 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4562	4308	4301	4301
q2	439	321	323	321
q3	2910	2679	2674	2674
q4	1938	1682	1643	1643
q5	5404	5370	5407	5370
q6	233	147	150	147
q7	2118	1690	1730	1690
q8	3241	3330	3347	3330
q9	8443	8418	8409	8409
q10	3408	3191	3191	3191
q11	606	532	530	530
q12	814	606	604	604
q13	17283	3007	3014	3007
q14	331	294	296	294
q15	564	519	528	519
q16	508	459	447	447
q17	1826	1479	1486	1479
q18	8063	7547	7570	7547
q19	2840	1666	1530	1530
q20	2085	1828	1815	1815
q21	14125	5126	5052	5052
q22	1128	1058	1017	1017
Total cold run time: 82869 ms
Total hot run time: 54917 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190803 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 664ac9170d5e676a8b6e60310876b0cec25a0d64, data reload: false

query1	981	393	398	393
query2	6755	2069	1910	1910
query3	6661	225	237	225
query4	33144	23594	23348	23348
query5	4399	641	633	633
query6	297	183	189	183
query7	4603	317	310	310
query8	413	365	361	361
query9	8790	2509	2490	2490
query10	491	324	317	317
query11	17786	14936	15107	14936
query12	177	125	122	122
query13	1700	430	412	412
query14	9818	7312	6899	6899
query15	269	193	195	193
query16	8128	509	498	498
query17	1672	598	594	594
query18	2176	333	343	333
query19	347	171	208	171
query20	141	133	134	133
query21	255	139	143	139
query22	4351	4125	4014	4014
query23	33991	33343	33319	33319
query24	11424	2893	3017	2893
query25	639	423	419	419
query26	771	179	179	179
query27	2101	300	308	300
query28	6682	2172	2159	2159
query29	794	441	443	441
query30	341	187	182	182
query31	1060	848	818	818
query32	124	78	80	78
query33	833	339	343	339
query34	932	506	557	506
query35	871	777	772	772
query36	1211	995	970	970
query37	161	105	100	100
query38	3964	3857	3869	3857
query39	1536	1479	1483	1479
query40	236	156	153	153
query41	141	139	136	136
query42	138	115	121	115
query43	547	512	516	512
query44	1294	801	791	791
query45	227	191	199	191
query46	1130	776	829	776
query47	1904	1833	1825	1825
query48	426	338	334	334
query49	1208	586	613	586
query50	862	475	479	475
query51	6819	6848	6804	6804
query52	118	111	112	111
query53	307	229	229	229
query54	970	511	501	501
query55	91	90	92	90
query56	338	324	315	315
query57	1190	1095	1128	1095
query58	317	309	311	309
query59	3042	2897	2914	2897
query60	358	330	335	330
query61	148	143	143	143
query62	935	716	695	695
query63	271	231	238	231
query64	5240	2439	1873	1873
query65	3217	3146	3176	3146
query66	1171	690	679	679
query67	15680	14939	14963	14939
query68	6251	600	609	600
query69	791	438	361	361
query70	1222	1156	1112	1112
query71	574	316	323	316
query72	7596	2306	2117	2117
query73	850	375	362	362
query74	9389	8869	8879	8869
query75	5524	2726	2701	2701
query76	4912	1020	997	997
query77	934	446	446	446
query78	9937	9107	9026	9026
query79	4689	564	557	557
query80	2239	613	609	609
query81	625	262	258	258
query82	973	165	159	159
query83	343	223	221	221
query84	299	105	104	104
query85	1520	346	345	345
query86	477	328	315	315
query87	4377	4283	4280	4280
query88	5007	2531	2524	2524
query89	472	334	332	332
query90	2013	235	230	230
query91	156	127	127	127
query92	91	74	75	74
query93	4984	562	559	559
query94	955	329	319	319
query95	397	292	292	292
query96	618	291	286	286
query97	3267	3119	3071	3071
query98	245	232	224	224
query99	1608	1309	1304	1304
Total cold run time: 312449 ms
Total hot run time: 190803 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 664ac9170d5e676a8b6e60310876b0cec25a0d64, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.08	0.07
query5	0.52	0.50	0.50
query6	1.14	0.74	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.06
query9	0.56	0.50	0.49
query10	0.54	0.55	0.54
query11	0.16	0.12	0.12
query12	0.16	0.12	0.13
query13	0.62	0.61	0.59
query14	0.78	0.79	0.78
query15	0.86	0.82	0.83
query16	0.38	0.37	0.38
query17	0.98	0.99	1.06
query18	0.23	0.22	0.21
query19	1.77	1.69	1.76
query20	0.02	0.01	0.01
query21	15.44	0.82	0.65
query22	4.65	7.32	1.96
query23	18.32	1.36	1.27
query24	2.10	0.22	0.23
query25	0.15	0.09	0.08
query26	0.30	0.22	0.22
query27	0.45	0.22	0.24
query28	13.30	1.04	1.02
query29	12.64	3.30	3.26
query30	0.38	0.19	0.20
query31	2.81	0.40	0.40
query32	3.23	0.49	0.48
query33	2.92	2.93	3.00
query34	16.79	4.37	4.36
query35	4.37	4.43	4.45
query36	0.68	0.49	0.51
query37	0.21	0.17	0.19
query38	0.17	0.17	0.18
query39	0.06	0.05	0.06
query40	0.18	0.15	0.15
query41	0.12	0.08	0.07
query42	0.08	0.06	0.07
query43	0.07	0.06	0.06
Total cold run time: 110.23 s
Total hot run time: 31.12 s

@mymeiyi mymeiyi force-pushed the cluster-key-ver-segment0-writer-2 branch from 664ac91 to 488bb20 Compare August 16, 2024 04:09
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Aug 16, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38278 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 488bb20cb884f19217631a7a2bda37548396c5c4, data reload: false

------ Round 1 ----------------------------------
q1	18165	4525	4430	4430
q2	2499	230	207	207
q3	11404	1258	1082	1082
q4	10805	773	753	753
q5	7833	2898	2886	2886
q6	275	148	150	148
q7	995	625	628	625
q8	9351	2073	2049	2049
q9	7354	6538	6522	6522
q10	7092	2326	2168	2168
q11	469	260	266	260
q12	419	243	254	243
q13	17778	3010	3014	3010
q14	297	246	258	246
q15	573	508	507	507
q16	527	401	404	401
q17	979	708	702	702
q18	7375	6863	6755	6755
q19	5739	1078	1055	1055
q20	765	351	365	351
q21	3849	2943	2860	2860
q22	1120	1018	1020	1018
Total cold run time: 115663 ms
Total hot run time: 38278 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4569	4254	4272	4254
q2	425	299	294	294
q3	2884	2631	2628	2628
q4	1923	1638	1617	1617
q5	5371	5376	5376	5376
q6	231	142	143	142
q7	2078	1704	1704	1704
q8	3203	3343	3323	3323
q9	8414	8437	8398	8398
q10	3433	3166	3177	3166
q11	615	506	531	506
q12	807	634	628	628
q13	16406	2991	2967	2967
q14	319	300	289	289
q15	559	507	512	507
q16	475	438	436	436
q17	1797	1505	1473	1473
q18	7682	7513	7392	7392
q19	1724	1573	1559	1559
q20	2043	1834	1826	1826
q21	9445	5194	5031	5031
q22	1121	1049	1026	1026
Total cold run time: 75524 ms
Total hot run time: 54542 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191127 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 488bb20cb884f19217631a7a2bda37548396c5c4, data reload: false

query1	949	391	396	391
query2	6745	1990	1966	1966
query3	6666	235	240	235
query4	34103	23708	23226	23226
query5	4336	696	703	696
query6	320	208	208	208
query7	4628	307	312	307
query8	406	362	359	359
query9	8700	2499	2495	2495
query10	468	345	315	315
query11	17886	14956	15046	14956
query12	164	122	127	122
query13	1678	418	407	407
query14	9794	7097	7411	7097
query15	341	189	194	189
query16	7886	516	474	474
query17	1760	612	587	587
query18	1812	336	345	336
query19	349	166	163	163
query20	138	133	135	133
query21	249	144	136	136
query22	4393	4176	4066	4066
query23	34162	33349	33423	33349
query24	11167	2920	2863	2863
query25	633	411	419	411
query26	1208	181	180	180
query27	2407	296	301	296
query28	6944	2139	2139	2139
query29	798	481	436	436
query30	320	186	180	180
query31	1069	785	862	785
query32	122	77	79	77
query33	805	343	333	333
query34	902	511	501	501
query35	920	785	761	761
query36	1138	977	967	967
query37	162	103	103	103
query38	4158	3848	3877	3848
query39	1549	1466	1464	1464
query40	236	153	155	153
query41	139	137	133	133
query42	138	117	118	117
query43	543	510	531	510
query44	1282	783	790	783
query45	229	199	196	196
query46	1140	736	792	736
query47	1969	1850	1884	1850
query48	426	331	344	331
query49	1219	583	578	578
query50	855	462	463	462
query51	6891	6739	6759	6739
query52	119	108	111	108
query53	297	228	228	228
query54	1066	497	496	496
query55	89	91	88	88
query56	325	302	310	302
query57	1198	1152	1117	1117
query58	315	300	301	300
query59	3021	2933	2808	2808
query60	343	325	317	317
query61	149	145	149	145
query62	875	681	745	681
query63	260	225	232	225
query64	5297	2441	1855	1855
query65	3225	3192	3173	3173
query66	1179	683	672	672
query67	15593	14881	14816	14816
query68	6245	577	581	577
query69	715	423	337	337
query70	1249	1153	1191	1153
query71	542	308	314	308
query72	7394	2323	2090	2090
query73	842	353	355	353
query74	9237	8836	8877	8836
query75	4550	2744	2774	2744
query76	4026	1041	963	963
query77	823	431	438	431
query78	9680	9814	10220	9814
query79	8318	566	568	566
query80	1049	603	622	603
query81	676	253	262	253
query82	629	157	156	156
query83	341	209	209	209
query84	295	97	101	97
query85	1443	362	352	352
query86	481	340	323	323
query87	4398	4190	4315	4190
query88	4464	2461	2456	2456
query89	480	325	322	322
query90	1914	237	231	231
query91	157	126	126	126
query92	92	75	76	75
query93	3869	550	551	550
query94	890	335	334	334
query95	380	296	302	296
query96	622	284	282	282
query97	3248	3091	3064	3064
query98	247	226	234	226
query99	1611	1354	1334	1334
Total cold run time: 311560 ms
Total hot run time: 191127 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 488bb20cb884f19217631a7a2bda37548396c5c4, data reload: false

query1	0.04	0.05	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.08	0.08
query5	0.51	0.49	0.50
query6	1.15	0.73	0.74
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.55	0.50	0.50
query10	0.55	0.55	0.54
query11	0.15	0.12	0.12
query12	0.16	0.13	0.14
query13	0.61	0.63	0.59
query14	0.79	0.78	0.78
query15	0.86	0.81	0.82
query16	0.36	0.39	0.37
query17	1.05	1.03	1.04
query18	0.23	0.22	0.26
query19	1.96	1.88	1.89
query20	0.01	0.02	0.01
query21	15.39	0.83	0.67
query22	4.16	7.67	1.70
query23	18.28	1.35	1.28
query24	2.08	0.24	0.22
query25	0.17	0.09	0.09
query26	0.32	0.21	0.21
query27	0.46	0.23	0.23
query28	13.31	1.02	1.01
query29	12.61	3.29	3.32
query30	0.43	0.23	0.23
query31	2.80	0.41	0.39
query32	3.25	0.49	0.50
query33	2.95	2.97	2.97
query34	16.94	4.33	4.36
query35	4.44	4.44	4.43
query36	0.67	0.50	0.50
query37	0.21	0.17	0.18
query38	0.18	0.17	0.18
query39	0.07	0.06	0.06
query40	0.19	0.14	0.14
query41	0.12	0.07	0.07
query42	0.09	0.08	0.07
query43	0.07	0.07	0.07
Total cold run time: 110.2 s
Total hot run time: 31.26 s

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Aug 16, 2024

run p0

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring previously approved these changes Aug 18, 2024
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mymeiyi mymeiyi force-pushed the cluster-key-ver-segment0-writer-2 branch from 488bb20 to 1e5a4a4 Compare August 19, 2024 10:32
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Aug 19, 2024

run buildall

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 19, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 38349 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e5a4a40dff465870daef8365a704852c7b06143, data reload: false

------ Round 1 ----------------------------------
q1	18466	4580	4397	4397
q2	2061	226	204	204
q3	10419	1133	1144	1133
q4	10155	737	792	737
q5	7777	2848	2806	2806
q6	265	158	158	158
q7	1027	657	647	647
q8	9394	2074	2089	2074
q9	7243	6554	6561	6554
q10	7047	2285	2216	2216
q11	488	282	271	271
q12	439	260	258	258
q13	17797	3032	3018	3018
q14	282	252	260	252
q15	551	519	517	517
q16	527	401	411	401
q17	988	657	660	657
q18	7519	6857	6685	6685
q19	6607	1111	1097	1097
q20	711	356	365	356
q21	3880	2913	2954	2913
q22	1096	998	1012	998
Total cold run time: 114739 ms
Total hot run time: 38349 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4561	4340	4272	4272
q2	398	304	305	304
q3	2856	2653	2668	2653
q4	1921	1592	1586	1586
q5	5406	5401	5364	5364
q6	236	147	145	145
q7	2049	1685	1705	1685
q8	3221	3360	3364	3360
q9	8579	8458	8441	8441
q10	3436	3160	3178	3160
q11	652	538	528	528
q12	819	656	654	654
q13	17576	3016	2982	2982
q14	322	298	296	296
q15	579	529	530	529
q16	513	446	444	444
q17	1820	1510	1494	1494
q18	7601	7558	7442	7442
q19	1720	1608	1663	1608
q20	2066	1857	1860	1857
q21	10347	5065	4897	4897
q22	1135	1044	1021	1021
Total cold run time: 77813 ms
Total hot run time: 54722 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190621 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e5a4a40dff465870daef8365a704852c7b06143, data reload: false

query1	999	400	392	392
query2	6744	2078	1927	1927
query3	6677	226	234	226
query4	29268	23243	23383	23243
query5	4386	729	703	703
query6	324	219	211	211
query7	4626	334	326	326
query8	482	429	441	429
query9	8876	2538	2508	2508
query10	506	357	353	353
query11	18167	15073	15099	15073
query12	185	139	137	137
query13	1712	467	464	464
query14	10083	6757	6896	6757
query15	288	189	195	189
query16	8078	482	486	482
query17	1609	606	592	592
query18	2129	342	332	332
query19	279	192	169	169
query20	145	137	140	137
query21	248	148	143	143
query22	4229	4099	3963	3963
query23	34234	33548	33410	33410
query24	11247	2970	2886	2886
query25	657	422	428	422
query26	1218	179	181	179
query27	2335	296	301	296
query28	7170	2150	2112	2112
query29	860	463	456	456
query30	346	186	181	181
query31	1052	844	825	825
query32	120	80	82	80
query33	818	347	349	347
query34	920	507	505	505
query35	886	764	783	764
query36	1150	967	990	967
query37	176	105	100	100
query38	4113	3850	3850	3850
query39	1514	1478	1460	1460
query40	239	155	153	153
query41	138	136	137	136
query42	138	117	117	117
query43	561	516	509	509
query44	1274	786	808	786
query45	227	199	200	199
query46	1149	773	761	761
query47	1924	1849	1858	1849
query48	416	337	337	337
query49	1250	584	591	584
query50	865	474	469	469
query51	6932	6684	6817	6684
query52	123	108	111	108
query53	305	232	228	228
query54	954	518	502	502
query55	89	87	87	87
query56	332	317	312	312
query57	1207	1131	1124	1124
query58	331	298	326	298
query59	2996	2850	2800	2800
query60	352	322	329	322
query61	149	144	149	144
query62	877	686	697	686
query63	264	230	229	229
query64	5318	2390	1887	1887
query65	3267	3185	3180	3180
query66	1203	672	666	666
query67	15472	14956	14926	14926
query68	6588	590	608	590
query69	726	437	330	330
query70	1277	1085	1142	1085
query71	571	315	315	315
query72	7686	2350	2115	2115
query73	869	354	359	354
query74	9210	8884	8974	8884
query75	4826	2756	2755	2755
query76	5052	1114	1004	1004
query77	933	452	437	437
query78	9683	9067	8944	8944
query79	8096	558	558	558
query80	1181	625	600	600
query81	624	267	266	266
query82	354	160	165	160
query83	396	221	217	217
query84	297	102	99	99
query85	1079	411	358	358
query86	372	314	330	314
query87	4465	4287	4300	4287
query88	4886	2487	2505	2487
query89	544	333	326	326
query90	2121	244	238	238
query91	162	128	133	128
query92	93	79	75	75
query93	6091	560	546	546
query94	1044	293	321	293
query95	410	298	296	296
query96	635	283	284	283
query97	3254	3091	3092	3091
query98	237	228	227	227
query99	1652	1280	1303	1280
Total cold run time: 312622 ms
Total hot run time: 190621 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e5a4a40dff465870daef8365a704852c7b06143, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.66	0.08	0.07
query5	0.50	0.50	0.49
query6	1.13	0.73	0.72
query7	0.03	0.02	0.01
query8	0.06	0.04	0.05
query9	0.56	0.47	0.48
query10	0.55	0.54	0.53
query11	0.17	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.59	0.59
query14	0.78	0.81	0.79
query15	0.87	0.83	0.84
query16	0.36	0.36	0.38
query17	0.98	0.98	0.97
query18	0.21	0.21	0.20
query19	1.89	1.80	1.81
query20	0.01	0.01	0.01
query21	15.40	0.68	0.67
query22	3.95	7.39	2.36
query23	18.23	1.41	1.29
query24	2.16	0.23	0.23
query25	0.15	0.08	0.08
query26	0.28	0.19	0.19
query27	0.08	0.09	0.09
query28	13.20	1.04	1.02
query29	12.64	3.35	3.26
query30	0.44	0.25	0.24
query31	2.80	0.41	0.41
query32	3.24	0.48	0.49
query33	2.93	2.97	2.99
query34	16.95	4.38	4.37
query35	4.44	4.50	4.45
query36	0.66	0.50	0.50
query37	0.20	0.17	0.18
query38	0.18	0.17	0.17
query39	0.07	0.06	0.06
query40	0.19	0.14	0.15
query41	0.11	0.06	0.06
query42	0.07	0.06	0.07
query43	0.06	0.06	0.06
Total cold run time: 109.32 s
Total hot run time: 31.6 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 74aa0ca into apache:master Aug 19, 2024
27 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
The mow cluster key does not support vertical_segment_writer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.3-merged doing reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants