Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](plan) only scan node with limit and no predicate can reduce to 1 instance (#31342) #31606

Merged
merged 1 commit into from
Mar 1, 2024

Conversation

morningman
Copy link
Contributor

bp #31342

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Feb 29, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 50005 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5328f20f213d289aa6fb1f400b08a44047e8612b, data reload: false

------ Round 1 ----------------------------------
q1	17720	4393	4370	4370
q2	2030	147	141	141
q3	10466	1910	1945	1910
q4	10299	1221	1313	1221
q5	8848	3957	3964	3957
q6	226	122	123	122
q7	2012	1584	1565	1565
q8	9280	2687	2694	2687
q9	11145	10707	10741	10707
q10	8639	3520	3459	3459
q11	420	232	242	232
q12	458	295	305	295
q13	18355	3946	4024	3946
q14	360	311	337	311
q15	509	449	450	449
q16	714	605	587	587
q17	1107	935	953	935
q18	7175	6842	6926	6842
q19	1690	1563	1539	1539
q20	517	298	294	294
q21	4477	4041	4094	4041
q22	506	395	410	395
Total cold run time: 116953 ms
Total hot run time: 50005 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4326	4265	4288	4265
q2	318	217	223	217
q3	4196	4152	4152	4152
q4	2755	2716	2716	2716
q5	7340	7192	7194	7192
q6	234	122	117	117
q7	3179	2871	2806	2806
q8	4321	4474	4440	4440
q9	17537	17409	17281	17281
q10	4210	4225	4245	4225
q11	754	693	674	674
q12	1031	843	828	828
q13	6797	3747	3722	3722
q14	449	414	422	414
q15	498	453	452	452
q16	770	686	687	686
q17	3787	3753	3877	3753
q18	8642	8715	8713	8713
q19	1695	1695	1656	1656
q20	2356	2163	2110	2110
q21	8501	8438	8473	8438
q22	1045	961	941	941
Total cold run time: 84741 ms
Total hot run time: 79798 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202310 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5328f20f213d289aa6fb1f400b08a44047e8612b, data reload: false

query1	921	390	385	385
query2	6540	2209	2245	2209
query3	6919	206	199	199
query4	21281	17910	17873	17873
query5	19728	6535	6529	6529
query6	279	210	233	210
query7	4165	302	305	302
query8	255	261	250	250
query9	3169	2708	2640	2640
query10	427	281	290	281
query11	11331	10633	10720	10633
query12	114	77	72	72
query13	5577	647	636	636
query14	17428	13260	13553	13260
query15	378	240	235	235
query16	6426	268	258	258
query17	1714	1458	862	862
query18	2322	409	395	395
query19	209	140	153	140
query20	82	76	72	72
query21	185	94	90	90
query22	5230	5118	5124	5118
query23	32461	31785	31979	31785
query24	6994	6501	6506	6501
query25	510	424	416	416
query26	536	163	154	154
query27	1851	291	292	291
query28	6073	2233	2183	2183
query29	3027	2840	2763	2763
query30	245	160	159	159
query31	899	727	723	723
query32	65	59	58	58
query33	400	251	242	242
query34	848	468	477	468
query35	1126	926	895	895
query36	1651	1812	1570	1570
query37	88	64	62	62
query38	3049	2958	2900	2900
query39	1390	1340	1329	1329
query40	199	98	101	98
query41	35	38	34	34
query42	90	96	78	78
query43	642	656	650	650
query44	1162	723	723	723
query45	254	237	238	237
query46	1252	998	975	975
query47	1894	1729	1673	1673
query48	967	693	652	652
query49	621	368	361	361
query50	857	575	604	575
query51	4808	4690	4735	4690
query52	84	79	80	79
query53	435	326	327	326
query54	2642	2480	2452	2452
query55	81	83	85	83
query56	216	206	211	206
query57	1158	1110	1183	1110
query58	216	208	191	191
query59	3360	3255	3295	3255
query60	210	204	192	192
query61	83	87	82	82
query62	789	450	467	450
query63	476	331	332	331
query64	2458	1477	1427	1427
query65	3584	3527	3564	3527
query66	772	374	371	371
query67	15841	15300	15688	15300
query68	9403	628	673	628
query69	561	337	344	337
query70	1956	1605	2163	1605
query71	425	307	304	304
query72	6510	3397	3472	3397
query73	738	333	320	320
query74	6303	5817	5829	5817
query75	5363	3741	3702	3702
query76	5966	1138	1187	1138
query77	1015	247	240	240
query78	12876	11836	11913	11836
query79	7282	651	641	641
query80	1007	395	400	395
query81	483	228	230	228
query82	1578	102	96	96
query83	169	133	130	130
query84	258	69	69	69
query85	853	288	286	286
query86	389	409	363	363
query87	3198	3063	3010	3010
query88	4693	2358	2343	2343
query89	369	267	286	267
query90	2044	211	217	211
query91	151	118	118	118
query92	56	51	49	49
query93	5412	583	556	556
query94	649	203	210	203
query95	1128	1065	1064	1064
query96	661	343	335	335
query97	6528	6325	6383	6325
query98	194	179	161	161
query99	2974	860	1052	860
Total cold run time: 312271 ms
Total hot run time: 202310 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.71 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5328f20f213d289aa6fb1f400b08a44047e8612b, data reload: false

query1	0.02	0.03	0.02
query2	0.05	0.02	0.02
query3	0.25	0.04	0.05
query4	1.83	0.07	0.06
query5	0.53	0.52	0.52
query6	1.31	0.62	0.62
query7	0.02	0.02	0.02
query8	0.03	0.02	0.02
query9	0.53	0.47	0.48
query10	0.52	0.53	0.54
query11	0.12	0.08	0.08
query12	0.11	0.09	0.09
query13	0.62	0.61	0.62
query14	0.77	0.80	0.80
query15	0.77	0.76	0.77
query16	0.35	0.37	0.37
query17	1.01	1.04	1.01
query18	0.23	0.24	0.26
query19	1.90	1.84	1.78
query20	0.01	0.01	0.01
query21	15.51	0.56	0.56
query22	2.13	2.32	2.14
query23	17.38	0.92	1.02
query24	7.87	1.00	0.50
query25	0.41	0.12	0.06
query26	0.76	0.13	0.16
query27	0.04	0.04	0.03
query28	6.46	0.73	0.70
query29	12.70	2.29	2.28
query30	0.59	0.53	0.52
query31	2.82	0.39	0.38
query32	3.41	0.49	0.49
query33	3.06	3.05	3.10
query34	15.32	4.83	4.80
query35	4.87	4.87	4.82
query36	1.03	1.01	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.15	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 105.71 s
Total hot run time: 30.71 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5328f20f213d289aa6fb1f400b08a44047e8612b with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.1 seconds inserted 10000000 Rows, about 497K ops/s

…1 instance (apache#31342)

This PR apache#25952 introduce a opt that if a scan node has limit and predicates, use only 1 instance to save cup and memory.
But this is wrong because we can not guarantee that the predicates can truly help to prune the data.
So I modify the logic to remove this opt.
Now, only scan node with limit and NO predicate can reduce to only 1 instance.
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49722 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a65ad40c4e61b46434d936b4d8d018fc00f73c35, data reload: false

------ Round 1 ----------------------------------
q1	17885	4381	4336	4336
q2	2034	155	138	138
q3	10278	1896	1947	1896
q4	10356	1213	1304	1213
q5	8856	3937	3991	3937
q6	230	121	125	121
q7	2009	1621	1601	1601
q8	9363	2692	2709	2692
q9	10856	10441	10536	10441
q10	8617	3498	3531	3498
q11	438	244	249	244
q12	466	296	304	296
q13	18366	3930	4014	3930
q14	356	326	325	325
q15	527	463	459	459
q16	692	589	581	581
q17	1120	950	972	950
q18	7190	6774	6906	6774
q19	1681	1555	1519	1519
q20	510	301	300	300
q21	4505	4071	4073	4071
q22	498	400	402	400
Total cold run time: 116833 ms
Total hot run time: 49722 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4324	4316	4323	4316
q2	320	220	225	220
q3	4165	4137	4114	4114
q4	2750	2732	2744	2732
q5	7239	7175	7179	7175
q6	234	121	119	119
q7	3185	2836	2816	2816
q8	4308	4493	4420	4420
q9	16986	17012	16883	16883
q10	4204	4237	4271	4237
q11	755	682	658	658
q12	1002	860	860	860
q13	4457	3729	3741	3729
q14	439	407	414	407
q15	498	448	449	448
q16	753	701	700	700
q17	3777	3864	3781	3781
q18	8827	8782	8706	8706
q19	1696	1689	1677	1677
q20	2356	2169	2150	2150
q21	8428	8386	8426	8386
q22	1022	967	979	967
Total cold run time: 81725 ms
Total hot run time: 79501 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202539 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a65ad40c4e61b46434d936b4d8d018fc00f73c35, data reload: false

query1	917	385	386	385
query2	6544	2203	2211	2203
query3	6925	205	206	205
query4	21423	17937	17945	17937
query5	19716	6465	6486	6465
query6	290	210	227	210
query7	4156	289	297	289
query8	259	273	244	244
query9	3130	2697	2623	2623
query10	424	301	278	278
query11	11233	10674	10669	10669
query12	113	76	66	66
query13	5582	634	627	627
query14	17966	13842	13581	13581
query15	356	237	238	237
query16	6448	257	262	257
query17	1731	1441	860	860
query18	2330	397	406	397
query19	205	145	149	145
query20	71	76	75	75
query21	188	94	97	94
query22	5149	4982	5049	4982
query23	32512	31901	31813	31813
query24	7003	6577	6472	6472
query25	520	440	419	419
query26	528	162	153	153
query27	1910	291	286	286
query28	6097	2235	2201	2201
query29	2872	2754	2648	2648
query30	247	158	167	158
query31	924	723	707	707
query32	63	57	58	57
query33	390	253	252	252
query34	833	469	487	469
query35	1100	913	914	913
query36	1691	1474	1369	1369
query37	92	59	58	58
query38	3075	2932	2962	2932
query39	1371	1324	1325	1324
query40	198	86	93	86
query41	37	33	31	31
query42	86	77	91	77
query43	619	581	571	571
query44	1119	704	710	704
query45	238	232	226	226
query46	1228	958	989	958
query47	1953	1668	1704	1668
query48	978	677	667	667
query49	610	357	372	357
query50	862	636	614	614
query51	4848	4686	4676	4676
query52	89	89	77	77
query53	437	318	319	318
query54	2667	2448	2493	2448
query55	92	74	74	74
query56	218	204	203	203
query57	1214	1194	1079	1079
query58	212	207	179	179
query59	3474	3145	3247	3145
query60	197	185	187	185
query61	84	88	80	80
query62	849	452	480	452
query63	465	337	338	337
query64	2356	1511	1295	1295
query65	3622	3569	3567	3567
query66	774	364	360	360
query67	16480	16115	18295	16115
query68	6360	633	671	633
query69	572	320	344	320
query70	1979	1837	1710	1710
query71	390	309	294	294
query72	6484	3415	3409	3409
query73	740	339	331	331
query74	6570	6243	5942	5942
query75	4635	3747	3703	3703
query76	4299	1145	1272	1145
query77	548	246	244	244
query78	12642	11561	11667	11561
query79	7404	626	628	626
query80	2494	387	392	387
query81	526	236	226	226
query82	1533	95	97	95
query83	184	136	129	129
query84	258	71	68	68
query85	1410	279	278	278
query86	405	366	367	366
query87	3202	2997	3004	2997
query88	5162	2338	2345	2338
query89	345	291	283	283
query90	1719	216	210	210
query91	157	115	111	111
query92	56	54	51	51
query93	3182	604	543	543
query94	885	198	202	198
query95	1129	1069	1079	1069
query96	640	322	320	320
query97	6458	6309	6429	6309
query98	200	181	165	165
query99	2984	940	869	869
Total cold run time: 307942 ms
Total hot run time: 202539 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a65ad40c4e61b46434d936b4d8d018fc00f73c35, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.04	0.05
query4	1.83	0.07	0.06
query5	0.52	0.52	0.52
query6	1.24	0.62	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.51	0.49	0.48
query10	0.54	0.53	0.53
query11	0.12	0.08	0.08
query12	0.12	0.09	0.10
query13	0.62	0.61	0.62
query14	0.79	0.79	0.78
query15	0.80	0.77	0.76
query16	0.37	0.37	0.37
query17	1.02	1.00	1.00
query18	0.21	0.26	0.25
query19	1.91	1.83	1.81
query20	0.01	0.01	0.01
query21	15.47	0.56	0.56
query22	1.83	2.38	1.35
query23	17.23	1.03	0.80
query24	5.07	1.45	1.31
query25	0.39	0.11	0.04
query26	0.66	0.13	0.13
query27	0.05	0.04	0.03
query28	7.60	0.73	0.70
query29	12.62	2.27	2.31
query30	0.57	0.49	0.53
query31	2.81	0.40	0.37
query32	3.42	0.50	0.50
query33	3.09	3.05	3.07
query34	15.26	4.77	4.75
query35	4.85	4.84	4.82
query36	1.06	1.02	1.02
query37	0.05	0.04	0.04
query38	0.04	0.02	0.03
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 103.33 s
Total hot run time: 30.5 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a65ad40c4e61b46434d936b4d8d018fc00f73c35 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.6 seconds inserted 10000000 Rows, about 485K ops/s

@morningman morningman merged commit 666a7d0 into apache:branch-2.0 Mar 1, 2024
24 of 26 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…1 instance (apache#31342) (apache#31606)

This PR apache#25952 introduce a opt that if a scan node has limit and predicates, use only 1 instance to save cup and memory.
But this is wrong because we can not guarantee that the predicates can truly help to prune the data.
So I modify the logic to remove this opt.
Now, only scan node with limit and NO predicate can reduce to only 1 instance.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/planner Issues or PRs related to the query planner
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants