Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](pipelinex) fix fragment instance progress reports (part 2) #40694

Merged
merged 4 commits into from
Sep 25, 2024
Merged

[fix](pipelinex) fix fragment instance progress reports (part 2) #40694

merged 4 commits into from
Sep 25, 2024

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented Sep 11, 2024 •
edited
Loading

Proposed changes

This PR is a followup of #40325. Because PipelineX has deprecated the old report.
This PR fixed the ScannedRows and LoadBytes in the progress of the SHOW LOAD result.

Currently the progress will only be updated when a fragment instance is done.
Timely progress updates will be supported in a later PR.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen
Copy link
Contributor Author

run buildall

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.90% (9455/25626)
Line Coverage: 28.24% (77753/275295)
Region Coverage: 27.64% (40132/145182)
Branch Coverage: 24.26% (20393/84076)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e4fe7a34224262315c5e43111c3e79ef7a50734_0e4fe7a34224262315c5e43111c3e79ef7a50734/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38287 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e4fe7a34224262315c5e43111c3e79ef7a50734, data reload: false

------ Round 1 ----------------------------------
q1	18130	4819	4404	4404
q2	2029	185	190	185
q3	11570	950	1103	950
q4	10178	731	730	730
q5	7750	2895	2879	2879
q6	234	141	138	138
q7	975	621	607	607
q8	9334	2132	2139	2132
q9	7182	6580	6584	6580
q10	7004	2219	2237	2219
q11	452	242	240	240
q12	396	230	228	228
q13	17769	3110	3112	3110
q14	272	230	236	230
q15	542	479	483	479
q16	543	459	440	440
q17	1000	706	686	686
q18	7530	6842	6971	6842
q19	1398	1075	1039	1039
q20	672	336	356	336
q21	4228	3262	2813	2813
q22	1114	1053	1020	1020
Total cold run time: 110302 ms
Total hot run time: 38287 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4388	4318	4311	4311
q2	389	273	272	272
q3	2911	2646	2711	2646
q4	1928	1618	1616	1616
q5	5712	5701	5819	5701
q6	235	140	136	136
q7	2258	1826	1800	1800
q8	3345	3429	3511	3429
q9	8880	8837	8824	8824
q10	3591	3417	3382	3382
q11	619	513	522	513
q12	827	706	666	666
q13	15138	3231	3300	3231
q14	317	300	287	287
q15	550	495	489	489
q16	545	506	501	501
q17	1879	1570	1511	1511
q18	8167	7929	7955	7929
q19	1760	1564	1616	1564
q20	2209	1914	1899	1899
q21	5728	5534	5473	5473
q22	1109	1004	1041	1004
Total cold run time: 72485 ms
Total hot run time: 57184 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197883 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e4fe7a34224262315c5e43111c3e79ef7a50734, data reload: false

query1	1255	880	898	880
query2	6369	1884	1868	1868
query3	10599	4148	3890	3890
query4	60004	28915	23141	23141
query5	4987	510	496	496
query6	393	181	181	181
query7	5608	307	293	293
query8	311	213	224	213
query9	7814	2510	2491	2491
query10	419	287	279	279
query11	17095	15136	15354	15136
query12	146	111	98	98
query13	1432	405	391	391
query14	10438	7487	7346	7346
query15	210	182	169	169
query16	6759	448	490	448
query17	1076	588	561	561
query18	1539	300	299	299
query19	196	152	158	152
query20	121	113	111	111
query21	216	109	106	106
query22	4705	4411	4505	4411
query23	34446	33833	33667	33667
query24	5991	2871	2878	2871
query25	516	406	406	406
query26	594	155	152	152
query27	1589	270	284	270
query28	3926	2057	2045	2045
query29	637	407	411	407
query30	234	154	154	154
query31	954	752	774	752
query32	70	54	56	54
query33	404	279	281	279
query34	880	487	460	460
query35	845	742	736	736
query36	1050	941	944	941
query37	145	87	82	82
query38	4054	3874	3983	3874
query39	1438	1391	1400	1391
query40	200	114	111	111
query41	45	45	44	44
query42	117	98	95	95
query43	505	470	471	470
query44	1076	754	758	754
query45	198	171	176	171
query46	1085	743	730	730
query47	1923	1783	1831	1783
query48	374	303	300	300
query49	786	467	449	449
query50	839	411	422	411
query51	7004	6787	6986	6787
query52	107	90	89	89
query53	258	183	185	183
query54	583	456	460	456
query55	75	75	76	75
query56	294	272	271	271
query57	1188	1074	1083	1074
query58	255	250	246	246
query59	2913	2830	2781	2781
query60	310	283	279	279
query61	121	122	124	122
query62	773	630	639	630
query63	214	194	189	189
query64	1450	746	751	746
query65	3218	3174	3159	3159
query66	678	349	341	341
query67	15704	15414	15405	15405
query68	1973	848	854	848
query69	435	332	325	325
query70	1186	1167	1173	1167
query71	358	339	345	339
query72	4779	3893	3667	3667
query73	581	571	573	571
query74	9023	8908	8931	8908
query75	2994	2959	2957	2957
query76	930	837	842	837
query77	544	406	392	392
query78	9315	9427	9198	9198
query79	872	881	845	845
query80	802	819	791	791
query81	446	259	259	259
query82	264	266	261	261
query83	189	190	189	189
query84	194	106	105	105
query85	652	413	383	383
query86	321	324	309	309
query87	4448	4317	4337	4317
query88	4276	4023	3999	3999
query89	369	363	356	356
query90	1480	308	308	308
query91	124	122	126	122
query92	77	77	73	73
query93	901	893	889	889
query94	542	387	352	352
query95	415	407	411	407
query96	464	464	467	464
query97	3108	3088	3084	3084
query98	226	222	228	222
query99	1312	1287	1297	1287
Total cold run time: 295358 ms
Total hot run time: 197883 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e4fe7a34224262315c5e43111c3e79ef7a50734, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.52	0.51	0.50
query6	1.14	0.74	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.55	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.11	0.12
query13	0.62	0.59	0.59
query14	1.38	1.43	1.45
query15	0.85	0.81	0.80
query16	0.38	0.38	0.39
query17	1.06	1.04	1.05
query18	0.18	0.18	0.17
query19	1.95	1.73	1.88
query20	0.01	0.01	0.01
query21	15.39	0.67	0.67
query22	4.43	5.77	2.70
query23	18.30	1.34	1.24
query24	2.10	0.23	0.22
query25	0.15	0.10	0.08
query26	0.27	0.18	0.19
query27	0.08	0.07	0.08
query28	13.21	1.03	1.00
query29	12.59	3.27	3.29
query30	0.24	0.05	0.06
query31	2.85	0.40	0.40
query32	3.26	0.48	0.50
query33	3.00	3.04	3.08
query34	17.04	4.41	4.40
query35	4.46	4.47	4.46
query36	0.66	0.47	0.48
query37	0.20	0.15	0.16
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.15	0.13	0.13
query41	0.10	0.05	0.06
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.37 s
Total hot run time: 32.2 s

@liaoxin01
Copy link
Contributor

Please add a regression test.

liaoxin01
liaoxin01 reviewed Sep 13, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java Outdated
if (!params.done) {
return false;
return true;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may bring some problems here, refer to the comments above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverted

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42903 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit af86c63d56915c2a013230b95c7d4170d49338f4, data reload: false

------ Round 1 ----------------------------------
q1	17586	7890	7211	7211
q2	2037	193	187	187
q3	10781	1232	1383	1232
q4	10233	1044	1019	1019
q5	7724	3195	3164	3164
q6	243	158	153	153
q7	1049	635	625	625
q8	9479	2083	2045	2045
q9	6781	6301	6320	6301
q10	7032	2517	2549	2517
q11	431	257	262	257
q12	414	232	229	229
q13	17764	3012	3041	3012
q14	284	251	254	251
q15	582	544	539	539
q16	524	425	428	425
q17	1023	948	969	948
q18	7446	6756	6707	6707
q19	1375	1247	1236	1236
q20	603	342	330	330
q21	3914	3556	3593	3556
q22	1062	1001	959	959
Total cold run time: 108367 ms
Total hot run time: 42903 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7181	7161	7239	7161
q2	356	242	236	236
q3	3088	3119	3051	3051
q4	2101	2158	2066	2066
q5	5710	5619	5709	5619
q6	243	150	152	150
q7	2172	1787	1763	1763
q8	3375	3462	3700	3462
q9	8852	8852	8822	8822
q10	3438	3595	3570	3570
q11	578	467	489	467
q12	849	606	632	606
q13	11343	3173	3198	3173
q14	310	291	276	276
q15	595	535	541	535
q16	495	487	466	466
q17	1798	1742	1716	1716
q18	8532	7983	8018	7983
q19	1787	1761	1742	1742
q20	2144	1890	1866	1866
q21	5905	5500	5681	5500
q22	1150	1036	1023	1023
Total cold run time: 72002 ms
Total hot run time: 61253 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.32% (9577/25663)
Line Coverage: 28.72% (79128/275515)
Region Coverage: 28.19% (40970/145335)
Branch Coverage: 24.82% (20884/84158)
Coverage Report: http://coverage.selectdb-in.cc/coverage/af86c63d56915c2a013230b95c7d4170d49338f4_af86c63d56915c2a013230b95c7d4170d49338f4/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 199620 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit af86c63d56915c2a013230b95c7d4170d49338f4, data reload: false

query1	1282	894	883	883
query2	6341	1788	1779	1779
query3	10679	3821	3884	3821
query4	56267	26846	23862	23862
query5	4993	520	547	520
query6	315	166	156	156
query7	5594	312	304	304
query8	284	220	221	220
query9	6071	2602	2590	2590
query10	422	278	266	266
query11	15956	15324	15511	15324
query12	150	104	99	99
query13	1452	396	388	388
query14	10188	6955	7416	6955
query15	207	181	184	181
query16	6989	481	500	481
query17	1159	593	600	593
query18	1897	306	308	306
query19	211	163	153	153
query20	130	115	114	114
query21	216	106	104	104
query22	4616	4381	4384	4381
query23	34609	34041	33829	33829
query24	6168	3135	3127	3127
query25	518	412	417	412
query26	610	154	158	154
query27	1571	281	287	281
query28	2845	2148	2114	2114
query29	692	438	440	438
query30	220	155	158	155
query31	975	786	772	772
query32	76	54	56	54
query33	457	312	311	311
query34	894	485	488	485
query35	862	724	744	724
query36	1037	903	916	903
query37	144	85	88	85
query38	4006	3939	4041	3939
query39	1449	1458	1384	1384
query40	209	117	117	117
query41	51	46	47	46
query42	122	96	103	96
query43	489	447	465	447
query44	1195	794	776	776
query45	201	174	174	174
query46	1103	801	786	786
query47	1915	1765	1783	1765
query48	381	298	309	298
query49	742	452	450	450
query50	917	448	439	439
query51	7007	6937	6973	6937
query52	98	89	85	85
query53	252	182	182	182
query54	555	465	459	459
query55	75	75	75	75
query56	275	262	272	262
query57	1213	1076	1082	1076
query58	246	226	239	226
query59	2918	2636	2625	2625
query60	284	275	276	275
query61	131	99	100	99
query62	783	694	684	684
query63	224	189	187	187
query64	1589	715	666	666
query65	3220	3180	3184	3180
query66	675	291	290	290
query67	16191	15778	15392	15392
query68	2071	581	579	579
query69	408	278	283	278
query70	1187	1127	1089	1089
query71	357	288	279	279
query72	5934	3988	4111	3988
query73	766	327	328	327
query74	9396	9219	9140	9140
query75	3354	2743	2688	2688
query76	1367	1300	1288	1288
query77	566	320	346	320
query78	9986	9231	9621	9231
query79	1387	881	874	874
query80	1093	848	824	824
query81	499	268	266	266
query82	1047	270	271	270
query83	196	187	188	187
query84	263	107	107	107
query85	735	407	404	404
query86	322	311	307	307
query87	4355	4516	4464	4464
query88	4234	4148	4150	4148
query89	398	369	365	365
query90	1924	316	312	312
query91	126	123	127	123
query92	78	73	74	73
query93	1269	1042	1030	1030
query94	797	371	370	370
query95	497	435	426	426
query96	492	479	482	479
query97	3148	3120	3133	3120
query98	229	239	232	232
query99	1722	1310	1291	1291
Total cold run time: 295816 ms
Total hot run time: 199620 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit af86c63d56915c2a013230b95c7d4170d49338f4, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.50	0.51
query6	1.13	0.72	0.72
query7	0.01	0.01	0.01
query8	0.06	0.05	0.04
query9	0.57	0.52	0.51
query10	0.56	0.58	0.57
query11	0.16	0.13	0.12
query12	0.15	0.12	0.12
query13	0.62	0.61	0.60
query14	1.47	1.45	1.50
query15	0.91	0.88	0.88
query16	0.36	0.36	0.36
query17	1.02	1.05	1.01
query18	0.16	0.16	0.16
query19	1.95	1.81	1.79
query20	0.02	0.01	0.01
query21	15.42	0.66	0.66
query22	3.92	8.91	1.22
query23	17.93	1.27	1.32
query24	2.27	0.22	0.21
query25	0.19	0.08	0.08
query26	0.29	0.18	0.18
query27	0.08	0.08	0.08
query28	13.19	1.12	1.10
query29	12.51	3.34	3.32
query30	0.25	0.06	0.05
query31	2.87	0.43	0.41
query32	3.22	0.50	0.50
query33	3.04	2.98	3.07
query34	15.45	4.32	4.33
query35	4.31	4.31	4.34
query36	0.69	0.47	0.50
query37	0.18	0.16	0.16
query38	0.17	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.14	0.14
query41	0.10	0.05	0.05
query42	0.07	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.09 s
Total hot run time: 30.83 s

@kaijchen
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.27% (9605/25774)
Line Coverage: 28.69% (79516/277192)
Region Coverage: 28.12% (41113/146190)
Branch Coverage: 24.77% (20958/84626)
Coverage Report: http://coverage.selectdb-in.cc/coverage/01e40feb126fe6aa806d45be6bfc3fb800e9b90f_01e40feb126fe6aa806d45be6bfc3fb800e9b90f/report/index.html

@gavinchou gavinchou added the cir label Sep 25, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 approved these changes Sep 25, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gavinchou gavinchou merged commit 3d4c22d into apache:master Sep 25, 2024
23 of 28 checks passed
gavinchou pushed a commit that referenced this pull request Sep 25, 2024
@kaijchen @gavinchou
[fix](pipelinex) fix fragment instance progress reports (part 2) (#40694
63cdc31 
)

## Proposed changes

This PR is a followup of #40325. Because PipelineX has deprecated the
old report.
This PR fixed the `ScannedRows` and `LoadBytes` in the progress of the
`SHOW LOAD` result.

Currently the progress will only be updated when a fragment instance is
done.
Timely progress updates will be supported in a later PR.
kaijchen added a commit to kaijchen/doris that referenced this pull request Oct 10, 2024
@kaijchen
[fix](pipelinex) fix fragment instance progress reports (part 2) (apa…
95e9531 
…che#40694)

This PR is a followup of apache#40325. Because PipelineX has deprecated the
old report.
This PR fixed the `ScannedRows` and `LoadBytes` in the progress of the
`SHOW LOAD` result.

Currently the progress will only be updated when a fragment instance is
done.
Timely progress updates will be supported in a later PR.
@kaijchen kaijchen mentioned this pull request Oct 10, 2024
Merged
yiguolei pushed a commit that referenced this pull request Oct 10, 2024
@kaijchen
[fix](pipelinex) fix fragment instance progress reports (part 2) (#40694
3120bfb 
) (#41641)

backport #40694
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
kaijchen added a commit to kaijchen/doris that referenced this pull request Oct 16, 2024
@kaijchen
[fix](pipelinex) fix fragment instance progress reports (part 2) (apa…
ae32148 
…che#40694)

This PR is a followup of apache#40325. Because PipelineX has deprecated the
old report.
This PR fixed the `ScannedRows` and `LoadBytes` in the progress of the
`SHOW LOAD` result.

Currently the progress will only be updated when a fragment instance is
done.
Timely progress updates will be supported in a later PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. cir dev/2.1.7-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@kaijchen @doris-robot @liaoxin01 @gavinchou @yiguolei