Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](memory) Fix refresh cgroup meminfo #41123

Merged
merged 1 commit into from
Sep 24, 2024
Merged

[fix](memory) Fix refresh cgroup meminfo #41123

merged 1 commit into from
Sep 24, 2024

Conversation

xinyiZzz
Copy link
Contributor

cgroup memory usage should be refreshed frequently.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

run buildall

wm1581066
wm1581066 suggested changes Sep 23, 2024
View reviewed changes
Copy link
Collaborator

@wm1581066 wm1581066 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any test cases ?

@xinyiZzz
Copy link
Contributor Author

Any test cases ?

@wm1581066 cgroup_util_test.cpp in BE UT will test the correctness of cgroup memory refresh.

but the bug fixed by this PR is refresh interval is too long, which makes it difficult to add tests.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.28% (9608/25775)
Line Coverage: 28.69% (79507/277172)
Region Coverage: 28.13% (41128/146181)
Branch Coverage: 24.78% (20965/84618)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d0d4cb2c3ae634c897abcd904dd568e7ae668647_d0d4cb2c3ae634c897abcd904dd568e7ae668647/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41632 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d0d4cb2c3ae634c897abcd904dd568e7ae668647, data reload: false

------ Round 1 ----------------------------------
q1	17567	7385	7371	7371
q2	2058	173	171	171
q3	10580	1075	1180	1075
q4	10568	771	735	735
q5	7738	3073	3088	3073
q6	237	149	146	146
q7	1004	621	606	606
q8	9442	2029	2040	2029
q9	6859	6418	6434	6418
q10	7022	2304	2309	2304
q11	429	250	255	250
q12	409	215	214	214
q13	17774	2965	3003	2965
q14	244	219	216	216
q15	589	524	517	517
q16	679	618	612	612
q17	972	811	774	774
q18	7239	6647	6813	6647
q19	1387	1110	1073	1073
q20	583	294	276	276
q21	4023	3233	3177	3177
q22	1089	1016	983	983
Total cold run time: 108492 ms
Total hot run time: 41632 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7268	7198	7222	7198
q2	327	242	233	233
q3	3032	2935	3019	2935
q4	2038	1857	1871	1857
q5	5584	5606	5634	5606
q6	246	146	144	144
q7	2240	1797	1796	1796
q8	3314	3431	3444	3431
q9	8795	9007	8786	8786
q10	3488	3486	3461	3461
q11	576	501	505	501
q12	829	595	613	595
q13	10139	3144	3124	3124
q14	308	295	289	289
q15	561	536	524	524
q16	733	667	671	667
q17	1808	1606	1603	1603
q18	8240	7727	7711	7711
q19	1719	1569	1553	1553
q20	2139	1893	1894	1893
q21	5616	5281	5434	5281
q22	1179	1067	1013	1013
Total cold run time: 70179 ms
Total hot run time: 60201 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194825 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d0d4cb2c3ae634c897abcd904dd568e7ae668647, data reload: false

query1	1273	869	904	869
query2	6248	2014	2041	2014
query3	10792	3909	3820	3820
query4	62792	29047	23403	23403
query5	5117	489	475	475
query6	403	160	159	159
query7	5442	318	305	305
query8	313	228	228	228
query9	8339	2663	2635	2635
query10	458	277	289	277
query11	17610	15186	15909	15186
query12	158	102	102	102
query13	1518	428	427	427
query14	10828	7031	7121	7031
query15	208	179	187	179
query16	6852	470	537	470
query17	1164	633	586	586
query18	1514	311	313	311
query19	222	158	160	158
query20	141	115	119	115
query21	211	103	108	103
query22	4884	4529	4828	4529
query23	34622	33900	33889	33889
query24	6084	2959	2874	2874
query25	511	409	403	403
query26	614	161	155	155
query27	1579	285	286	285
query28	4285	2466	2419	2419
query29	674	431	440	431
query30	236	157	159	157
query31	988	804	810	804
query32	80	56	58	56
query33	419	296	295	295
query34	899	493	490	490
query35	849	723	743	723
query36	1087	948	973	948
query37	147	87	88	87
query38	4084	3967	3986	3967
query39	1474	1429	1426	1426
query40	216	95	99	95
query41	50	48	49	48
query42	120	96	93	93
query43	517	475	501	475
query44	1175	821	806	806
query45	201	164	167	164
query46	1141	774	751	751
query47	1920	1811	1845	1811
query48	481	359	377	359
query49	703	422	410	410
query50	856	433	402	402
query51	7030	6835	6959	6835
query52	94	90	81	81
query53	247	182	181	181
query54	574	451	461	451
query55	79	76	79	76
query56	292	264	241	241
query57	1233	1082	1119	1082
query58	222	263	231	231
query59	3038	2895	2816	2816
query60	293	269	274	269
query61	109	106	106	106
query62	769	659	650	650
query63	207	182	181	181
query64	1275	644	671	644
query65	3247	3155	3161	3155
query66	693	290	303	290
query67	15945	15475	15552	15475
query68	4039	593	561	561
query69	553	300	296	296
query70	1202	1145	1117	1117
query71	394	272	270	270
query72	6982	4057	3944	3944
query73	764	333	327	327
query74	10468	8940	9054	8940
query75	3427	2680	2667	2667
query76	2722	973	933	933
query77	714	302	301	301
query78	10039	9195	9074	9074
query79	2335	547	551	547
query80	1095	454	439	439
query81	553	238	245	238
query82	572	145	141	141
query83	343	137	133	133
query84	297	79	78	78
query85	1016	292	281	281
query86	396	301	300	300
query87	4513	4380	4338	4338
query88	3606	2340	2359	2340
query89	396	290	282	282
query90	2068	196	186	186
query91	175	141	143	141
query92	63	48	50	48
query93	2604	540	542	540
query94	1009	292	301	292
query95	352	252	254	252
query96	648	281	286	281
query97	3286	3082	3097	3082
query98	212	203	200	200
query99	1525	1316	1287	1287
Total cold run time: 316214 ms
Total hot run time: 194825 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d0d4cb2c3ae634c897abcd904dd568e7ae668647, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.53	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.03	0.03	0.03
query9	0.57	0.51	0.52
query10	0.55	0.58	0.56
query11	0.14	0.11	0.11
query12	0.14	0.10	0.10
query13	0.60	0.60	0.60
query14	3.04	2.95	2.94
query15	0.90	0.82	0.82
query16	0.38	0.38	0.38
query17	1.00	1.02	0.97
query18	0.20	0.19	0.20
query19	1.84	1.86	2.02
query20	0.02	0.01	0.01
query21	15.37	0.60	0.57
query22	2.32	2.75	1.78
query23	17.52	0.92	0.81
query24	2.72	1.03	0.67
query25	0.27	0.15	0.05
query26	0.41	0.14	0.13
query27	0.03	0.04	0.04
query28	11.11	1.11	1.07
query29	12.56	3.22	3.21
query30	0.24	0.06	0.05
query31	2.90	0.37	0.38
query32	3.27	0.47	0.46
query33	2.96	2.99	3.04
query34	16.82	4.39	4.46
query35	4.48	4.46	4.37
query36	0.65	0.48	0.48
query37	0.08	0.06	0.05
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.13 s
Total hot run time: 32.27 s

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 24, 2024
wangbo
wangbo approved these changes Sep 24, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit 601cc58 into apache:master Sep 24, 2024
25 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@xinyiZzz @dataroaring
[fix](memory) Fix refresh cgroup meminfo (#41123)
2749a8e 
cgroup memory usage should be refreshed frequently.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Oct 28, 2024
@xinyiZzz
[fix](memory) Fix refresh cgroup meminfo (apache#41123)
c0bf905 
cgroup memory usage should be refreshed frequently.
yiguolei pushed a commit that referenced this pull request Oct 28, 2024
@xinyiZzz
[pick](branch-2.1) pick #41123 (#42541)
d319daf 
cgroup memory usage should be refreshed frequently.
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wm1581066 wm1581066 wm1581066 requested changes

@wangbo wangbo wangbo approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@xinyiZzz @doris-robot @yiguolei @wangbo @wm1581066 @dataroaring