Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](ForkJoinPool) we should not new a thread pool every call #44891

Merged
merged 1 commit into from
Dec 3, 2024

Conversation

cambyzju
Copy link
Contributor

@cambyzju cambyzju commented Dec 3, 2024

What problem does this PR solve?

come from: #34988

Problem:
Use JDK11 and call show proc "/cluster_health/tablet_health" frequently, make large number of ForkJoinPool thread leak

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@cambyzju
Copy link
Contributor Author

cambyzju commented Dec 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39950 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a17f2833d69caaed5b8a48e6a28857e10beeaca, data reload: false

------ Round 1 ----------------------------------
q1	17591	8070	7261	7261
q2	2048	176	182	176
q3	10612	1072	1166	1072
q4	10565	730	723	723
q5	7605	2727	2744	2727
q6	240	144	145	144
q7	1011	635	603	603
q8	9272	1874	1943	1874
q9	6729	6500	6435	6435
q10	7022	2270	2303	2270
q11	470	259	254	254
q12	422	218	226	218
q13	17801	3015	3001	3001
q14	242	224	219	219
q15	563	523	526	523
q16	651	586	588	586
q17	983	534	570	534
q18	7212	6790	6727	6727
q19	1334	1102	1011	1011
q20	459	178	179	178
q21	4009	3295	3103	3103
q22	384	320	311	311
Total cold run time: 107225 ms
Total hot run time: 39950 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7200	7216	7220	7216
q2	328	231	234	231
q3	2878	2853	2943	2853
q4	2040	1844	1836	1836
q5	5632	5667	5637	5637
q6	223	144	141	141
q7	2271	1804	1799	1799
q8	3413	3511	3515	3511
q9	9048	9079	9014	9014
q10	3618	3571	3571	3571
q11	604	517	517	517
q12	814	625	627	625
q13	11407	3366	3231	3231
q14	320	277	288	277
q15	575	533	537	533
q16	696	652	674	652
q17	1887	1622	1609	1609
q18	8496	7813	7516	7516
q19	1785	1648	1567	1567
q20	2101	1889	1878	1878
q21	5665	5528	5408	5408
q22	651	561	547	547
Total cold run time: 71652 ms
Total hot run time: 60169 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198262 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a17f2833d69caaed5b8a48e6a28857e10beeaca, data reload: false

query1	1252	954	917	917
query2	6228	2129	2091	2091
query3	10946	4429	4260	4260
query4	67482	28268	23547	23547
query5	4926	473	455	455
query6	430	188	175	175
query7	5494	304	297	297
query8	327	240	239	239
query9	8422	2698	2690	2690
query10	449	252	240	240
query11	17256	15359	15957	15359
query12	150	104	108	104
query13	1465	402	412	402
query14	9838	7771	8015	7771
query15	217	187	199	187
query16	7031	498	493	493
query17	1004	570	553	553
query18	1577	319	303	303
query19	203	153	158	153
query20	129	116	108	108
query21	205	100	106	100
query22	4744	4740	4552	4552
query23	34823	34390	34238	34238
query24	5446	2513	2542	2513
query25	485	384	384	384
query26	636	155	151	151
query27	1778	282	290	282
query28	4613	2508	2497	2497
query29	669	451	453	451
query30	220	161	158	158
query31	1015	834	819	819
query32	71	56	64	56
query33	443	302	297	297
query34	945	537	569	537
query35	901	800	746	746
query36	1072	976	980	976
query37	117	79	69	69
query38	4450	4391	4435	4391
query39	1507	1498	1490	1490
query40	218	100	100	100
query41	45	43	46	43
query42	112	104	102	102
query43	559	515	510	510
query44	1190	815	817	815
query45	184	178	166	166
query46	1204	736	727	727
query47	2113	1986	1945	1945
query48	420	323	321	321
query49	714	419	389	389
query50	908	400	384	384
query51	7449	7289	7230	7230
query52	97	87	85	85
query53	255	175	173	173
query54	506	392	391	391
query55	78	74	76	74
query56	261	230	220	220
query57	1319	1114	1143	1114
query58	213	202	207	202
query59	3515	3101	3086	3086
query60	260	271	243	243
query61	102	107	106	106
query62	811	673	690	673
query63	212	179	187	179
query64	1353	662	646	646
query65	3253	3231	3278	3231
query66	631	302	311	302
query67	16207	15595	16017	15595
query68	3955	576	557	557
query69	425	265	253	253
query70	1173	1135	1140	1135
query71	337	257	253	253
query72	6600	4145	4008	4008
query73	785	355	363	355
query74	10071	9084	9001	9001
query75	3389	2682	2674	2674
query76	1735	1083	1117	1083
query77	472	264	263	263
query78	10359	9423	9528	9423
query79	1699	589	601	589
query80	1337	419	430	419
query81	513	226	234	226
query82	1238	119	114	114
query83	202	141	141	141
query84	282	74	72	72
query85	1000	334	299	299
query86	432	305	307	305
query87	4765	4835	4668	4668
query88	3796	2199	2185	2185
query89	425	291	291	291
query90	1913	184	182	182
query91	136	103	108	103
query92	64	49	50	49
query93	2389	540	549	540
query94	837	279	297	279
query95	361	245	247	245
query96	628	284	285	284
query97	2841	2692	2661	2661
query98	228	203	190	190
query99	1621	1312	1319	1312
Total cold run time: 319163 ms
Total hot run time: 198262 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a17f2833d69caaed5b8a48e6a28857e10beeaca, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.26	0.07	0.06
query4	1.64	0.11	0.11
query5	0.45	0.41	0.42
query6	1.18	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.49
query10	0.55	0.55	0.56
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.60	0.62
query14	2.82	2.80	2.79
query15	0.90	0.83	0.82
query16	0.38	0.40	0.38
query17	1.04	1.08	0.99
query18	0.22	0.20	0.21
query19	1.85	1.79	2.04
query20	0.02	0.01	0.01
query21	15.36	0.58	0.57
query22	2.55	2.58	1.69
query23	17.44	0.77	0.86
query24	3.17	1.54	2.06
query25	0.24	0.24	0.05
query26	0.49	0.13	0.14
query27	0.04	0.04	0.05
query28	9.26	1.10	1.09
query29	12.55	3.24	3.20
query30	0.25	0.06	0.07
query31	2.87	0.37	0.38
query32	3.28	0.46	0.47
query33	3.07	3.06	3.01
query34	17.24	4.44	4.46
query35	4.52	4.55	4.45
query36	0.64	0.49	0.48
query37	0.09	0.06	0.07
query38	0.05	0.03	0.04
query39	0.04	0.03	0.02
query40	0.16	0.13	0.14
query41	0.09	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.41 s
Total hot run time: 32.82 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 3, 2024
Copy link
Contributor

github-actions bot commented Dec 3, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Dec 3, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@xy720 xy720 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lide-reed lide-reed merged commit 8bc3c43 into apache:master Dec 3, 2024
29 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 3, 2024
### What problem does this PR solve?

come from: #34988

Problem:
Use JDK11 and call `show proc "/cluster_health/tablet_health"`
frequently, make large number of ForkJoinPool thread leak
github-actions bot pushed a commit that referenced this pull request Dec 3, 2024
### What problem does this PR solve?

come from: #34988

Problem:
Use JDK11 and call `show proc "/cluster_health/tablet_health"`
frequently, make large number of ForkJoinPool thread leak
dataroaring pushed a commit that referenced this pull request Dec 3, 2024
@cambyzju cambyzju removed the dev/2.0.x label Dec 4, 2024
lide-reed pushed a commit that referenced this pull request Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants