Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [Fix](Insert-Job)Resolving Task Status Discrepancies in Query Execution Records #45342 #45405

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

Cherry-picked from #45342

…on Records (#45342)

### What problem does this PR solve?


When querying the execution records of tasks, the current system has two
sources that may describe task status:

Task status in the scheduling module: Represents the processing status
of tasks within the scheduling system.
Task status in LoadManager: Specifically refers to the status of
Insert-type tasks, containing more detailed execution information.

When the data in the LoadManager is deleted or expires, the completion
status will revert to the task status from the scheduling system, which
could cause confusion. Therefore, consistency needs to be maintained.

Since these two sources may have inconsistencies in their status
information, they can easily cause confusion for users and developers.
@Thearas
Copy link
Contributor

Thearas commented Dec 13, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Dec 13, 2024
@Thearas
Copy link
Contributor

Thearas commented Dec 13, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40082 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit efc5fa13cfc2892a3f5e8cdd445fe21d4e2df729, data reload: false

------ Round 1 ----------------------------------
q1	17667	7348	7255	7255
q2	2056	171	167	167
q3	10720	1050	1121	1050
q4	10547	714	664	664
q5	7725	2800	2769	2769
q6	241	147	149	147
q7	969	607	592	592
q8	9573	1908	1981	1908
q9	7038	6339	6331	6331
q10	6937	2266	2271	2266
q11	454	258	260	258
q12	400	212	205	205
q13	17784	2958	2974	2958
q14	229	217	203	203
q15	550	514	526	514
q16	652	594	613	594
q17	954	601	520	520
q18	7176	6601	6503	6503
q19	1546	997	962	962
q20	492	190	191	190
q21	3965	3103	3060	3060
q22	1084	979	966	966
Total cold run time: 108759 ms
Total hot run time: 40082 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7275	7222	7194	7194
q2	323	235	231	231
q3	2828	2800	2829	2800
q4	2024	1741	1805	1741
q5	5624	5693	5702	5693
q6	222	137	139	137
q7	2187	1753	1785	1753
q8	3283	3527	3460	3460
q9	8662	8804	8776	8776
q10	3507	3500	3489	3489
q11	609	510	496	496
q12	762	602	622	602
q13	16449	3149	3130	3130
q14	306	269	275	269
q15	566	512	518	512
q16	697	669	667	667
q17	1816	1616	1624	1616
q18	8222	7852	7514	7514
q19	4664	1536	1437	1437
q20	2069	1856	1834	1834
q21	5474	5206	5239	5206
q22	1070	1029	988	988
Total cold run time: 78639 ms
Total hot run time: 59545 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195021 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit efc5fa13cfc2892a3f5e8cdd445fe21d4e2df729, data reload: false

query1	1277	907	913	907
query2	6249	2042	2022	2022
query3	10796	4022	3992	3992
query4	66787	29077	23406	23406
query5	5383	442	418	418
query6	432	170	178	170
query7	5632	317	309	309
query8	306	231	232	231
query9	9394	2697	2681	2681
query10	525	266	251	251
query11	17756	15274	15827	15274
query12	163	102	99	99
query13	1554	446	432	432
query14	10450	6482	6764	6482
query15	216	181	179	179
query16	7383	498	500	498
query17	1085	577	568	568
query18	1987	316	327	316
query19	215	156	153	153
query20	115	114	111	111
query21	70	45	50	45
query22	4680	4749	4529	4529
query23	35362	34085	34291	34085
query24	6080	2941	2856	2856
query25	514	424	408	408
query26	692	169	165	165
query27	1898	296	300	296
query28	4241	2565	2519	2519
query29	715	479	440	440
query30	252	183	168	168
query31	1027	854	794	794
query32	62	55	55	55
query33	424	275	286	275
query34	916	497	511	497
query35	839	762	757	757
query36	1085	949	943	943
query37	130	72	72	72
query38	4034	4000	3962	3962
query39	1528	1493	1499	1493
query40	150	83	81	81
query41	51	46	47	46
query42	108	98	101	98
query43	530	492	492	492
query44	1166	797	814	797
query45	201	178	180	178
query46	1179	733	724	724
query47	1993	1949	1987	1949
query48	463	374	371	371
query49	732	376	393	376
query50	844	423	422	422
query51	7278	7155	7096	7096
query52	94	84	84	84
query53	250	184	175	175
query54	567	451	437	437
query55	73	72	73	72
query56	241	233	231	231
query57	1185	1095	1106	1095
query58	202	190	196	190
query59	3281	3122	2789	2789
query60	273	241	235	235
query61	112	103	103	103
query62	768	661	653	653
query63	204	177	178	177
query64	1780	683	654	654
query65	3247	3152	3182	3152
query66	707	299	295	295
query67	16055	15339	15542	15339
query68	4613	566	544	544
query69	415	260	252	252
query70	1162	1115	1135	1115
query71	406	245	249	245
query72	6278	3869	3898	3869
query73	754	342	339	339
query74	10163	8864	8985	8864
query75	3320	2638	2639	2638
query76	2165	1114	1057	1057
query77	502	260	258	258
query78	10649	9660	9443	9443
query79	8022	599	586	586
query80	2054	417	415	415
query81	556	238	239	238
query82	1402	113	116	113
query83	262	139	140	139
query84	287	78	82	78
query85	1722	304	292	292
query86	462	272	288	272
query87	4418	4172	4226	4172
query88	5602	2360	2382	2360
query89	415	284	282	282
query90	2084	176	178	176
query91	179	160	161	160
query92	61	45	46	45
query93	7056	536	533	533
query94	900	290	271	271
query95	339	244	242	242
query96	614	277	292	277
query97	3360	3124	3107	3107
query98	219	201	200	200
query99	1638	1270	1272	1270
Total cold run time: 338067 ms
Total hot run time: 195021 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants