Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve](stream load) set NumberLoadedRows to zero when stream load transaction failed (#41946) #42291

Merged
merged 1 commit into from
Oct 23, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Oct 22, 2024

pick (#41946)

Set NumberLoadedRows to zero when stream load failed.

before:

stream load result: {
    "TxnId": 8589,
    "Label": "c8e7c4fe-56b2-4e3b-b4cc-4f2a94cdd003",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 3,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 2,
    "NumberUnselectedRows": 0,
    "LoadBytes": 77,
    "LoadTimeMs": 78,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 72,
    "ReceiveDataTimeMs": 7,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "XXX"
}

now:

stream load result: {
    "TxnId": 8589,
    "Label": "c8e7c4fe-56b2-4e3b-b4cc-4f2a94cdd003",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 3,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 2,
    "NumberUnselectedRows": 0,
    "LoadBytes": 77,
    "LoadTimeMs": 78,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 72,
    "ReceiveDataTimeMs": 7,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "XXX"
}

…transaction failed (#41946)

Set NumberLoadedRows to zero when stream load failed.

before:
```
stream load result: {
    "TxnId": 8589,
    "Label": "c8e7c4fe-56b2-4e3b-b4cc-4f2a94cdd003",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 3,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 2,
    "NumberUnselectedRows": 0,
    "LoadBytes": 77,
    "LoadTimeMs": 78,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 72,
    "ReceiveDataTimeMs": 7,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "XXX"
}
```

now:
```
stream load result: {
    "TxnId": 8589,
    "Label": "c8e7c4fe-56b2-4e3b-b4cc-4f2a94cdd003",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 3,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 2,
    "NumberUnselectedRows": 0,
    "LoadBytes": 77,
    "LoadTimeMs": 78,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 72,
    "ReceiveDataTimeMs": 7,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "XXX"
}
```
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Oct 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40584 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b3b4bda7b313c63d24b8192a7a22b83751ae8a7e, data reload: false

------ Round 1 ----------------------------------
q1	17586	7434	7301	7301
q2	2038	151	150	150
q3	10581	1135	1156	1135
q4	10211	763	742	742
q5	7742	2782	2825	2782
q6	234	150	150	150
q7	974	616	604	604
q8	9358	1915	1926	1915
q9	7924	6422	6390	6390
q10	7018	2262	2316	2262
q11	445	244	246	244
q12	478	210	209	209
q13	17779	2973	2966	2966
q14	256	203	207	203
q15	562	514	536	514
q16	682	605	606	605
q17	978	503	535	503
q18	7323	6597	6610	6597
q19	1389	999	995	995
q20	579	269	274	269
q21	3897	3234	3064	3064
q22	1114	984	1012	984
Total cold run time: 109148 ms
Total hot run time: 40584 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7278	7296	7255	7255
q2	322	232	226	226
q3	2881	2760	2717	2717
q4	1928	1695	1671	1671
q5	5386	5399	5429	5399
q6	227	141	143	141
q7	2038	1672	1668	1668
q8	3206	3392	3391	3391
q9	8531	8478	8500	8478
q10	3437	3422	3377	3377
q11	567	474	479	474
q12	750	553	562	553
q13	16928	3003	2982	2982
q14	287	255	253	253
q15	544	510	498	498
q16	681	654	653	653
q17	1777	1572	1546	1546
q18	7653	7406	7338	7338
q19	1630	1553	1484	1484
q20	2007	1755	1785	1755
q21	5159	4951	5009	4951
q22	1102	981	979	979
Total cold run time: 74319 ms
Total hot run time: 57789 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189934 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b3b4bda7b313c63d24b8192a7a22b83751ae8a7e, data reload: false

query1	991	370	365	365
query2	6530	2083	2017	2017
query3	6696	209	216	209
query4	34140	23420	23426	23420
query5	4380	484	487	484
query6	269	174	181	174
query7	4642	310	327	310
query8	257	210	217	210
query9	9265	2683	2670	2670
query10	476	297	280	280
query11	18094	15180	15161	15161
query12	153	98	99	98
query13	1631	407	401	401
query14	9950	7248	7263	7248
query15	244	166	170	166
query16	8043	468	468	468
query17	1602	542	534	534
query18	2056	294	297	294
query19	216	136	141	136
query20	119	111	104	104
query21	204	98	98	98
query22	4417	4033	4071	4033
query23	34594	33994	34676	33994
query24	11174	2795	2844	2795
query25	550	376	377	376
query26	698	162	162	162
query27	2095	297	293	293
query28	6839	2489	2471	2471
query29	636	418	409	409
query30	317	160	151	151
query31	1023	777	818	777
query32	95	54	56	54
query33	774	274	284	274
query34	1043	527	493	493
query35	881	713	723	713
query36	1130	923	938	923
query37	139	85	84	84
query38	3945	3859	3904	3859
query39	1477	1408	1395	1395
query40	203	99	93	93
query41	49	47	45	45
query42	118	96	98	96
query43	516	477	491	477
query44	1301	769	785	769
query45	192	165	163	163
query46	1130	708	714	708
query47	1866	1772	1773	1772
query48	458	357	374	357
query49	1136	371	377	371
query50	826	398	404	398
query51	7052	6778	6916	6778
query52	99	94	86	86
query53	263	182	182	182
query54	1136	463	453	453
query55	74	73	77	73
query56	253	237	241	237
query57	1188	1094	1119	1094
query58	246	263	247	247
query59	3061	2899	2867	2867
query60	279	267	250	250
query61	103	97	100	97
query62	870	662	674	662
query63	218	192	184	184
query64	3876	729	705	705
query65	3250	3217	3159	3159
query66	866	321	313	313
query67	15567	15402	15255	15255
query68	3643	558	552	552
query69	538	306	291	291
query70	1157	1078	1120	1078
query71	348	271	274	271
query72	7218	4028	4052	4028
query73	766	350	344	344
query74	10052	8917	8993	8917
query75	3403	2630	2623	2623
query76	2162	893	898	893
query77	735	289	294	289
query78	9847	9145	9539	9145
query79	2339	591	590	590
query80	2721	454	465	454
query81	597	243	248	243
query82	670	151	150	150
query83	318	148	185	148
query84	302	76	78	76
query85	2187	297	283	283
query86	488	298	311	298
query87	4539	4226	4258	4226
query88	3941	2424	2409	2409
query89	405	290	287	287
query90	2074	183	191	183
query91	173	138	145	138
query92	71	50	45	45
query93	1687	538	533	533
query94	1116	303	294	294
query95	354	254	257	254
query96	617	280	284	280
query97	3326	3131	3126	3126
query98	222	196	195	195
query99	1926	1297	1298	1297
Total cold run time: 297923 ms
Total hot run time: 189934 ms

@liaoxin01 liaoxin01 merged commit 7630384 into apache:branch-3.0 Oct 23, 2024
20 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants