Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](csv reader) fix core dump when parsing csv with enclose #45485

Merged
merged 1 commit into from
Dec 25, 2024
Merged

[fix](csv reader) fix core dump when parsing csv with enclose #45485

merged 1 commit into from
Dec 25, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Dec 16, 2024 •
edited
Loading

What problem does this PR solve?

Problem Summary:
Core dump when load csv file to Doris:

==878493==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7fe0fda1c800 at pc 0x559d3db60c48 bp 0x7fde739b2b20 sp 0x7fde739b22d8
READ of size 6 at 0x7fe0fda1c800 thread T545 (_scanner_scan)
    #0 0x559d3db60c47 in memmem (/mnt/disk1/laihui/doris/output/be/lib/doris_be+0x61eefc47) (BuildId: 5f823a8a2b6a81c7)
    #1 0x559d6d02ea60 in unsigned char const* doris::EncloseCsvLineReaderContext::look_for_column_sep_pos<false>(unsigned char const*, unsigned long, char const*, unsigned long) /mnt/disk1/laihui/doris/be/src/vec/exec/format/file_reader/
new_plain_text_line_reader.cpp:111:33
    #2 0x559d6d01f21e in doris::EncloseCsvLineReaderContext::_on_match_enclose(unsigned char const*, unsigned long&) /mnt/disk1/laihui/doris/be/src/vec/exec/format/file_reader/new_plain_text_line_reader.cpp:177:13
    #3 0x559d6d01c7aa in doris::EncloseCsvLineReaderContext::read_line_impl(unsigned char const*, unsigned long) /mnt/disk1/laihui/doris/be/src/vec/exec/format/file_reader/new_plain_text_line_reader.cpp:68:13
    #4 0x559d6cf9e616 in doris::BaseTextLineReaderContext<doris::EncloseCsvLineReaderContext>::read_line(unsigned char const*, unsigned long) /mnt/disk1/laihui/doris/be/src/vec/exec/format/file_reader/new_plain_text_line_reader.h:77:41
    #5 0x559d6d0258a3 in doris::NewPlainTextLineReader::read_line(unsigned char const**, unsigned long*, bool*, doris::io::IOContext const*) /mnt/disk1/laihui/doris/be/src/vec/exec/format/file_reader/new_plain_text_line_reader.cpp:335:48
    #6 0x559d6cf5fc8c in doris::vectorized::CsvReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) /mnt/disk1/laihui/doris/be/src/vec/exec/format/csv/csv_reader.cpp:538:13
    #7 0x559d90f388bb in doris::vectorized::VFileScanner::_get_block_wrapped(doris::RuntimeState*, doris::vectorized::Block*, bool*) /mnt/disk1/laihui/doris/be/src/vec/exec/scan/vfile_scanner.cpp:306:13
    #8 0x559d90f37222 in doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) /mnt/disk1/laihui/doris/be/src/vec/exec/scan/vfile_scanner.cpp:248:17

const uint8_t* delim_pos = find_col_sep_func(curr_start, curr_len, _column_sep.c_str(), _column_sep_len);

curr_len should len - _idx rather than _column_sep_len.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 16, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui sollhui force-pushed the fix_csv_reader_core branch from 977c2e5 to cca1e1d Compare December 16, 2024 11:40
@sollhui
Copy link
Contributor Author

sollhui commented Dec 16, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.85% (10125/26064)
Line Coverage: 29.76% (85087/285911)
Region Coverage: 28.81% (43644/151484)
Branch Coverage: 25.35% (22173/87452)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cca1e1d36a66b40b395d67dcdc8a245cc74a0e90_cca1e1d36a66b40b395d67dcdc8a245cc74a0e90/report/index.html

@sollhui sollhui force-pushed the fix_csv_reader_core branch from cca1e1d to 23d2c1c Compare December 17, 2024 03:43
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.86% (10128/26064)
Line Coverage: 29.78% (85149/285914)
Region Coverage: 28.83% (43671/151495)
Branch Coverage: 25.36% (22182/87454)
Coverage Report: http://coverage.selectdb-in.cc/coverage/23d2c1c27c10df271fa2ea350c64e4497af17e99_23d2c1c27c10df271fa2ea350c64e4497af17e99/report/index.html

@sollhui sollhui force-pushed the fix_csv_reader_core branch from 23d2c1c to 1c35f6c Compare December 19, 2024 07:00
@sollhui
Copy link
Contributor Author

sollhui commented Dec 19, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.87% (10133/26066)
Line Coverage: 29.78% (85175/285970)
Region Coverage: 28.84% (43694/151524)
Branch Coverage: 25.37% (22196/87482)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1c35f6c4d4bc099c0f6d7710a305627cac274abb_1c35f6c4d4bc099c0f6d7710a305627cac274abb/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 39934 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c35f6c4d4bc099c0f6d7710a305627cac274abb, data reload: false

------ Round 1 ----------------------------------
q1	17570	7438	7270	7270
q2	2045	185	172	172
q3	10626	1075	1119	1075
q4	10557	755	683	683
q5	7617	2700	2741	2700
q6	234	148	144	144
q7	1008	616	596	596
q8	9229	1783	1907	1783
q9	6535	6438	6500	6438
q10	7017	2309	2340	2309
q11	471	270	262	262
q12	426	223	221	221
q13	17860	3035	3157	3035
q14	261	214	222	214
q15	546	510	493	493
q16	681	591	589	589
q17	994	583	576	576
q18	7178	6709	6734	6709
q19	1336	1025	1025	1025
q20	472	175	191	175
q21	3976	3258	3160	3160
q22	367	314	305	305
Total cold run time: 107006 ms
Total hot run time: 39934 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7230	7181	7266	7181
q2	325	230	238	230
q3	2905	2776	2982	2776
q4	2113	1785	1826	1785
q5	5690	5618	5661	5618
q6	239	140	141	140
q7	2208	1807	1809	1807
q8	3382	3568	3497	3497
q9	8803	9018	8966	8966
q10	3627	3508	3559	3508
q11	607	519	528	519
q12	849	581	592	581
q13	11124	3171	3078	3078
q14	306	290	283	283
q15	577	505	500	500
q16	692	646	652	646
q17	1858	1631	1597	1597
q18	8422	7698	7817	7698
q19	1710	1608	1508	1508
q20	2110	1903	1933	1903
q21	5560	5427	5466	5427
q22	632	594	590	590
Total cold run time: 70969 ms
Total hot run time: 59838 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196322 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c35f6c4d4bc099c0f6d7710a305627cac274abb, data reload: false

query1	1296	995	916	916
query2	6206	2372	2318	2318
query3	11104	4946	4839	4839
query4	33027	23359	23477	23359
query5	4811	495	450	450
query6	311	190	180	180
query7	3998	305	299	299
query8	301	250	237	237
query9	9607	2682	2701	2682
query10	487	243	236	236
query11	17868	15148	15338	15148
query12	154	110	101	101
query13	1733	420	434	420
query14	11513	6627	7374	6627
query15	255	188	180	180
query16	7741	448	489	448
query17	1577	619	593	593
query18	1927	324	332	324
query19	374	157	167	157
query20	127	128	125	125
query21	209	107	118	107
query22	4619	4860	4658	4658
query23	37631	33668	34177	33668
query24	9982	2467	2541	2467
query25	528	394	395	394
query26	726	147	147	147
query27	2537	337	331	331
query28	7392	2468	2468	2468
query29	685	428	409	409
query30	232	155	153	153
query31	1046	825	847	825
query32	102	57	53	53
query33	763	338	312	312
query34	946	545	520	520
query35	874	781	743	743
query36	1108	942	951	942
query37	123	73	73	73
query38	4136	4258	4427	4258
query39	1527	1485	1491	1485
query40	214	100	97	97
query41	45	41	45	41
query42	114	105	98	98
query43	552	518	509	509
query44	1247	840	833	833
query45	193	167	170	167
query46	1188	757	722	722
query47	2018	1944	1933	1933
query48	430	316	322	316
query49	882	391	404	391
query50	823	396	386	386
query51	7349	7208	7146	7146
query52	101	87	92	87
query53	265	181	182	181
query54	1150	436	405	405
query55	82	79	83	79
query56	265	269	252	252
query57	1301	1150	1176	1150
query58	237	222	224	222
query59	3343	3232	3251	3232
query60	284	263	247	247
query61	108	106	106	106
query62	865	700	690	690
query63	220	184	191	184
query64	3897	674	660	660
query65	3247	3211	3210	3210
query66	750	300	298	298
query67	15961	15622	15532	15532
query68	5929	543	527	527
query69	493	252	253	252
query70	1200	1129	1123	1123
query71	506	256	250	250
query72	6813	4090	4088	4088
query73	781	352	373	352
query74	9902	8892	8796	8796
query75	3461	2625	2621	2621
query76	3957	985	1116	985
query77	557	258	270	258
query78	10249	9497	9391	9391
query79	1119	605	599	599
query80	778	406	457	406
query81	607	236	229	229
query82	676	115	130	115
query83	166	146	140	140
query84	243	71	73	71
query85	1255	318	323	318
query86	394	301	292	292
query87	4772	4340	4327	4327
query88	2992	2231	2166	2166
query89	429	292	295	292
query90	2024	188	185	185
query91	151	104	105	104
query92	62	52	51	51
query93	2477	545	532	532
query94	709	288	327	288
query95	345	244	245	244
query96	632	282	281	281
query97	2824	2696	2708	2696
query98	219	195	193	193
query99	1593	1326	1324	1324
Total cold run time: 305029 ms
Total hot run time: 196322 ms

@sollhui sollhui force-pushed the fix_csv_reader_core branch from 1c35f6c to 0dd0815 Compare December 19, 2024 07:58
@doris-robot
Copy link

ClickBench: Total hot run time: 32.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c35f6c4d4bc099c0f6d7710a305627cac274abb, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.08	0.07
query4	1.61	0.11	0.10
query5	0.41	0.41	0.40
query6	1.13	0.65	0.66
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.52	0.50
query10	0.55	0.58	0.56
query11	0.14	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.62	0.59
query14	2.87	2.89	2.70
query15	0.89	0.82	0.83
query16	0.39	0.38	0.39
query17	1.03	1.02	1.01
query18	0.22	0.20	0.21
query19	1.99	1.90	2.03
query20	0.01	0.01	0.01
query21	15.36	0.58	0.58
query22	2.62	2.00	1.62
query23	16.98	0.96	0.70
query24	3.11	0.64	0.50
query25	0.11	0.15	0.05
query26	0.60	0.13	0.13
query27	0.06	0.04	0.05
query28	11.40	1.09	1.08
query29	12.60	3.14	3.25
query30	0.24	0.06	0.07
query31	2.85	0.39	0.37
query32	3.25	0.46	0.46
query33	3.16	3.30	3.22
query34	17.31	4.52	4.53
query35	4.53	4.54	4.55
query36	0.69	0.48	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.12	0.15
query41	0.09	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 108.35 s
Total hot run time: 32.01 s

@sollhui
Copy link
Contributor Author

sollhui commented Dec 19, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39972 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0dd081551b2fc92fcc4ca3bb4049a591ec1001db, data reload: false

------ Round 1 ----------------------------------
q1	17610	7453	7280	7280
q2	2048	176	180	176
q3	10540	1123	1210	1123
q4	10213	684	712	684
q5	7599	2680	2701	2680
q6	243	148	146	146
q7	995	629	628	628
q8	9257	1875	1958	1875
q9	6620	6449	6395	6395
q10	6955	2317	2343	2317
q11	463	262	264	262
q12	429	227	225	225
q13	17744	3010	2995	2995
q14	251	222	217	217
q15	559	504	494	494
q16	624	589	597	589
q17	978	552	586	552
q18	7355	6678	6746	6678
q19	1346	1076	959	959
q20	483	184	189	184
q21	3993	3208	3286	3208
q22	376	310	305	305
Total cold run time: 106681 ms
Total hot run time: 39972 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7218	7202	7254	7202
q2	326	226	233	226
q3	2918	2777	2995	2777
q4	2065	1768	1802	1768
q5	5633	5671	5635	5635
q6	216	135	137	135
q7	2221	1802	1810	1802
q8	3355	3584	3510	3510
q9	8941	8940	8989	8940
q10	3605	3601	3516	3516
q11	600	504	501	501
q12	803	609	605	605
q13	12164	3108	3090	3090
q14	299	286	279	279
q15	545	507	529	507
q16	680	664	633	633
q17	1846	1581	1566	1566
q18	7846	7406	7443	7406
q19	1675	1508	1458	1458
q20	2021	1822	1844	1822
q21	5494	5337	5382	5337
q22	641	563	566	563
Total cold run time: 71112 ms
Total hot run time: 59278 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26066)
Line Coverage: 29.80% (85205/285970)
Region Coverage: 28.84% (43694/151524)
Branch Coverage: 25.38% (22203/87482)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0dd081551b2fc92fcc4ca3bb4049a591ec1001db_0dd081551b2fc92fcc4ca3bb4049a591ec1001db/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 190951 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0dd081551b2fc92fcc4ca3bb4049a591ec1001db, data reload: false

query1	973	386	374	374
query2	6516	2442	2328	2328
query3	6716	225	223	223
query4	33544	23597	23373	23373
query5	4363	483	472	472
query6	275	194	196	194
query7	4658	304	306	304
query8	313	237	233	233
query9	9791	2710	2701	2701
query10	484	262	248	248
query11	18169	15277	15109	15109
query12	158	108	106	106
query13	1673	423	415	415
query14	9877	7842	7145	7145
query15	315	182	181	181
query16	8312	457	416	416
query17	1829	593	576	576
query18	2143	305	321	305
query19	400	157	149	149
query20	119	108	110	108
query21	208	107	100	100
query22	4595	4135	4200	4135
query23	34451	33659	33824	33659
query24	11582	2525	2522	2522
query25	684	383	382	382
query26	1855	153	151	151
query27	2857	320	335	320
query28	7804	2390	2390	2390
query29	1055	412	407	407
query30	304	147	147	147
query31	1071	788	804	788
query32	98	58	61	58
query33	788	321	288	288
query34	1026	518	525	518
query35	895	747	753	747
query36	1137	928	963	928
query37	290	79	79	79
query38	4172	4186	4141	4141
query39	1487	1448	1409	1409
query40	288	105	103	103
query41	46	44	47	44
query42	121	101	101	101
query43	546	498	497	497
query44	1249	809	805	805
query45	186	173	173	173
query46	1205	734	747	734
query47	1988	1827	1864	1827
query48	438	313	312	312
query49	1289	389	388	388
query50	874	382	390	382
query51	7214	7055	7034	7034
query52	103	100	91	91
query53	272	186	186	186
query54	1204	407	404	404
query55	82	78	77	77
query56	251	242	220	220
query57	1273	1123	1101	1101
query58	247	219	239	219
query59	3307	3113	3060	3060
query60	285	243	249	243
query61	108	109	112	109
query62	919	665	670	665
query63	220	197	191	191
query64	5094	663	630	630
query65	3282	3181	3217	3181
query66	1442	318	320	318
query67	15847	15692	15668	15668
query68	5883	542	551	542
query69	449	256	261	256
query70	1223	1137	1078	1078
query71	373	258	304	258
query72	6612	4377	4286	4286
query73	822	360	378	360
query74	10273	8843	8846	8843
query75	3547	2615	2729	2615
query76	3568	1279	1140	1140
query77	566	269	275	269
query78	10363	9417	9946	9417
query79	2196	600	611	600
query80	965	432	458	432
query81	531	241	227	227
query82	708	126	122	122
query83	251	148	158	148
query84	233	76	78	76
query85	1354	310	301	301
query86	439	310	279	279
query87	4535	4456	4587	4456
query88	3764	2220	2189	2189
query89	429	294	296	294
query90	2212	193	189	189
query91	143	106	111	106
query92	65	54	52	52
query93	1143	538	528	528
query94	1198	295	294	294
query95	359	245	251	245
query96	678	279	288	279
query97	2890	2658	2693	2658
query98	224	198	193	193
query99	1579	1327	1332	1327
Total cold run time: 305942 ms
Total hot run time: 190951 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0dd081551b2fc92fcc4ca3bb4049a591ec1001db, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.61	0.10	0.11
query5	0.43	0.39	0.41
query6	1.17	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.55	0.58	0.57
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.62	0.60
query14	2.73	2.72	2.74
query15	0.90	0.82	0.82
query16	0.39	0.36	0.39
query17	0.96	0.99	1.05
query18	0.23	0.22	0.20
query19	1.86	1.83	1.98
query20	0.01	0.01	0.01
query21	15.38	0.59	0.58
query22	3.10	1.62	2.19
query23	17.12	0.93	0.98
query24	3.53	0.90	2.00
query25	0.18	0.08	0.15
query26	0.59	0.13	0.14
query27	0.05	0.05	0.05
query28	9.66	1.13	1.07
query29	12.55	3.19	3.21
query30	0.24	0.06	0.06
query31	2.86	0.38	0.39
query32	3.25	0.46	0.45
query33	3.18	3.03	3.08
query34	16.64	4.51	4.55
query35	4.52	4.54	4.57
query36	0.65	0.48	0.47
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.03
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 106.75 s
Total hot run time: 32.41 s

@sollhui sollhui force-pushed the fix_csv_reader_core branch from 0dd0815 to 4a41bb5 Compare December 19, 2024 09:24
@sollhui sollhui force-pushed the fix_csv_reader_core branch from f2269ba to 8ddf4c7 Compare December 24, 2024 08:45
@sollhui sollhui reopened this Dec 24, 2024
@sollhui
Copy link
Contributor Author

sollhui commented Dec 24, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39939 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac701eca52bb3a71c01e7392060636db35ca5d9d, data reload: false

------ Round 1 ----------------------------------
q1	17594	7445	7237	7237
q2	2044	192	177	177
q3	10528	1101	1118	1101
q4	10221	744	711	711
q5	7585	2682	2666	2666
q6	234	152	153	152
q7	961	621	617	617
q8	9247	1866	1887	1866
q9	6618	6384	6401	6384
q10	7018	2305	2313	2305
q11	470	267	271	267
q12	423	227	221	221
q13	17780	2942	2989	2942
q14	245	221	218	218
q15	556	514	511	511
q16	668	598	615	598
q17	980	490	592	490
q18	7427	6836	6742	6742
q19	1364	1042	999	999
q20	475	186	196	186
q21	4153	3239	3304	3239
q22	377	310	330	310
Total cold run time: 106968 ms
Total hot run time: 39939 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7222	7217	7202	7202
q2	332	235	235	235
q3	2924	2782	3017	2782
q4	2153	1851	1804	1804
q5	5551	5591	5626	5591
q6	224	142	144	142
q7	2254	1795	1775	1775
q8	3352	3488	3478	3478
q9	9004	8850	8995	8850
q10	3601	3599	3543	3543
q11	599	498	507	498
q12	799	597	595	595
q13	12128	3154	3123	3123
q14	292	276	285	276
q15	597	520	496	496
q16	738	648	651	648
q17	1806	1586	1580	1580
q18	7859	7542	7526	7526
q19	1664	1499	1475	1475
q20	2092	1817	1813	1813
q21	5476	5133	5421	5133
q22	593	553	598	553
Total cold run time: 71260 ms
Total hot run time: 59118 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.80% (10093/26015)
Line Coverage: 29.79% (85151/285863)
Region Coverage: 28.90% (43465/150403)
Branch Coverage: 25.44% (22158/87116)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ac701eca52bb3a71c01e7392060636db35ca5d9d_ac701eca52bb3a71c01e7392060636db35ca5d9d/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 190445 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac701eca52bb3a71c01e7392060636db35ca5d9d, data reload: false

query1	974	376	365	365
query2	6524	2602	2494	2494
query3	6716	224	223	223
query4	33618	23535	23487	23487
query5	4368	489	468	468
query6	293	192	195	192
query7	4626	315	306	306
query8	303	247	228	228
query9	9650	2747	2749	2747
query10	482	258	239	239
query11	18413	15223	15235	15223
query12	159	101	106	101
query13	1706	445	411	411
query14	10815	6634	6715	6634
query15	256	181	190	181
query16	8121	462	431	431
query17	1624	600	590	590
query18	2171	304	305	304
query19	378	158	163	158
query20	124	111	113	111
query21	208	107	108	107
query22	4555	4441	4098	4098
query23	35135	33235	33231	33231
query24	11508	2490	2454	2454
query25	705	416	420	416
query26	1916	154	155	154
query27	2965	323	320	320
query28	8267	2440	2442	2440
query29	1051	409	419	409
query30	305	148	149	148
query31	1007	830	813	813
query32	102	56	56	56
query33	769	279	292	279
query34	1029	506	528	506
query35	856	730	718	718
query36	1125	974	966	966
query37	283	128	77	77
query38	4206	4185	3999	3999
query39	1475	1496	1442	1442
query40	290	102	103	102
query41	50	44	47	44
query42	112	99	101	99
query43	525	494	510	494
query44	1256	803	796	796
query45	184	174	168	168
query46	1161	698	735	698
query47	1960	1857	1854	1854
query48	419	332	323	323
query49	1238	387	414	387
query50	835	388	377	377
query51	7244	7067	6930	6930
query52	99	93	91	91
query53	261	182	192	182
query54	1150	417	404	404
query55	81	75	75	75
query56	264	241	239	239
query57	1263	1116	1113	1113
query58	225	239	241	239
query59	3316	3015	3028	3015
query60	285	261	283	261
query61	111	112	106	106
query62	895	731	731	731
query63	226	186	190	186
query64	5148	685	696	685
query65	3281	3215	3210	3210
query66	1171	317	321	317
query67	15877	15605	15566	15566
query68	5929	556	568	556
query69	428	291	252	252
query70	1213	1146	1143	1143
query71	429	246	249	246
query72	6615	4150	4059	4059
query73	795	357	361	357
query74	10354	9324	8920	8920
query75	3406	2637	2666	2637
query76	3536	1173	1153	1153
query77	586	278	275	275
query78	10350	10040	9562	9562
query79	1114	600	614	600
query80	760	438	430	430
query81	533	230	228	228
query82	651	128	121	121
query83	273	147	149	147
query84	235	69	71	69
query85	974	324	328	324
query86	385	300	269	269
query87	4653	4365	4572	4365
query88	3438	2251	2200	2200
query89	405	286	297	286
query90	1959	187	182	182
query91	139	114	111	111
query92	60	53	51	51
query93	1083	556	563	556
query94	750	281	295	281
query95	367	251	248	248
query96	617	285	291	285
query97	2837	2703	2665	2665
query98	216	196	207	196
query99	1629	1453	1457	1453
Total cold run time: 304434 ms
Total hot run time: 190445 ms

@sollhui sollhui force-pushed the fix_csv_reader_core branch from ac701ec to 61e19ad Compare December 24, 2024 09:27
@sollhui
Copy link
Contributor Author

sollhui commented Dec 24, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39766 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 61e19ad887a032fcaf000401baa31a769f9e0042, data reload: false

------ Round 1 ----------------------------------
q1	17624	7416	7251	7251
q2	2046	182	179	179
q3	10600	1136	1176	1136
q4	10223	709	713	709
q5	7598	2687	2685	2685
q6	240	151	148	148
q7	964	634	602	602
q8	9259	1840	1915	1840
q9	6617	6438	6379	6379
q10	6999	2355	2326	2326
q11	474	262	271	262
q12	417	216	219	216
q13	17774	2888	2976	2888
q14	261	228	223	223
q15	556	519	520	519
q16	664	582	597	582
q17	975	559	503	503
q18	7520	6796	6720	6720
q19	1350	1045	950	950
q20	477	186	194	186
q21	4085	3204	3152	3152
q22	380	310	317	310
Total cold run time: 107103 ms
Total hot run time: 39766 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7273	7236	7287	7236
q2	326	234	232	232
q3	2924	2743	2782	2743
q4	1934	1731	1664	1664
q5	5350	5383	5417	5383
q6	218	138	138	138
q7	2101	1703	1704	1703
q8	3224	3377	3430	3377
q9	8628	8613	8702	8613
q10	3511	3488	3428	3428
q11	607	502	495	495
q12	789	559	578	559
q13	11114	2983	2930	2930
q14	293	267	268	267
q15	549	492	491	491
q16	677	656	661	656
q17	1787	1589	1592	1589
q18	7931	7494	7399	7399
q19	1689	1385	1433	1385
q20	2076	1848	1764	1764
q21	5377	5258	5315	5258
q22	631	568	579	568
Total cold run time: 69009 ms
Total hot run time: 57878 ms

@sollhui sollhui force-pushed the fix_csv_reader_core branch from 61e19ad to a922aca Compare December 24, 2024 09:52
@sollhui
Copy link
Contributor Author

sollhui commented Dec 24, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39847 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a922acae5b0a7a32e4c569d4c06627f8f873f70e, data reload: false

------ Round 1 ----------------------------------
q1	17604	7772	7262	7262
q2	2044	187	169	169
q3	10538	1117	1123	1117
q4	10213	785	735	735
q5	7587	2760	2651	2651
q6	240	147	146	146
q7	980	625	606	606
q8	9242	1884	1943	1884
q9	6665	6423	6357	6357
q10	6942	2293	2337	2293
q11	458	261	267	261
q12	412	222	222	222
q13	17777	2977	2933	2933
q14	243	207	208	207
q15	572	529	496	496
q16	649	601	591	591
q17	984	528	571	528
q18	7212	6816	6788	6788
q19	1335	964	981	964
q20	464	184	187	184
q21	3999	3130	3157	3130
q22	384	323	323	323
Total cold run time: 106544 ms
Total hot run time: 39847 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7204	7222	7246	7222
q2	330	228	236	228
q3	2884	2766	2757	2757
q4	1970	1736	1670	1670
q5	5355	5389	5385	5385
q6	216	139	138	138
q7	2129	1732	1698	1698
q8	3268	3392	3380	3380
q9	8559	8562	8581	8562
q10	3511	3462	3468	3462
q11	594	505	484	484
q12	770	546	603	546
q13	11977	2925	2918	2918
q14	305	261	283	261
q15	554	505	499	499
q16	669	643	642	642
q17	1817	1576	1565	1565
q18	7851	7321	7491	7321
q19	1678	1474	1510	1474
q20	2072	1805	1819	1805
q21	5485	5304	5355	5304
q22	625	610	571	571
Total cold run time: 69823 ms
Total hot run time: 57892 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.80% (10093/26015)
Line Coverage: 29.77% (85112/285863)
Region Coverage: 28.90% (43461/150403)
Branch Coverage: 25.43% (22155/87116)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a922acae5b0a7a32e4c569d4c06627f8f873f70e_a922acae5b0a7a32e4c569d4c06627f8f873f70e/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 190626 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a922acae5b0a7a32e4c569d4c06627f8f873f70e, data reload: false

query1	964	372	374	372
query2	6523	2416	2439	2416
query3	6703	215	215	215
query4	33831	23361	23841	23361
query5	4743	481	468	468
query6	284	184	197	184
query7	4629	302	296	296
query8	295	226	219	219
query9	9834	2754	2755	2754
query10	484	266	250	250
query11	18216	15176	15223	15176
query12	153	103	99	99
query13	1649	415	402	402
query14	9529	7790	7056	7056
query15	311	175	182	175
query16	7459	425	459	425
query17	1673	579	548	548
query18	1941	297	297	297
query19	374	150	147	147
query20	117	107	110	107
query21	207	105	102	102
query22	4446	4194	4191	4191
query23	34264	33365	33540	33365
query24	11469	2505	2446	2446
query25	715	419	414	414
query26	1867	157	166	157
query27	2938	329	329	329
query28	8054	2473	2456	2456
query29	982	431	438	431
query30	309	152	150	150
query31	1062	800	819	800
query32	104	59	58	58
query33	791	307	299	299
query34	981	521	557	521
query35	858	761	739	739
query36	1092	960	949	949
query37	235	73	71	71
query38	4133	4306	4151	4151
query39	1483	1439	1427	1427
query40	289	99	98	98
query41	50	45	49	45
query42	118	100	98	98
query43	523	496	480	480
query44	1241	809	805	805
query45	186	168	159	159
query46	1163	693	696	693
query47	1952	1842	1874	1842
query48	409	328	314	314
query49	1280	392	404	392
query50	812	378	379	378
query51	7209	7083	7046	7046
query52	100	93	89	89
query53	257	187	182	182
query54	1259	414	412	412
query55	83	77	77	77
query56	254	264	231	231
query57	1293	1113	1145	1113
query58	230	218	222	218
query59	3190	3130	3093	3093
query60	274	245	248	245
query61	110	114	111	111
query62	946	738	729	729
query63	218	184	196	184
query64	5131	675	639	639
query65	3321	3205	3220	3205
query66	1414	314	310	310
query67	15915	15495	15572	15495
query68	5816	554	554	554
query69	445	249	261	249
query70	1211	1136	1149	1136
query71	430	244	247	244
query72	6403	4103	4047	4047
query73	773	369	357	357
query74	10681	9258	8884	8884
query75	3483	2649	2683	2649
query76	3552	1241	1067	1067
query77	569	273	270	270
query78	10289	9383	9291	9291
query79	2534	622	617	617
query80	1078	479	439	439
query81	531	228	232	228
query82	838	124	118	118
query83	245	159	147	147
query84	231	71	71	71
query85	1438	327	310	310
query86	465	298	297	297
query87	4524	4417	4514	4417
query88	4155	2222	2189	2189
query89	412	297	299	297
query90	2139	188	189	188
query91	137	101	116	101
query92	66	50	56	50
query93	1492	555	561	555
query94	1051	281	288	281
query95	350	250	250	250
query96	618	277	285	277
query97	2862	2693	2700	2693
query98	215	201	201	201
query99	1692	1446	1450	1446
Total cold run time: 305654 ms
Total hot run time: 190626 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a922acae5b0a7a32e4c569d4c06627f8f873f70e, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.24	0.07	0.06
query4	1.61	0.10	0.09
query5	0.43	0.39	0.41
query6	1.17	0.65	0.65
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.52	0.49
query10	0.56	0.58	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.74	2.76	2.71
query15	0.90	0.83	0.82
query16	0.40	0.39	0.38
query17	1.04	1.02	0.98
query18	0.23	0.22	0.21
query19	1.88	1.83	1.89
query20	0.01	0.01	0.01
query21	15.38	0.62	0.58
query22	2.64	2.87	1.86
query23	16.95	0.96	0.89
query24	2.92	1.44	0.75
query25	0.27	0.35	0.05
query26	0.33	0.15	0.14
query27	0.04	0.04	0.05
query28	10.57	1.11	1.08
query29	12.82	3.28	3.24
query30	0.25	0.06	0.07
query31	2.85	0.38	0.39
query32	3.24	0.48	0.46
query33	3.06	3.18	3.17
query34	17.11	4.47	4.44
query35	4.50	4.48	4.47
query36	0.68	0.48	0.48
query37	0.09	0.05	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.92 s
Total hot run time: 32.47 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@liaoxin01 liaoxin01 merged commit 3216257 into apache:master Dec 25, 2024
22 of 23 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 25, 2024
@sollhui
[fix](csv reader) fix core dump when parsing csv with enclose (#45485)
95f0040
@github-actions github-actions bot mentioned this pull request Dec 25, 2024
Merged
github-actions bot pushed a commit that referenced this pull request Dec 25, 2024
@sollhui
[fix](csv reader) fix core dump when parsing csv with enclose (#45485)
9ddde64
@github-actions github-actions bot mentioned this pull request Dec 25, 2024
Merged
yiguolei pushed a commit that referenced this pull request Dec 25, 2024
@github-actions @sollhui
branch-2.1: [fix](csv reader) fix core dump when parsing csv with enc…
547e88b 
…lose #45485 (#45889)

Cherry-picked from #45485

Co-authored-by: hui lai <[email protected]>
dataroaring pushed a commit that referenced this pull request Dec 26, 2024
@github-actions @sollhui
branch-3.0: [fix](csv reader) fix core dump when parsing csv with enc…
d729009 
…lose #45485 (#45888)

Cherry-picked from #45485

Co-authored-by: hui lai <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TangSiyang2001 TangSiyang2001 TangSiyang2001 approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@kaijchen kaijchen kaijchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@sollhui @Thearas @doris-robot @dataroaring @kaijchen @liaoxin01 @TangSiyang2001 @yiguolei