[enhancement](exception) pageio method should return error when allocate memory failed #40020

yiguolei · 2024-08-28T03:17:52Z

Proposed changes

pageio method should return error when allocate memory failed just for point query.
variant parsing logic should return error when allocate memory failed.

…ate memory failed

doris-robot · 2024-08-28T03:17:56Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

yiguolei · 2024-08-28T03:18:57Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-08-28T03:24:49Z

be/src/olap/rowset/segment_v2/page_io.cpp

+    RETURN_IF_CATCH_EXCEPTION({return read_and_decompress_page_(opts, handle, body, footer)});
+}
+
+Status PageIO::read_and_decompress_page_(const PageReadOptions& opts, PageHandle* handle,


warning: function 'read_and_decompress_page_' exceeds recommended size/complexity thresholds [readability-function-size]

Status PageIO::read_and_decompress_page_(const PageReadOptions& opts, PageHandle* handle, ^

Additional context

be/src/olap/rowset/segment_v2/page_io.cpp:118: 115 lines including whitespace and comments (threshold 80)

Status PageIO::read_and_decompress_page_(const PageReadOptions& opts, PageHandle* handle, ^

yiguolei · 2024-08-28T06:06:51Z

run buildall

yiguolei · 2024-08-28T07:03:41Z

run buildall

github-actions

clang-tidy made some suggestions

be/src/olap/rowset/segment_v2/page_io.cpp

yiguolei · 2024-08-28T08:02:27Z

run buildall

yiguolei · 2024-08-28T08:22:58Z

run buildall

yiguolei · 2024-08-28T10:08:59Z

run buildall

doris-robot · 2024-08-28T10:48:32Z

TPC-H: Total hot run time: 37820 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f097eaf7bbd4e7f8117be0c9ce5474dc5db07a06, data reload: false

------ Round 1 ----------------------------------
q1	17622	4478	4290	4290
q2	2022	182	174	174
q3	11969	958	1148	958
q4	10514	755	729	729
q5	7751	2794	2847	2794
q6	225	138	133	133
q7	950	605	598	598
q8	9349	2055	2073	2055
q9	7220	6509	6561	6509
q10	6993	2211	2160	2160
q11	460	243	242	242
q12	395	226	222	222
q13	17771	3090	3005	3005
q14	280	231	235	231
q15	532	491	503	491
q16	586	510	511	510
q17	972	699	742	699
q18	7310	6884	6813	6813
q19	1386	1148	1086	1086
q20	682	325	336	325
q21	3931	2775	2974	2775
q22	1140	1035	1021	1021
Total cold run time: 110060 ms
Total hot run time: 37820 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4287	4294	4287
q2	372	274	280	274
q3	2879	2704	2689	2689
q4	1927	1640	1617	1617
q5	5566	5664	5677	5664
q6	234	140	137	137
q7	2243	1834	1845	1834
q8	3297	3452	3436	3436
q9	8845	8791	8767	8767
q10	3552	3371	3399	3371
q11	615	511	503	503
q12	815	659	695	659
q13	14474	3277	3169	3169
q14	313	291	291	291
q15	542	481	495	481
q16	631	593	589	589
q17	1820	1547	1511	1511
q18	8125	7955	7994	7955
q19	1704	1537	1531	1531
q20	2152	1910	1937	1910
q21	5804	5543	5375	5375
q22	1148	1079	1047	1047
Total cold run time: 71441 ms
Total hot run time: 57097 ms

doris-robot · 2024-08-28T11:00:49Z

TPC-DS: Total hot run time: 194573 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f097eaf7bbd4e7f8117be0c9ce5474dc5db07a06, data reload: false

query1	1223	876	861	861
query2	6289	2052	1901	1901
query3	10731	4179	4233	4179
query4	60247	25202	23170	23170
query5	5495	509	519	509
query6	412	165	169	165
query7	5867	297	298	297
query8	282	198	204	198
query9	8963	2511	2499	2499
query10	490	287	259	259
query11	17356	15133	15294	15133
query12	160	109	103	103
query13	1552	420	390	390
query14	11459	7070	7546	7070
query15	255	179	181	179
query16	7093	506	405	405
query17	1127	577	601	577
query18	1276	297	306	297
query19	286	149	152	149
query20	124	111	114	111
query21	205	105	104	104
query22	4699	4761	4614	4614
query23	34297	33618	33428	33428
query24	5976	2880	2909	2880
query25	565	394	405	394
query26	688	157	159	157
query27	1790	277	278	277
query28	3810	2122	2115	2115
query29	700	422	433	422
query30	238	157	158	157
query31	928	781	789	781
query32	83	51	55	51
query33	454	287	278	278
query34	907	479	468	468
query35	833	727	704	704
query36	1064	945	936	936
query37	147	91	96	91
query38	3983	3783	3846	3783
query39	1450	1386	1353	1353
query40	202	127	114	114
query41	45	44	44	44
query42	115	96	95	95
query43	506	472	481	472
query44	1107	743	734	734
query45	196	164	162	162
query46	1139	735	769	735
query47	1924	1876	1837	1837
query48	367	306	297	297
query49	770	447	436	436
query50	864	418	415	415
query51	7076	7068	7003	7003
query52	94	88	86	86
query53	258	183	181	181
query54	577	449	441	441
query55	75	78	75	75
query56	283	253	256	253
query57	1195	1069	1108	1069
query58	219	234	241	234
query59	3057	2911	2777	2777
query60	296	281	277	277
query61	104	100	100	100
query62	774	673	645	645
query63	214	186	187	186
query64	2836	715	656	656
query65	3253	3140	3160	3140
query66	706	336	343	336
query67	15351	15478	15455	15455
query68	2995	579	597	579
query69	389	286	282	282
query70	1170	1128	1084	1084
query71	358	275	274	274
query72	5332	4116	4180	4116
query73	804	337	328	328
query74	9075	8696	8787	8696
query75	3363	2677	2691	2677
query76	1552	1079	1012	1012
query77	543	329	374	329
query78	9844	10245	11110	10245
query79	1105	537	525	525
query80	716	487	489	487
query81	523	236	229	229
query82	260	143	145	143
query83	173	149	148	148
query84	301	76	78	76
query85	692	291	281	281
query86	304	290	308	290
query87	4516	4216	4305	4216
query88	3519	2346	2410	2346
query89	384	286	286	286
query90	1973	191	190	190
query91	124	97	96	96
query92	65	50	51	50
query93	1080	532	541	532
query94	728	302	296	296
query95	352	263	261	261
query96	648	271	273	271
query97	3230	3086	3049	3049
query98	215	204	196	196
query99	1518	1286	1283	1283
Total cold run time: 304958 ms
Total hot run time: 194573 ms

doris-robot · 2024-08-28T11:06:26Z

ClickBench: Total hot run time: 31.87 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f097eaf7bbd4e7f8117be0c9ce5474dc5db07a06, data reload: false

query1	0.05	0.04	0.03
query2	0.09	0.04	0.04
query3	0.22	0.06	0.05
query4	1.70	0.08	0.08
query5	0.51	0.49	0.48
query6	1.13	0.75	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.56	0.49	0.49
query10	0.54	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.62	0.59	0.58
query14	2.02	2.07	2.06
query15	0.88	0.82	0.81
query16	0.37	0.39	0.38
query17	0.95	1.01	0.99
query18	0.21	0.20	0.20
query19	1.80	1.76	1.75
query20	0.01	0.02	0.01
query21	15.38	0.66	0.66
query22	4.42	6.89	1.81
query23	18.29	1.38	1.29
query24	2.08	0.23	0.23
query25	0.16	0.08	0.09
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.28	1.04	0.99
query29	12.61	3.29	3.31
query30	0.24	0.06	0.05
query31	2.89	0.40	0.38
query32	3.25	0.47	0.47
query33	2.95	2.98	3.04
query34	17.20	4.42	4.44
query35	4.46	4.50	4.41
query36	0.66	0.49	0.48
query37	0.19	0.16	0.16
query38	0.16	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.05	0.04
Total cold run time: 111.04 s
Total hot run time: 31.87 s

HappenLee

LGTM

github-actions · 2024-08-28T12:58:15Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-28T12:58:17Z

PR approved by anyone and no changes requested.

eldenmoon

LGTM

…ate memory failed (apache#40020) 1. pageio method should return error when allocate memory failed just for point query. 2. variant parsing logic should return error when allocate memory failed. --------- Co-authored-by: yiguolei <[email protected]>

…40883) ## Proposed changes pick #39536 #39913 #39964 #40020 --------- Co-authored-by: yiguolei <[email protected]>

…ate memory failed (#40020) ## Proposed changes 1. pageio method should return error when allocate memory failed just for point query. 2. variant parsing logic should return error when allocate memory failed. --------- Co-authored-by: yiguolei <[email protected]>

Doris-Extras added 2 commits August 28, 2024 10:52

[enhancement](exception) pageio method should return error when alloc…

5897f74

…ate memory failed

f

c765ae9

github-actions bot reviewed Aug 28, 2024

View reviewed changes

f

439bba7

f

961dab7

github-actions bot reviewed Aug 28, 2024

View reviewed changes

be/src/olap/rowset/segment_v2/page_io.cpp Show resolved Hide resolved

f

ab17602

f

f95c435

f

f097eaf

zhangstar333 approved these changes Aug 28, 2024

View reviewed changes

HappenLee approved these changes Aug 28, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 28, 2024

github-actions bot added the reviewed label Aug 28, 2024

eldenmoon approved these changes Aug 28, 2024

View reviewed changes

yiguolei merged commit f174a30 into apache:master Aug 28, 2024
29 of 32 checks passed

yiguolei added the dev/2.1.x label Aug 28, 2024

yiguolei mentioned this pull request Sep 9, 2024

pick exception handle logic to 2.1 #40547

Closed

xinyiZzz mentioned this pull request Sep 9, 2024

[branch-2.1](memory) Pick query memory tracker value is equal to 0 and exception handle #40564

Closed

yiguolei mentioned this pull request Sep 15, 2024

[cherrypick](branch21) pick some exception handleing pr in page io #40883

Merged

yiguolei added a commit that referenced this pull request Sep 16, 2024

[cherrypick](branch21) pick some exception handleing pr in page io (#…

a6e2364

…40883) ## Proposed changes pick #39536 #39913 #39964 #40020 --------- Co-authored-by: yiguolei <[email protected]>

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Sep 16, 2024

dataroaring added the dev/3.0.3-merged label Oct 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[enhancement](exception) pageio method should return error when allocate memory failed #40020

[enhancement](exception) pageio method should return error when allocate memory failed #40020

yiguolei commented Aug 28, 2024

doris-robot commented Aug 28, 2024

yiguolei commented Aug 28, 2024

github-actions bot left a comment

github-actions bot Aug 28, 2024

yiguolei commented Aug 28, 2024

yiguolei commented Aug 28, 2024

github-actions bot left a comment

yiguolei commented Aug 28, 2024

yiguolei commented Aug 28, 2024

yiguolei commented Aug 28, 2024

doris-robot commented Aug 28, 2024

doris-robot commented Aug 28, 2024

doris-robot commented Aug 28, 2024

HappenLee left a comment

github-actions bot commented Aug 28, 2024

github-actions bot commented Aug 28, 2024

eldenmoon left a comment

[enhancement](exception) pageio method should return error when allocate memory failed #40020

[enhancement](exception) pageio method should return error when allocate memory failed #40020

Conversation

yiguolei commented Aug 28, 2024

Proposed changes

doris-robot commented Aug 28, 2024

yiguolei commented Aug 28, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Aug 28, 2024

Choose a reason for hiding this comment

yiguolei commented Aug 28, 2024

yiguolei commented Aug 28, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

yiguolei commented Aug 28, 2024

yiguolei commented Aug 28, 2024

yiguolei commented Aug 28, 2024

doris-robot commented Aug 28, 2024

doris-robot commented Aug 28, 2024

doris-robot commented Aug 28, 2024

HappenLee left a comment

Choose a reason for hiding this comment

github-actions bot commented Aug 28, 2024

github-actions bot commented Aug 28, 2024

eldenmoon left a comment

Choose a reason for hiding this comment