Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Export) Fix an export error when lower_case_table_names=1 #28389

Merged
merged 2 commits into from
Dec 17, 2023
Merged

[fix](Export) Fix an export error when lower_case_table_names=1 #28389

merged 2 commits into from
Dec 17, 2023

Conversation

BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Dec 14, 2023 •
edited
Loading

Proposed changes

Issue Number: close #xxx

if lower_case_table_names=1 and using old optimizer, when the table we want to export belongs to a database containing a capital letter, we will get an error.
For example, if the db named XXX, and the table we want to export is named yyy, then when we execute export job, we will get the error:
Unknown table 'yyy' in default_cluster:XXX

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@BePPPower BePPPower changed the title [fix](Export) Fix an export error when lower case table names=1 [fix](Export) Fix an export error when lower_case_table_names=1 Dec 14, 2023
@BePPPower BePPPower mentioned this pull request Dec 14, 2023
Merged
@BePPPower
Copy link
Contributor Author

run buildall

1 similar comment
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 6ee8f0f233b43d5db4a95f758aabe7c460e87cce, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4752	4481	4482	4481
q2	377	166	159	159
q3	1485	1270	1217	1217
q4	1135	909	920	909
q5	3175	3162	3210	3162
q6	244	129	129	129
q7	983	482	477	477
q8	2208	2222	2202	2202
q9	6705	6690	6666	6666
q10	3217	3280	3277	3277
q11	334	195	207	195
q12	351	211	206	206
q13	4566	3828	3816	3816
q14	241	218	211	211
q15	571	524	534	524
q16	439	389	380	380
q17	1017	596	563	563
q18	7230	6987	7086	6987
q19	1525	1392	1424	1392
q20	505	305	316	305
q21	3098	2656	2652	2652
q22	351	288	293	288
Total cold run time: 44509 ms
Total hot run time: 40198 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4380	4386	4412	4386
q2	269	167	176	167
q3	3535	3527	3523	3523
q4	2402	2410	2404	2404
q5	5734	5740	5756	5740
q6	239	121	123	121
q7	2379	1868	1868	1868
q8	3550	3543	3527	3527
q9	9016	9041	8999	8999
q10	3924	3994	4010	3994
q11	510	395	383	383
q12	765	596	578	578
q13	4287	3571	3558	3558
q14	282	255	254	254
q15	575	519	520	519
q16	521	446	461	446
q17	1885	1835	1854	1835
q18	8706	8315	8345	8315
q19	1719	1746	1760	1746
q20	2261	1942	1945	1942
q21	6550	6181	6175	6175
q22	512	431	420	420
Total cold run time: 64001 ms
Total hot run time: 60900 ms

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 6ee8f0f233b43d5db4a95f758aabe7c460e87cce, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4742	4469	4479	4469
q2	365	149	158	149
q3	1465	1280	1179	1179
q4	1119	935	918	918
q5	3133	3166	3168	3166
q6	248	126	128	126
q7	994	494	486	486
q8	2213	2190	2204	2190
q9	6690	6701	6652	6652
q10	3216	3247	3300	3247
q11	332	211	199	199
q12	351	206	207	206
q13	4581	3812	3830	3812
q14	238	207	211	207
q15	566	530	528	528
q16	439	382	391	382
q17	1002	574	605	574
q18	7125	6947	7088	6947
q19	1522	1376	1408	1376
q20	509	322	289	289
q21	3086	2644	2622	2622
q22	353	280	288	280
Total cold run time: 44289 ms
Total hot run time: 40004 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4408	4396	4407	4396
q2	265	162	167	162
q3	3528	3532	3512	3512
q4	2403	2383	2381	2381
q5	5727	5748	5726	5726
q6	240	122	121	121
q7	2383	1890	1842	1842
q8	3517	3524	3522	3522
q9	9052	8993	9025	8993
q10	3901	3994	3998	3994
q11	507	395	386	386
q12	765	595	598	595
q13	4293	3581	3568	3568
q14	277	253	254	253
q15	575	526	537	526
q16	490	451	466	451
q17	1888	1882	1840	1840
q18	8679	8280	8284	8280
q19	1728	1734	1713	1713
q20	2250	1943	1933	1933
q21	6520	6194	6177	6177
q22	523	444	405	405
Total cold run time: 63919 ms
Total hot run time: 60776 ms

@xiaokang xiaokang added usercase Important user case type label dev/2.0.4 labels Dec 14, 2023
@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.99 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17222374405 Bytes

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.5 seconds
stream load tsv: 587 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17224021727 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit d72c585487a32941bc8b8f216447b2db232d7e89, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4698	4433	4440	4433
q2	368	165	159	159
q3	1467	1269	1238	1238
q4	1109	914	864	864
q5	3173	3172	3243	3172
q6	246	126	131	126
q7	1023	490	484	484
q8	2205	2242	2214	2214
q9	6686	6704	6659	6659
q10	3235	3298	3287	3287
q11	327	198	189	189
q12	347	207	207	207
q13	4577	3826	3793	3793
q14	244	213	217	213
q15	576	525	533	525
q16	442	384	386	384
q17	1029	669	587	587
q18	7063	6807	6912	6807
q19	1541	1464	1340	1340
q20	512	317	326	317
q21	3052	2620	2679	2620
q22	346	279	282	279
Total cold run time: 44266 ms
Total hot run time: 39897 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4452	4335	4383	4335
q2	263	159	174	159
q3	3509	3491	3487	3487
q4	2382	2381	2373	2373
q5	5726	5736	5781	5736
q6	236	119	120	119
q7	2351	1853	1831	1831
q8	3515	3523	3535	3523
q9	9036	8979	8981	8979
q10	3948	3997	4011	3997
q11	509	388	374	374
q12	756	589	606	589
q13	4271	3571	3578	3571
q14	285	249	265	249
q15	569	525	525	525
q16	505	451	462	451
q17	1876	1853	1845	1845
q18	8480	8019	8099	8019
q19	1761	1707	1745	1707
q20	2247	1974	1924	1924
q21	6440	6087	6114	6087
q22	498	415	410	410
Total cold run time: 63615 ms
Total hot run time: 60290 ms

morningman
morningman approved these changes Dec 17, 2023
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 17, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

kaka11chen
kaka11chen approved these changes Dec 17, 2023
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

xiaokang
xiaokang approved these changes Dec 17, 2023
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit 9b3d4bb into apache:master Dec 17, 2023
xiaokang pushed a commit that referenced this pull request Dec 17, 2023
@BePPPower
(fix) Fix an export error when lower case table names=1 #28389 (#28388
d956f69 
)
hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Dec 28, 2023
@BePPPower
[fix](Export) Fix an export error when lower_case_table_names=1 (apac…
597a723 
…he#28389)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@BePPPower @HappenLee
[fix](Export) Fix an export error when lower_case_table_names=1 (apac…
4bdc40b 
…he#28389)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang xiaokang approved these changes

@kaka11chen kaka11chen kaka11chen approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.4-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@BePPPower @hello-stephen @doris-robot @xiaokang @kaka11chen @morningman