Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](Nereids) support reuse sql cache between different comment #40049

Merged
merged 2 commits into from
Aug 28, 2024
Merged

[enhancement](Nereids) support reuse sql cache between different comment #40049

merged 2 commits into from
Aug 28, 2024

Conversation

924060929
Copy link
Contributor

Proposed changes

support reuse sql cache between different comment like this:

SQL 1:

select /* trace_id: 100001 */ from tbl;

SQL 2:

select /* trace_id: 100002 */ from tbl;

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@924060929
Copy link
Contributor Author

run buildall

@924060929
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 38027 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ced14448e64f750045d73759c5b24668e0433c10, data reload: false

------ Round 1 ----------------------------------
q1	17702	4433	4301	4301
q2	2012	201	180	180
q3	11849	981	1096	981
q4	10513	665	786	665
q5	7764	2836	2824	2824
q6	230	143	142	142
q7	970	615	620	615
q8	9559	2054	2086	2054
q9	7392	6546	6545	6545
q10	6985	2231	2200	2200
q11	438	240	245	240
q12	396	221	229	221
q13	17749	3030	3022	3022
q14	289	237	242	237
q15	518	475	492	475
q16	582	500	509	500
q17	979	667	788	667
q18	7300	6780	6711	6711
q19	1393	1055	1140	1055
q20	660	340	334	334
q21	4204	3269	3056	3056
q22	1116	1002	1002	1002
Total cold run time: 110600 ms
Total hot run time: 38027 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4390	4348	4355	4348
q2	374	292	281	281
q3	2919	2648	2665	2648
q4	1954	1644	1677	1644
q5	5635	5654	5726	5654
q6	226	144	141	141
q7	2230	1839	1818	1818
q8	3319	3463	3449	3449
q9	8708	8828	8777	8777
q10	3550	3325	3344	3325
q11	631	520	511	511
q12	827	684	645	645
q13	12018	3162	3250	3162
q14	331	293	295	293
q15	555	497	476	476
q16	629	566	560	560
q17	1839	1554	1547	1547
q18	8087	7713	7748	7713
q19	1748	1468	1515	1468
q20	2145	1908	1894	1894
q21	5835	5543	5262	5262
q22	1167	1038	1021	1021
Total cold run time: 69117 ms
Total hot run time: 56637 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193321 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ced14448e64f750045d73759c5b24668e0433c10, data reload: false

query1	1281	893	889	889
query2	6282	1894	1893	1893
query3	10604	4060	3945	3945
query4	59663	25907	23248	23248
query5	5481	517	497	497
query6	423	169	171	169
query7	5816	302	297	297
query8	292	212	197	197
query9	8961	2521	2510	2510
query10	487	279	260	260
query11	16109	15148	15148	15148
query12	171	106	108	106
query13	1619	383	391	383
query14	10964	7434	7385	7385
query15	221	185	183	183
query16	7485	467	470	467
query17	1168	593	577	577
query18	2081	311	315	311
query19	301	147	148	147
query20	118	112	127	112
query21	205	100	110	100
query22	4674	4507	4446	4446
query23	34129	33659	33785	33659
query24	5901	2879	2838	2838
query25	543	375	390	375
query26	691	158	158	158
query27	1784	282	278	278
query28	3805	2120	2106	2106
query29	676	403	410	403
query30	238	154	159	154
query31	928	756	766	756
query32	84	54	57	54
query33	450	296	296	296
query34	863	495	483	483
query35	832	729	742	729
query36	1078	939	942	939
query37	144	95	92	92
query38	3922	3825	3962	3825
query39	1473	1375	1405	1375
query40	201	119	115	115
query41	48	46	45	45
query42	121	98	98	98
query43	518	466	471	466
query44	1074	747	735	735
query45	197	165	163	163
query46	1118	760	776	760
query47	1917	1802	1814	1802
query48	399	298	287	287
query49	751	423	433	423
query50	814	403	420	403
query51	7069	7030	7018	7018
query52	99	94	91	91
query53	256	180	178	178
query54	565	448	445	445
query55	80	77	76	76
query56	282	261	255	255
query57	1171	1064	1034	1034
query58	225	254	227	227
query59	3091	3012	2832	2832
query60	288	276	260	260
query61	106	95	98	95
query62	718	649	653	649
query63	214	186	188	186
query64	2783	663	694	663
query65	3220	3137	3186	3137
query66	624	342	348	342
query67	15570	15316	15158	15158
query68	2979	576	583	576
query69	389	280	290	280
query70	1156	1139	1041	1041
query71	348	274	271	271
query72	5072	4000	3948	3948
query73	758	333	336	333
query74	9123	8781	8806	8781
query75	3406	2626	2698	2626
query76	1339	1008	916	916
query77	540	321	349	321
query78	9569	9245	9149	9149
query79	1039	549	532	532
query80	670	491	501	491
query81	470	246	243	243
query82	248	149	147	147
query83	178	155	208	155
query84	254	76	76	76
query85	682	292	284	284
query86	305	301	299	299
query87	4445	4243	4244	4243
query88	2919	2371	2355	2355
query89	376	286	285	285
query90	2004	197	196	196
query91	121	99	97	97
query92	64	50	58	50
query93	1033	529	548	529
query94	658	291	282	282
query95	311	266	263	263
query96	594	275	273	273
query97	3236	3088	3073	3073
query98	222	217	210	210
query99	1500	1263	1347	1263
Total cold run time: 301474 ms
Total hot run time: 193321 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ced14448e64f750045d73759c5b24668e0433c10, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.08
query5	0.48	0.51	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.54	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.62	0.58	0.58
query14	2.02	2.07	2.10
query15	0.85	0.81	0.82
query16	0.38	0.39	0.36
query17	1.00	1.05	1.04
query18	0.22	0.21	0.20
query19	1.90	1.72	1.72
query20	0.01	0.01	0.01
query21	15.40	0.67	0.67
query22	4.40	7.03	1.93
query23	18.25	1.38	1.30
query24	2.07	0.25	0.22
query25	0.15	0.09	0.08
query26	0.27	0.18	0.17
query27	0.07	0.08	0.08
query28	13.26	1.01	1.00
query29	12.67	3.36	3.40
query30	0.24	0.05	0.05
query31	2.90	0.40	0.41
query32	3.24	0.48	0.47
query33	2.96	3.02	3.00
query34	17.11	4.39	4.40
query35	4.43	4.50	4.50
query36	0.66	0.48	0.48
query37	0.19	0.16	0.16
query38	0.16	0.14	0.14
query39	0.05	0.03	0.03
query40	0.16	0.13	0.12
query41	0.10	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 110.96 s
Total hot run time: 32.22 s

@starocean999 starocean999 merged commit 7a2b214 into apache:master Aug 28, 2024
27 of 30 checks passed
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Aug 28, 2024
@924060929
[enhancement](Nereids) support reuse sql cache between different comm…
8c22135 
…ent (apache#40049)

## Proposed changes

support reuse sql cache between different comment like this:

SQL 1:
```sql
select /* trace_id: 100001 */ from tbl;
```

SQL 2:
```sql
select /* trace_id: 100002 */ from tbl;
```

(cherry picked from commit 7a2b214)
@924060929 924060929 mentioned this pull request Aug 28, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 28, 2024
@924060929
[enhancement](Nereids) support reuse sql cache between different comm…
688b97e 
…ent (#40065)

cherry pick from #40049
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@924060929 @dataroaring
[enhancement](Nereids) support reuse sql cache between different comm…
e33d48b 
…ent (#40049)

## Proposed changes

support reuse sql cache between different comment like this:

SQL 1:
```sql
select /* trace_id: 100001 */ from tbl;
```

SQL 2:
```sql
select /* trace_id: 100002 */ from tbl;
```
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@924060929 @doris-robot @starocean999 @morrySnow @yiguolei @dataroaring