Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pipelineX](fix) Fix pip scanner context bug #29229

Merged
merged 4 commits into from
Dec 29, 2023

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Dec 28, 2023

Proposed changes

*** Current BE git commitID: d96278a ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 2068650 (TID 2069026 OR 0x7f757e3c7700) from PID 0; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/zfr/doris/be/src/common/signal_handler.h:417
1# 0x00007F76193FF2B7 in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
3# 0x00007F76193F80AC in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
4# 0x00007F761DA820C0 in /lib/x86_64-linux-gnu/libc.so.6
5# doris::vectorized::Block::swap(doris::vectorized::Block&) at /root/zfr/doris/be/src/vec/core/block.cpp:711
6# doris::pipeline::ScanOperatorXdoris::pipeline::OlapScanLocalState::get_block(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState&) at /root/zfr/doris/be/src/pipeline/exec/scan_operator.cpp:1507
7# doris::pipeline::ScanOperatorXdoris::pipeline::OlapScanLocalState::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState&) at /root/zfr/doris/be/src/pipeline/exec/scan_operator.h:432
8# doris::pipeline::PipelineXTask::execute(bool*) at /root/zfr/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:275
9# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /root/zfr/doris/be/src/pipeline/task_scheduler.cpp:284
10# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/STRESS_ENV/be/lib/doris_be
11# doris::Thread::supervise_thread(void*) at /root/zfr/doris/be/src/util/thread.cpp:499
12# start_thread at /build/glibc-sMfBJT/glibc-2.31/nptl/pthread_create.c:478
13# __clone in /lib/x86_64-linux-gnu/libc.so.6

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

ScannerContext(RuntimeState* state_, const TupleDescriptor* output_tuple_desc,
const RowDescriptor* output_row_descriptor,
const std::list<VScannerSPtr>& scanners_, int64_t limit_,
int64_t max_bytes_in_blocks_queue_, const int num_parallel_instances,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: parameter 'num_parallel_instances' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change
int64_t max_bytes_in_blocks_queue_, const int num_parallel_instances,
int64_t max_bytes_in_blocks_queue_, int num_parallel_instances,

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.61% (8589/23460)
Line Coverage: 28.69% (69874/243562)
Region Coverage: 27.68% (36163/130629)
Branch Coverage: 24.39% (18472/75746)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a61357619ad5c5e5cb7a195c8ec6026b1a9d2270_a61357619ad5c5e5cb7a195c8ec6026b1a9d2270/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit a61357619ad5c5e5cb7a195c8ec6026b1a9d2270, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4987	4649	4631	4631
q2	389	163	158	158
q3	1462	1372	1282	1282
q4	1145	956	927	927
q5	3190	3154	3169	3154
q6	253	128	132	128
q7	1053	511	529	511
q8	2253	2283	2259	2259
q9	6730	6696	6709	6696
q10	3201	3275	3288	3275
q11	321	213	214	213
q12	360	217	212	212
q13	4172	3414	3420	3414
q14	243	213	216	213
q15	573	520	532	520
q16	450	392	380	380
q17	1045	807	617	617
q18	7040	6865	6764	6764
q19	1633	1638	1655	1638
q20	616	327	297	297
q21	3181	2744	2722	2722
q22	373	314	310	310
Total cold run time: 44670 ms
Total hot run time: 40321 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4578	4585	4551	4551
q2	276	170	170	170
q3	3378	3370	3365	3365
q4	2233	2217	2203	2203
q5	5741	5711	5724	5711
q6	238	121	120	120
q7	2374	1837	1827	1827
q8	3636	3614	3626	3614
q9	9004	8954	8918	8918
q10	3814	3920	3922	3920
q11	477	380	389	380
q12	774	604	596	596
q13	3887	3187	3208	3187
q14	288	248	256	248
q15	581	522	529	522
q16	514	466	456	456
q17	1987	1959	1954	1954
q18	8747	8261	8316	8261
q19	1752	1751	1762	1751
q20	2244	1942	1933	1933
q21	6162	5787	5825	5787
q22	560	466	452	452
Total cold run time: 63245 ms
Total hot run time: 59926 ms

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.58% (8590/23480)
Line Coverage: 28.66% (69878/243808)
Region Coverage: 27.66% (36162/130731)
Branch Coverage: 24.37% (18479/75812)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b4c4fbc860470d118f8e1e7a9be65dca3276380c_b4c4fbc860470d118f8e1e7a9be65dca3276380c/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.88 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17183539307 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit b4c4fbc860470d118f8e1e7a9be65dca3276380c, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4970	4647	4620	4620
q2	358	157	159	157
q3	1461	1257	1206	1206
q4	1135	952	918	918
q5	3171	3167	3174	3167
q6	251	127	126	126
q7	1047	498	505	498
q8	2304	2256	2249	2249
q9	6691	6716	6695	6695
q10	3204	3257	3259	3257
q11	329	205	204	204
q12	349	205	210	205
q13	4158	3426	3409	3409
q14	246	217	212	212
q15	582	521	520	520
q16	443	379	385	379
q17	1055	733	526	526
q18	7045	6857	7841	6857
q19	1614	1624	1640	1624
q20	566	304	305	304
q21	3160	2722	2703	2703
q22	371	302	320	302
Total cold run time: 44510 ms
Total hot run time: 40138 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4589	4591	4540	4540
q2	270	164	171	164
q3	3369	3360	3359	3359
q4	2236	2210	2204	2204
q5	5727	5720	5709	5709
q6	240	118	119	118
q7	2382	1849	1887	1849
q8	3606	3623	3613	3613
q9	9055	8948	8987	8948
q10	3794	3869	3906	3869
q11	497	390	381	381
q12	776	596	587	587
q13	3889	3156	3180	3156
q14	286	252	254	252
q15	573	525	510	510
q16	525	450	460	450
q17	1971	1963	1945	1945
q18	8598	8251	8291	8251
q19	1753	1769	1763	1763
q20	2250	1940	1920	1920
q21	6196	5802	5796	5796
q22	557	459	458	458
Total cold run time: 63139 ms
Total hot run time: 59842 ms

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.57% (8587/23481)
Line Coverage: 28.65% (69850/243787)
Region Coverage: 27.65% (36142/130724)
Branch Coverage: 24.36% (18466/75804)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7d0db566dca9d00a3d55c6e3212e4e085c7ec5ec_7d0db566dca9d00a3d55c6e3212e4e085c7ec5ec/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.07 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17183748935 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 7d0db566dca9d00a3d55c6e3212e4e085c7ec5ec, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4996	4650	4679	4650
q2	382	171	160	160
q3	1471	1284	1217	1217
q4	1129	947	917	917
q5	3177	3151	3181	3151
q6	254	127	126	126
q7	1038	487	496	487
q8	2284	2268	2249	2249
q9	6713	6675	6717	6675
q10	3168	3257	3268	3257
q11	322	203	209	203
q12	347	206	203	203
q13	4186	3407	3422	3407
q14	238	216	217	216
q15	573	522	521	521
q16	440	384	382	382
q17	1037	762	566	566
q18	7133	6765	6982	6765
q19	1614	1629	1643	1629
q20	557	320	328	320
q21	3185	2703	2721	2703
q22	361	295	309	295
Total cold run time: 44605 ms
Total hot run time: 40099 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4573	4622	4549	4549
q2	267	165	171	165
q3	3384	3358	3359	3358
q4	2226	2210	2216	2210
q5	5721	5725	5728	5725
q6	241	121	118	118
q7	2367	1839	1846	1839
q8	3613	3621	3598	3598
q9	9053	8976	8941	8941
q10	3801	3889	3910	3889
q11	511	365	376	365
q12	764	599	616	599
q13	3905	3202	3213	3202
q14	288	252	248	248
q15	575	521	520	520
q16	492	441	458	441
q17	1976	1966	1967	1966
q18	8766	8066	8126	8066
q19	1755	1766	1755	1755
q20	2250	1940	1953	1940
q21	6134	5756	5739	5739
q22	527	467	447	447
Total cold run time: 63189 ms
Total hot run time: 59680 ms

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.58% (8590/23482)
Line Coverage: 28.65% (69871/243872)
Region Coverage: 27.65% (36163/130781)
Branch Coverage: 24.36% (18480/75854)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7d0db566dca9d00a3d55c6e3212e4e085c7ec5ec_7d0db566dca9d00a3d55c6e3212e4e085c7ec5ec/report/index.html

Status get_block_from_queue(RuntimeState* state, vectorized::BlockUPtr* block, bool* eos,
int id, bool wait = false) override {
{
std::unique_lock l(_transfer_lock);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line301 and line313 do not need lock 2 times.
merge these 2 code blocks

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 7d0db566dca9d00a3d55c6e3212e4e085c7ec5ec, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5465	5207	5158	5158
q2	386	168	187	168
q3	1489	1236	1187	1187
q4	1094	810	802	802
q5	3171	3162	3050	3050
q6	238	142	143	142
q7	985	557	542	542
q8	2172	2182	2225	2182
q9	6884	6857	6837	6837
q10	3188	3211	3165	3165
q11	357	238	229	229
q12	397	246	248	246
q13	4377	3611	3649	3611
q14	250	226	226	226
q15	626	558	575	558
q16	465	422	404	404
q17	1047	512	510	510
q18	7060	6802	6808	6802
q19	1670	1571	1530	1530
q20	582	347	760	347
q21	2926	2549	2533	2533
q22	393	333	333	333
Total cold run time: 45222 ms
Total hot run time: 40562 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5118	5082	5095	5082
q2	354	250	257	250
q3	3396	3310	3335	3310
q4	2174	2009	2018	2009
q5	5926	5907	5925	5907
q6	239	137	138	137
q7	2392	1923	1915	1915
q8	3582	3643	3648	3643
q9	9094	8975	9059	8975
q10	3914	3950	3931	3931
q11	596	482	500	482
q12	811	641	652	641
q13	3900	3171	3181	3171
q14	295	272	276	272
q15	627	568	554	554
q16	568	525	532	525
q17	2028	1791	1809	1791
q18	8739	8334	8386	8334
q19	1763	1713	1692	1692
q20	2296	2000	1994	1994
q21	5813	5442	5360	5360
q22	564	505	511	505
Total cold run time: 64189 ms
Total hot run time: 60480 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.45 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.8 seconds inserted 10000000 Rows, about 359K ops/s
storage size: 17189099129 Bytes

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.59% (8591/23482)
Line Coverage: 28.65% (69887/243903)
Region Coverage: 27.66% (36178/130794)
Branch Coverage: 24.37% (18491/75868)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9b7ace6efd32cb7183be0ef63822da0d2f966668_9b7ace6efd32cb7183be0ef63822da0d2f966668/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 9b7ace6efd32cb7183be0ef63822da0d2f966668, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5467	5116	5167	5116
q2	423	172	178	172
q3	1464	1214	1199	1199
q4	1092	813	800	800
q5	3138	3114	3117	3114
q6	236	140	136	136
q7	1008	577	523	523
q8	2157	2224	2232	2224
q9	6717	6666	6639	6639
q10	3218	3195	3161	3161
q11	345	235	234	234
q12	394	239	242	239
q13	4362	3642	3630	3630
q14	262	227	221	221
q15	628	559	556	556
q16	452	417	400	400
q17	1045	663	526	526
q18	7106	6756	6766	6756
q19	1646	1554	1576	1554
q20	614	354	363	354
q21	2852	2538	2510	2510
q22	404	333	327	327
Total cold run time: 45030 ms
Total hot run time: 40391 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5032	5073	5010	5010
q2	342	251	256	251
q3	3348	3321	3302	3302
q4	2151	2030	2032	2030
q5	5964	5927	5951	5927
q6	236	133	130	130
q7	2375	1913	1948	1913
q8	3567	3692	3684	3684
q9	9031	9000	9009	9000
q10	3867	3957	3956	3956
q11	587	499	474	474
q12	812	663	629	629
q13	3855	3188	3193	3188
q14	293	289	275	275
q15	614	564	570	564
q16	592	529	532	529
q17	2038	1830	1799	1799
q18	8708	8439	8456	8439
q19	1769	1715	1731	1715
q20	2281	2015	1991	1991
q21	5727	5214	5281	5214
q22	565	468	473	468
Total cold run time: 63754 ms
Total hot run time: 60488 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.41 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17184031975 Bytes

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 29, 2023
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 linked an issue Dec 29, 2023 that may be closed by this pull request
@Gabriel39 Gabriel39 removed a link to an issue Dec 29, 2023
@Gabriel39 Gabriel39 merged commit b9572f9 into apache:master Dec 29, 2023
25 of 28 checks passed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants