Batching for upload test results #12838

naspirato · 2024-12-20T12:49:57Z

Changelog entry

In asan, msan and tsan test runs faced with issue

Asan :

	debug_error_string = "UNKNOWN:Error received from peer  ***created_time:"2024-12-20T07:00:24.812682876+00:00", grpc_status:8, 
grpc_message:"CLIENT: Sent message larger than max (68044368 vs. 64000000)"***"
>
+ result='upserting runs: 31664 rows

=> x1,06 size 148394093 vs. 64000000

Msan (very large error messages)

	debug_error_string = "UNKNOWN:Error received from peer  ***created_time:"2024-12-20T04:45:30.016096641+00:00", grpc_status:8, 
grpc_message:"CLIENT: Sent message larger than max (148394093 vs. 64000000)"***"
>
+ result='upserting runs: 10183 rows

=> x2,3 size 148394093 vs. 64000000

So 1000 rows shoud be about 20% of maximum

testing in msan here #12843

Changelog category

Not for changelog (changelog entry is not required)

Additional information

...

github-actions · 2024-12-20T12:51:59Z

⚪ 2024-12-20 12:51:58 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9cd8f17 has started.
⚪ 2024-12-20 12:52:10 UTC Artifacts will be uploaded here
⚪ 2024-12-20 12:55:10 UTC ya make is running...
🟡 2024-12-20 14:16:04 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19028	17497	0	2	1415	114

⚪ 2024-12-20 14:18:18 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-20 14:30:13 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
159 (only retried tests)	49	0	0	0	110

🟢 2024-12-20 14:30:19 UTC Build successful.
🟢 2024-12-20 14:30:37 UTC ydbd size 2.1 GiB changed* by +68.0 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `d30aab1`	merge: `9cd8f17`	diff	diff %
ydbd size	2 235 463 720 Bytes	2 235 533 360 Bytes	+68.0 KiB	+0.003%
ydbd stripped size	477 915 408 Bytes	477 922 896 Bytes	+7.3 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-20T12:52:04Z

⚪ 2024-12-20 12:52:03 UTC Pre-commit check linux-x86_64-release-asan for 9cd8f17 has started.
⚪ 2024-12-20 12:52:15 UTC Artifacts will be uploaded here
⚪ 2024-12-20 12:55:10 UTC ya make is running...
🟡 2024-12-20 14:26:26 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12055	11980	0	22	9	44

🟢 2024-12-20 14:27:39 UTC Build successful.
🟡 2024-12-20 14:28:01 UTC ydbd size 3.6 GiB changed* by +109.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `d30aab1`	merge: `9cd8f17`	diff	diff %
ydbd size	3 866 120 024 Bytes	3 866 231 832 Bytes	+109.2 KiB	+0.003%
ydbd stripped size	1 350 476 240 Bytes	1 350 507 536 Bytes	+30.6 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

maximyurchuk · 2024-12-20T14:44:04Z

.github/scripts/analytics/upload_tests_results.py

+            for start in range(0, len(prepared_for_upload_rows), batch_rows_for_upload_size):
+                batch_rows_for_upload = prepared_for_upload_rows[start:start + batch_rows_for_upload_size]
+                with ydb.SessionPool(driver) as pool:
+                    create_tables(pool, test_table_name)


Вынести вне цикла наверное стоит

github-actions · 2024-12-23T09:39:29Z

⚪ 2024-12-23 09:39:28 UTC Pre-commit check linux-x86_64-release-asan for fe432c3 has started.
⚪ 2024-12-23 09:39:39 UTC Artifacts will be uploaded here
⚪ 2024-12-23 09:42:28 UTC ya make is running...
🟡 2024-12-23 10:46:30 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
12058	11988	0	20	6	44

🟢 2024-12-23 10:47:33 UTC Build successful.
🟡 2024-12-23 10:47:54 UTC ydbd size 3.6 GiB changed* by +513.7 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ac47d76`	merge: `fe432c3`	diff	diff %
ydbd size	3 864 972 872 Bytes	3 865 498 912 Bytes	+513.7 KiB	+0.014%
ydbd stripped size	1 350 438 672 Bytes	1 350 613 584 Bytes	+170.8 KiB	+0.013%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-23T09:39:41Z

⚪ 2024-12-23 09:39:41 UTC Pre-commit check linux-x86_64-relwithdebinfo for fe432c3 has started.
⚪ 2024-12-23 09:39:52 UTC Artifacts will be uploaded here
⚪ 2024-12-23 09:42:48 UTC ya make is running...
🟡 2024-12-23 10:34:47 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
19032	17499	0	4	1416	113

⚪ 2024-12-23 10:37:25 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-23 10:49:10 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
164 (only retried tests)	54	0	0	1	109

🟢 2024-12-23 10:49:18 UTC Build successful.
🟡 2024-12-23 10:49:36 UTC ydbd size 2.1 GiB changed* by +231.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `ac47d76`	merge: `fe432c3`	diff	diff %
ydbd size	2 234 720 224 Bytes	2 234 957 304 Bytes	+231.5 KiB	+0.011%
ydbd stripped size	477 909 392 Bytes	477 966 224 Bytes	+55.5 KiB	+0.012%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-23T11:12:06Z

⚪ 2024-12-23 11:12:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for 88cf196 has started.
⚪ 2024-12-23 11:12:17 UTC Artifacts will be uploaded here
⚪ 2024-12-23 11:14:41 UTC ya make is running...
🟢 2024-12-23 11:14:46 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
0	0	0	0	0	0

🟢 2024-12-23 11:14:52 UTC Build successful.

github-actions · 2024-12-23T11:12:07Z

⚪ 2024-12-23 11:12:06 UTC Pre-commit check linux-x86_64-release-asan for 88cf196 has started.
⚪ 2024-12-23 11:12:19 UTC Artifacts will be uploaded here
⚪ 2024-12-23 11:14:48 UTC ya make is running...
🟢 2024-12-23 11:14:54 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
0	0	0	0	0	0

🟢 2024-12-23 11:15:00 UTC Build successful.

naspirato added 2 commits December 20, 2024 12:31

init

a0b9253

for rollback

66c8c66

github-actions bot added not-for-changelog and removed not-for-changelog labels Dec 20, 2024

naspirato marked this pull request as ready for review December 20, 2024 13:19

naspirato requested a review from a team as a code owner December 20, 2024 13:19

github-actions bot added not-for-changelog and removed not-for-changelog labels Dec 20, 2024

maximyurchuk reviewed Dec 20, 2024

View reviewed changes

create table out of for

270275c

rollback

345c58f

naspirato enabled auto-merge (squash) December 23, 2024 17:47

naspirato requested a review from maximyurchuk December 23, 2024 17:47

maximyurchuk approved these changes Dec 23, 2024

View reviewed changes

naspirato merged commit d0bcede into ydb-platform:main Dec 23, 2024
19 of 22 checks passed

This was referenced Dec 25, 2024

Fixed empty error from CMS create tenant request #12977

Merged

YQ-3949 fix pq source sensors names #13088

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batching for upload test results #12838

Batching for upload test results #12838

naspirato commented Dec 20, 2024 •

edited

Loading

github-actions bot commented Dec 20, 2024 •

edited

Loading

github-actions bot commented Dec 20, 2024 •

edited

Loading

maximyurchuk Dec 20, 2024

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

Batching for upload test results #12838

Batching for upload test results #12838

Conversation

naspirato commented Dec 20, 2024 • edited Loading

Changelog entry

Changelog category

Additional information

github-actions bot commented Dec 20, 2024 • edited Loading

github-actions bot commented Dec 20, 2024 • edited Loading

maximyurchuk Dec 20, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 23, 2024 • edited Loading

github-actions bot commented Dec 23, 2024 • edited Loading

github-actions bot commented Dec 23, 2024 • edited Loading

github-actions bot commented Dec 23, 2024 • edited Loading

naspirato commented Dec 20, 2024 •

edited

Loading

github-actions bot commented Dec 20, 2024 •

edited

Loading

github-actions bot commented Dec 20, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading

github-actions bot commented Dec 23, 2024 •

edited

Loading