[Enhancement] Loading task should abort transaction if fail because of timeout #51939

banmoy · 2024-10-15T11:53:34Z

Why I'm doing:

In #38183, broker load will retry if loading task fails because of timeout. But currently the retry will be executed until transaction timeouts even if the loading task has failed before transaction timeout. For example, the task fails because BE BRPC timeouts which is half of transaction timeout by default, and the load should retry immediately after the task fails.

What I'm doing:

If the task fails because of timeout, abort the transaction so that it can try as soon as possible

Fixes #issue

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

github-actions · 2024-10-16T05:22:47Z

[BE Incremental Coverage Report]

❌ fail : 1 / 5 (20.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/runtime/local_tablets_channel.cpp	1	5	20.00%	[395, 396, 397, 398]

…timeout Signed-off-by: PengFei Li <[email protected]>

sonarcloud · 2024-10-16T11:15:59Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-10-16T12:38:16Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-10-16T12:41:09Z

[FE Incremental Coverage Report]

✅ pass : 24 / 25 (96.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/load/loadv2/LoadErrorUtils.java	10	11	90.91%	[17]
🔵	com/starrocks/load/loadv2/BrokerLoadJob.java	3	3	100.00%	[]
🔵	com/starrocks/load/loadv2/BulkLoadJob.java	11	11	100.00%	[]

wyb · 2024-10-17T02:24:24Z

fe/fe-core/src/main/java/com/starrocks/load/loadv2/LoadErrorUtils.java

+
+public class LoadErrorUtils {
+
+    public static class ErrorMeta {


Suggested change

public static class ErrorMeta {

private static class ErrorMeta {

github-actions · 2024-10-17T03:14:38Z

@Mergifyio backport branch-3.3

github-actions · 2024-10-17T03:14:39Z

@Mergifyio backport branch-3.2

github-actions · 2024-10-17T03:14:40Z

@Mergifyio backport branch-3.1

mergify · 2024-10-17T03:15:06Z

backport branch-3.3

✅ Backports have been created

#52010 [Enhancement] Loading task should abort transaction if fail because of timeout (backport #51939) has been created for branch branch-3.3

mergify · 2024-10-17T03:15:07Z

backport branch-3.2

✅ Backports have been created

#52012 [Enhancement] Loading task should abort transaction if fail because of timeout (backport #51939) has been created for branch branch-3.2

mergify · 2024-10-17T03:15:08Z

backport branch-3.1

✅ Backports have been created

#52011 [Enhancement] Loading task should abort transaction if fail because of timeout (backport #51939) has been created for branch branch-3.1 but encountered conflicts

…f timeout (#51939) Signed-off-by: PengFei Li <[email protected]> (cherry picked from commit 73172c3)

…f timeout (#51939) Signed-off-by: PengFei Li <[email protected]> (cherry picked from commit 73172c3) # Conflicts: # fe/fe-core/src/test/java/com/starrocks/load/loadv2/BrokerLoadJobTest.java

…f timeout (#51939) Signed-off-by: PengFei Li <[email protected]> (cherry picked from commit 73172c3)

…f timeout (StarRocks#51939) Signed-off-by: PengFei Li <[email protected]>

…f timeout (backport #51939) (#52010) Co-authored-by: PengFei Li <[email protected]>

…f timeout (backport #51939) (#52012) Co-authored-by: PengFei Li <[email protected]>

…f timeout (backport #51939) (#52054) Signed-off-by: PengFei Li <[email protected]>

…f timeout (StarRocks#51939) Signed-off-by: PengFei Li <[email protected]>

…f timeout (StarRocks#51939) Signed-off-by: PengFei Li <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>

banmoy requested review from a team as code owners October 15, 2024 11:53

mergify bot assigned banmoy Oct 15, 2024

banmoy force-pushed the broker_load branch from 7a79429 to 415c479 Compare October 15, 2024 12:24

andyziye requested review from a team and removed request for a team October 15, 2024 12:31

banmoy force-pushed the broker_load branch from 415c479 to 10f6bae Compare October 16, 2024 01:38

banmoy force-pushed the broker_load branch from 178dd1c to e9cf8a3 Compare October 16, 2024 09:32

banmoy changed the title ~~[Enhancement] Broker load task should abort transaction if fail with timeout~~ [Enhancement] Loading task should abort transaction if fail because of timeout Oct 16, 2024

github-actions bot added 3.3 3.2 3.1 labels Oct 16, 2024

[Enhancement] Broker load task should abort transaction if fail with …

f544e0e

…timeout Signed-off-by: PengFei Li <[email protected]>

banmoy force-pushed the broker_load branch from e9cf8a3 to f544e0e Compare October 16, 2024 11:09

allencelerdata approved these changes Oct 17, 2024

View reviewed changes

chaoyli approved these changes Oct 17, 2024

View reviewed changes

wyb approved these changes Oct 17, 2024

View reviewed changes

wyb merged commit 73172c3 into StarRocks:main Oct 17, 2024
58 checks passed

github-actions bot removed the 3.3 label Oct 17, 2024

github-actions bot removed 3.2 3.1 labels Oct 17, 2024

mergify bot pushed a commit that referenced this pull request Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because o…

97852d1

…f timeout (#51939) Signed-off-by: PengFei Li <[email protected]> (cherry picked from commit 73172c3)

mergify bot mentioned this pull request Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because of timeout (backport #51939) #52010

Merged

42 tasks

mergify bot pushed a commit that referenced this pull request Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because o…

9b5ef5c

…f timeout (#51939) Signed-off-by: PengFei Li <[email protected]> (cherry picked from commit 73172c3)

This was referenced Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because of timeout (backport #51939) #52011

Closed

[Enhancement] Loading task should abort transaction if fail because of timeout (backport #51939) #52012

Merged

banmoy added a commit to banmoy/starrocks that referenced this pull request Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because o…

7957b18

…f timeout (StarRocks#51939) Signed-off-by: PengFei Li <[email protected]>

banmoy mentioned this pull request Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because of timeout (backport #51939) #52054

Merged

24 tasks

wanpengfei-git pushed a commit that referenced this pull request Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because o…

d9b650c

…f timeout (backport #51939) (#52010) Co-authored-by: PengFei Li <[email protected]>

github-actions bot added the 3.3-merged label Oct 17, 2024

wanpengfei-git pushed a commit that referenced this pull request Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because o…

dec2775

…f timeout (backport #51939) (#52012) Co-authored-by: PengFei Li <[email protected]>

github-actions bot added the 3.2-merged label Oct 17, 2024

wyb pushed a commit that referenced this pull request Oct 17, 2024

[Enhancement] Loading task should abort transaction if fail because o…

3d50e33

…f timeout (backport #51939) (#52054) Signed-off-by: PengFei Li <[email protected]>

github-actions bot added the 3.1-merged label Oct 17, 2024

dirtysalt pushed a commit to dirtysalt/starrocks that referenced this pull request Oct 18, 2024

[Enhancement] Loading task should abort transaction if fail because o…

dfd2813

…f timeout (StarRocks#51939) Signed-off-by: PengFei Li <[email protected]>

ZiheLiu pushed a commit to ZiheLiu/starrocks that referenced this pull request Oct 31, 2024

[Enhancement] Loading task should abort transaction if fail because o…

f375089

…f timeout (StarRocks#51939) Signed-off-by: PengFei Li <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Loading task should abort transaction if fail because of timeout #51939

[Enhancement] Loading task should abort transaction if fail because of timeout #51939

banmoy commented Oct 15, 2024 •

edited

Loading

github-actions bot commented Oct 16, 2024

sonarcloud bot commented Oct 16, 2024

github-actions bot commented Oct 16, 2024

github-actions bot commented Oct 16, 2024

wyb Oct 17, 2024

github-actions bot commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

mergify bot commented Oct 17, 2024 •

edited

Loading

mergify bot commented Oct 17, 2024 •

edited

Loading

mergify bot commented Oct 17, 2024 •

edited

Loading


		public class LoadErrorUtils {

		public static class ErrorMeta {

	public static class ErrorMeta {
	private static class ErrorMeta {

[Enhancement] Loading task should abort transaction if fail because of timeout #51939

[Enhancement] Loading task should abort transaction if fail because of timeout #51939

Conversation

banmoy commented Oct 15, 2024 • edited Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

github-actions bot commented Oct 16, 2024

[BE Incremental Coverage Report]

file detail

sonarcloud bot commented Oct 16, 2024

Quality Gate passed

github-actions bot commented Oct 16, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Oct 16, 2024

[FE Incremental Coverage Report]

file detail

wyb Oct 17, 2024

Choose a reason for hiding this comment

github-actions bot commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

github-actions bot commented Oct 17, 2024

mergify bot commented Oct 17, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Oct 17, 2024 • edited Loading

✅ Backports have been created

mergify bot commented Oct 17, 2024 • edited Loading

✅ Backports have been created

banmoy commented Oct 15, 2024 •

edited

Loading

mergify bot commented Oct 17, 2024 •

edited

Loading

mergify bot commented Oct 17, 2024 •

edited

Loading

mergify bot commented Oct 17, 2024 •

edited

Loading