HBASE-27490 Locating regions for all actions of batch requests can exceed operation timeout #4908

bbeaudreault · 2022-12-01T19:42:52Z

This is a continuation of HBASE-27487. That jira protects against the case where operation timeout is exceeded just before the call is executed. This jira further ensures that the operation timeout cannot be exceeded while locating regions for the requested actions. In both cases, we should not clear meta cache.

The two are similar, but need to be handled separately due to the complicated call stack in AsyncProcess. I've added a new test so that we have test cases covering both situations.

sunhelly

Failed UTs seems related.

java.lang.NullPointerException
at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:423)
at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:379)
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:213)
at org.apache.hadoop.hbase.client.HTable.doCheckAndMutate(HTable.java:853)

bbeaudreault · 2022-12-02T13:39:52Z

@sunhelly thanks for taking a look! I just pushed a fix. I checked a few of the classes with failing tests, and it fixed them all.

bbeaudreault · 2022-12-02T17:20:30Z

@sunhelly pre-commit checks are all passing now

…ceed operation timeout

Apache-HBase · 2022-12-05T22:05:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 36s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 26s	Maven dependency ordering for branch
+1 💚	mvninstall	4m 21s	branch-2 passed
+1 💚	compile	3m 55s	branch-2 passed
+1 💚	checkstyle	0m 56s	branch-2 passed
+1 💚	spotless	0m 50s	branch has no errors when running spotless:check.
+1 💚	spotbugs	2m 54s	branch-2 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	4m 6s	the patch passed
+1 💚	compile	4m 20s	the patch passed
+1 💚	javac	4m 20s	the patch passed
+1 💚	checkstyle	1m 16s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	17m 48s	Patch does not cause any errors with Hadoop 2.10.2 or 3.2.4 3.3.4.
+1 💚	spotless	1m 3s	patch has no errors when running spotless:check.
+1 💚	spotbugs	3m 33s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 22s	The patch does not generate ASF License warnings.
		49m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4908/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4908
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux fe1fd4554b8b 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `8c4f0e3`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4908/4/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-12-06T00:50:39Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 54s	Docker mode activated.
-0 ⚠️	yetus	0m 6s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 18s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 27s	branch-2 passed
+1 💚	compile	0m 51s	branch-2 passed
+1 💚	shadedjars	3m 54s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	branch-2 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 2s	the patch passed
+1 💚	compile	0m 51s	the patch passed
+1 💚	javac	0m 51s	the patch passed
+1 💚	shadedjars	3m 54s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 33s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 32s	hbase-client in the patch passed.
+1 💚	unit	191m 28s	hbase-server in the patch passed.
		215m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4908/4/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR	#4908
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 661f71747b25 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `8c4f0e3`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4908/4/testReport/
Max. process+thread count	2299 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4908/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-12-06T00:51:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 54s	Docker mode activated.
-0 ⚠️	yetus	0m 5s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ branch-2 Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 46s	branch-2 passed
+1 💚	compile	1m 3s	branch-2 passed
+1 💚	shadedjars	4m 9s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 38s	branch-2 passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 15s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 27s	the patch passed
+1 💚	compile	0m 58s	the patch passed
+1 💚	javac	0m 58s	the patch passed
+1 💚	shadedjars	4m 3s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	the patch passed
		_ Other Tests _
+1 💚	unit	2m 41s	hbase-client in the patch passed.
+1 💚	unit	191m 17s	hbase-server in the patch passed.
		216m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4908/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4908
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 1cccce641f0e 5.4.0-1085-aws #92~18.04.1-Ubuntu SMP Wed Aug 31 17:21:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	branch-2 / `8c4f0e3`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4908/4/testReport/
Max. process+thread count	3054 (vs. ulimit of 30000)
modules	C: hbase-client hbase-server U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4908/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault · 2022-12-06T01:20:58Z

@sunhelly Any chance you can give this one more look?

sunhelly

+1

bbeaudreault · 2022-12-06T13:02:51Z

Thanks!

…ceed operation timeout (#4908) Signed-off-by: Xiaolin Ha <[email protected]>

…ch requests can exceed operation timeout (apache#4908) Conficts: TestClientOperationTimeout Signed-off-by: Xiaolin Ha <[email protected]>

…s of batch requests can exceed operation timeout (apache#4908)" This reverts commit 55fae15.

bbeaudreault requested a review from Apache9 December 1, 2022 19:42

This comment was marked as outdated.

Sign in to view

sunhelly approved these changes Dec 2, 2022

View reviewed changes

sunhelly requested changes Dec 2, 2022

View reviewed changes

This comment was marked as outdated.

Sign in to view

bbeaudreault removed the request for review from Apache9 December 2, 2022 16:21

This comment was marked as outdated.

Sign in to view

bbeaudreault requested a review from sunhelly December 2, 2022 17:20

bbeaudreault added 2 commits December 5, 2022 16:04

HBASE-27490 Locating regions for all actions of batch requests can ex…

7768edd

…ceed operation timeout

Fix case where callable comes with its own RetryingTimeTracker

fe0e41f

bbeaudreault force-pushed the HBASE-27490 branch from 7ba30c7 to fe0e41f Compare December 5, 2022 21:05

sunhelly approved these changes Dec 6, 2022

View reviewed changes

bbeaudreault merged commit f320ad9 into apache:branch-2 Dec 6, 2022

bbeaudreault deleted the HBASE-27490 branch December 6, 2022 13:03

bbeaudreault added a commit that referenced this pull request Dec 6, 2022

HBASE-27490 Locating regions for all actions of batch requests can ex…

ed8549b

…ceed operation timeout (#4908) Signed-off-by: Xiaolin Ha <[email protected]>

bbeaudreault added a commit to HubSpot/hbase that referenced this pull request Dec 9, 2022

Revert "HubSpot Backport: HBASE-27490 Locating regions for all action…

a1fe7a2

…s of batch requests can exceed operation timeout (apache#4908)" This reverts commit 55fae15.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27490 Locating regions for all actions of batch requests can exceed operation timeout #4908

HBASE-27490 Locating regions for all actions of batch requests can exceed operation timeout #4908

bbeaudreault commented Dec 1, 2022

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

sunhelly left a comment

bbeaudreault commented Dec 2, 2022

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

bbeaudreault commented Dec 2, 2022

Apache-HBase commented Dec 5, 2022

Apache-HBase commented Dec 6, 2022

Apache-HBase commented Dec 6, 2022

bbeaudreault commented Dec 6, 2022

sunhelly left a comment

bbeaudreault commented Dec 6, 2022

HBASE-27490 Locating regions for all actions of batch requests can exceed operation timeout #4908

HBASE-27490 Locating regions for all actions of batch requests can exceed operation timeout #4908

Conversation

bbeaudreault commented Dec 1, 2022

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

sunhelly left a comment

Choose a reason for hiding this comment

bbeaudreault commented Dec 2, 2022

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

bbeaudreault commented Dec 2, 2022

Apache-HBase commented Dec 5, 2022

Apache-HBase commented Dec 6, 2022

Apache-HBase commented Dec 6, 2022

bbeaudreault commented Dec 6, 2022

sunhelly left a comment

Choose a reason for hiding this comment

bbeaudreault commented Dec 6, 2022