HBASE-27676 Scan handlers in the RPC executor should match at least o… #5074

sunhelly · 2023-03-02T11:25:27Z

…ne scan queues

This issue is try to avoid NO scan queues for some scan handlers.
For example, if we set hbase.regionserver.handler.count=150, hbase.ipc.server.callqueue.scan.ratio=0.1, hbase.ipc.server.callqueue.read.ratio=0.5, hbase.ipc.server.callqueue.handler.factor=0.1, then there will be 1500.50.1=7 scan handlers, but there are 1500.10.5*0.1=0 scan RPC queues.
When there are no scan rpc queues, all the scan and get requests will be dispatched to the read rpc queues, while we we thought they had been dealt with separately, since the scan handler count is not 0. When there are not enough handlers for large scan requests under this circumstance, the small get requests will be blocked in the rpc queues.

Apache-HBase · 2023-03-02T13:39:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 8s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 42s	master passed
+1 💚	compile	2m 22s	master passed
+1 💚	checkstyle	0m 36s	master passed
+1 💚	spotless	0m 43s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 16s	the patch passed
+1 💚	compile	2m 22s	the patch passed
+1 💚	javac	2m 22s	the patch passed
+1 💚	checkstyle	0m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 12s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 36s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 11s	The patch does not generate ASF License warnings.
		39m 15s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5074
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux d1b6c0262061 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d2b0074`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	85 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-03-02T14:40:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	4m 48s	master passed
+1 💚	compile	0m 55s	master passed
+1 💚	shadedjars	5m 42s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 30s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 15s	the patch passed
+1 💚	compile	0m 59s	the patch passed
+1 💚	javac	0m 59s	the patch passed
+1 💚	shadedjars	6m 30s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 35s	the patch passed
		_ Other Tests _
-1 ❌	unit	13m 52s	hbase-server in the patch failed.
		40m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5074
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 5828b5f34c0f 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d2b0074`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/testReport/
Max. process+thread count	593 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-03-02T15:27:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 57s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 3s	master passed
+1 💚	compile	0m 41s	master passed
+1 💚	shadedjars	4m 15s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 51s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 14s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	the patch passed
		_ Other Tests _
-1 ❌	unit	8m 31s	hbase-server in the patch failed.
		27m 55s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5074
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 4c36e9caffdc 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `d2b0074`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/testReport/
Max. process+thread count	1107 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…ne scan queues

Apache-HBase · 2023-03-09T11:06:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 24s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	6m 18s	master passed
+1 💚	compile	3m 14s	master passed
+1 💚	checkstyle	0m 49s	master passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 18s	the patch passed
+1 💚	compile	2m 22s	the patch passed
+1 💚	javac	2m 22s	the patch passed
+1 💚	checkstyle	0m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	12m 7s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 38s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		41m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5074
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 9c24f928089a 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0491524`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	86 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-03-09T14:31:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 11s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 4s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	4m 16s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 46s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	4m 17s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	219m 37s	hbase-server in the patch passed.
		245m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5074
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 6818308a1df1 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0491524`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/2/testReport/
Max. process+thread count	2575 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2023-03-09T14:54:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 45s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	5m 30s	master passed
+1 💚	compile	1m 6s	master passed
+1 💚	shadedjars	6m 0s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 37s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	4m 52s	the patch passed
+1 💚	compile	0m 58s	the patch passed
+1 💚	javac	0m 58s	the patch passed
+1 💚	shadedjars	5m 58s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 31s	the patch passed
		_ Other Tests _
+1 💚	unit	237m 32s	hbase-server in the patch passed.
		268m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5074
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 8d57064b95d3 5.4.0-1092-aws #100~18.04.2-Ubuntu SMP Tue Nov 29 08:39:52 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `0491524`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/2/testReport/
Max. process+thread count	2673 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5074/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

bbeaudreault · 2023-03-21T11:27:56Z

Makes sense to me, but the UT failures look related

sunhelly · 2023-03-21T11:56:21Z

Thanks, @bbeaudreault . The last failed UT is related, I have fixed it.

bbeaudreault · 2023-03-21T14:30:30Z

Oh sorry, my GitHub was outdated. I see pre-commit is green now. LGTM

…ne scan queues (#5074) Signed-off-by: Bryan Beaudreault <[email protected]>

…ne scan queues (apache#5074) Signed-off-by: Bryan Beaudreault <[email protected]>

…ne scan queues (apache#5074) Signed-off-by: Bryan Beaudreault <[email protected]> (cherry picked from commit 8b0ae80) Change-Id: I35de74ba99deb0adac3795d769b7eeaea4b97769

HBASE-27676 Scan handlers in the RPC executor should match at least o…

842435a

…ne scan queues

sunhelly force-pushed the HBASE-27676 branch from c2349db to 842435a Compare March 9, 2023 10:17

sunhelly requested review from Apache9 and bbeaudreault March 21, 2023 02:10

bbeaudreault approved these changes Mar 21, 2023

View reviewed changes

sunhelly merged commit 7d6a79b into apache:master Mar 22, 2023

asfgit pushed a commit that referenced this pull request Mar 22, 2023

HBASE-27676 Scan handlers in the RPC executor should match at least o…

d4edfe4

…ne scan queues (#5074) Signed-off-by: Bryan Beaudreault <[email protected]>

asfgit pushed a commit that referenced this pull request Mar 22, 2023

HBASE-27676 Scan handlers in the RPC executor should match at least o…

8b0ae80

…ne scan queues (#5074) Signed-off-by: Bryan Beaudreault <[email protected]>

asfgit pushed a commit that referenced this pull request Mar 22, 2023

HBASE-27676 Scan handlers in the RPC executor should match at least o…

40d2545

…ne scan queues (#5074) Signed-off-by: Bryan Beaudreault <[email protected]>

wesiq pushed a commit to web-iq/hbase that referenced this pull request Mar 27, 2023

HBASE-27676 Scan handlers in the RPC executor should match at least o…

610e1ce

…ne scan queues (apache#5074) Signed-off-by: Bryan Beaudreault <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27676 Scan handlers in the RPC executor should match at least o… #5074

HBASE-27676 Scan handlers in the RPC executor should match at least o… #5074

sunhelly commented Mar 2, 2023 •

edited

Loading

Apache-HBase commented Mar 2, 2023

Apache-HBase commented Mar 2, 2023

Apache-HBase commented Mar 2, 2023

Apache-HBase commented Mar 9, 2023

Apache-HBase commented Mar 9, 2023

Apache-HBase commented Mar 9, 2023

bbeaudreault commented Mar 21, 2023

sunhelly commented Mar 21, 2023

bbeaudreault commented Mar 21, 2023

HBASE-27676 Scan handlers in the RPC executor should match at least o… #5074

HBASE-27676 Scan handlers in the RPC executor should match at least o… #5074

Conversation

sunhelly commented Mar 2, 2023 • edited Loading

Apache-HBase commented Mar 2, 2023

Apache-HBase commented Mar 2, 2023

Apache-HBase commented Mar 2, 2023

Apache-HBase commented Mar 9, 2023

Apache-HBase commented Mar 9, 2023

Apache-HBase commented Mar 9, 2023

bbeaudreault commented Mar 21, 2023

sunhelly commented Mar 21, 2023

bbeaudreault commented Mar 21, 2023

sunhelly commented Mar 2, 2023 •

edited

Loading