YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class #3800

szilard-nemeth · 2021-12-14T22:11:03Z

Description of PR

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2021-12-15T01:19:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 27s		trunk passed
+1 💚	compile	1m 7s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 57s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 49s		trunk passed
+1 💚	mvnsite	1m 3s		trunk passed
+1 💚	javadoc	0m 52s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 45s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 53s		trunk passed
+1 💚	shadedclient	21m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 54s		the patch passed
+1 💚	compile	0m 57s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 57s		the patch passed
+1 💚	compile	0m 50s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 50s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 40s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 124 unchanged - 4 fixed = 127 total (was 128)
+1 💚	mvnsite	0m 52s		the patch passed
+1 💚	javadoc	0m 39s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 38s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
-1 ❌	spotbugs	1m 57s	/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	20m 57s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	96m 24s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		187m 5s

Reason	Tests
SpotBugs	module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
	Unread field:field be static? At CapacityScheduler.java:[line 3456]
Failed junit tests	hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
	hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/1/artifact/out/Dockerfile
GITHUB PR	#3800
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 44bcf4a465ed 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `5bbf36f`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/1/testReport/
Max. process+thread count	941 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-16T02:03:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 44s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 51s		trunk passed
+1 💚	compile	1m 6s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 59s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 53s		trunk passed
+1 💚	mvnsite	1m 5s		trunk passed
+1 💚	javadoc	0m 52s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 47s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 54s		trunk passed
+1 💚	shadedclient	21m 12s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 53s		the patch passed
+1 💚	compile	0m 59s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 59s		the patch passed
+1 💚	compile	0m 49s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 49s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	0m 44s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 493 unchanged - 4 fixed = 495 total (was 497)
+1 💚	mvnsite	0m 52s		the patch passed
+1 💚	javadoc	0m 39s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 38s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 54s		the patch passed
+1 💚	shadedclient	21m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	104m 9s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		197m 31s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/2/artifact/out/Dockerfile
GITHUB PR	#3800
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 76a6fe9d3a78 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `063e156`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/2/testReport/
Max. process+thread count	948 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

shuzirra · 2021-12-16T15:27:06Z

@szilard-nemeth thank you for working on this, patch seems fine, but let's wait for a green jenkins run :)

hadoop-yetus · 2021-12-16T18:02:30Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 6s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 42s		trunk passed
+1 💚	compile	1m 6s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 58s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 51s		trunk passed
+1 💚	mvnsite	1m 2s		trunk passed
+1 💚	javadoc	0m 50s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 44s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 55s		trunk passed
+1 💚	shadedclient	22m 6s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 57s		the patch passed
+1 💚	compile	1m 3s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	1m 3s		the patch passed
+1 💚	compile	0m 51s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 41s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 603 unchanged - 4 fixed = 605 total (was 607)
+1 💚	mvnsite	0m 59s		the patch passed
+1 💚	javadoc	0m 41s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 37s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	2m 3s		the patch passed
+1 💚	shadedclient	22m 14s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	111m 16s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		206m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/3/artifact/out/Dockerfile
GITHUB PR	#3800
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 16ebf4264f90 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `973bfb9`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/3/testReport/
Max. process+thread count	970 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3800/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

shuzirra · 2021-12-16T23:18:42Z

@szilard-nemeth thank you for working on this, LGTM+1, merged to trunk.

…hat belongs to async scheduling to a new class (apache#3800). Contributed by Szilard Nemeth

…lizer code that belongs to async scheduling to a new class (apache#3800). Contributed by Szilard Nemeth (cherry picked from commit 8d214cb) Change-Id: I4379dd9ea1bbe31b53a69540ca2fc6294caebcf4

YARN-10951

5bbf36f

szilard-nemeth mentioned this pull request Dec 14, 2021

YARN-10951. Move async scheduling fields to a new AsyncSchedulingConfiguration #3438

Closed

szilard-nemeth added 2 commits December 15, 2021 17:26

fixes

4dbb66f

Pass AsyncSchedulingConf to TestUtils#applyResourceCommitRequest

063e156

Fix tests again

973bfb9

shuzirra self-requested a review December 16, 2021 15:10

shuzirra approved these changes Dec 16, 2021

View reviewed changes

shuzirra merged commit 8d214cb into apache:trunk Dec 16, 2021

ashutoshcipher pushed a commit to ashutoshcipher/hadoop that referenced this pull request Dec 22, 2021

YARN-10951. CapacityScheduler: Move all fields and initializer code t…

4d54a0e

…hat belongs to async scheduling to a new class (apache#3800). Contributed by Szilard Nemeth

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

YARN-10951. CapacityScheduler: Move all fields and initializer code t…

0e46b74

…hat belongs to async scheduling to a new class (apache#3800). Contributed by Szilard Nemeth

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class #3800

YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class #3800

szilard-nemeth commented Dec 14, 2021

hadoop-yetus commented Dec 15, 2021

hadoop-yetus commented Dec 16, 2021

shuzirra commented Dec 16, 2021

hadoop-yetus commented Dec 16, 2021

shuzirra commented Dec 16, 2021

YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class #3800

YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class #3800

Conversation

szilard-nemeth commented Dec 14, 2021

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Dec 15, 2021

hadoop-yetus commented Dec 16, 2021

shuzirra commented Dec 16, 2021

hadoop-yetus commented Dec 16, 2021

shuzirra commented Dec 16, 2021