YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled #4703

ashutoshcipher · 2022-08-04T18:19:48Z

Description of PR

Add uncleaning option for local app log file with log-aggregation enabled

JIRA - YARN-11241

How was this patch tested?

Unit test

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

…regation enabled

ashutoshcipher · 2022-08-05T15:48:28Z

FYI: Checkstyle -0 showing here is not due changes in this PR.

aajisaka

I think this option is very useful for testing, but we need to make sure the user will not set this parameter to false in any production systems. Added a few comments.

...che/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java

hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml

...che/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java

...adoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java

ashutoshcipher · 2022-08-31T13:32:00Z

@aajisaka - I have addressed you comments. Can you please review it again. Thank you so much.

ashutoshcipher · 2022-09-07T22:19:01Z

@aajisaka - Please help in reviewing the PR. Thanks.

ashutoshcipher · 2022-09-10T18:36:07Z

@aajisaka - I have addressed your last comments. Please help in reviewing it again.

Thanks.

...adoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java

hadoop-yetus · 2022-09-12T08:35:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 43s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 22s		trunk passed
+1 💚	compile	10m 40s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	9m 20s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	2m 0s		trunk passed
+1 💚	mvnsite	3m 58s		trunk passed
+1 💚	javadoc	3m 33s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 17s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	6m 46s		trunk passed
+1 💚	shadedclient	21m 50s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 30s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 18s		the patch passed
+1 💚	compile	10m 0s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	10m 0s		the patch passed
+1 💚	compile	11m 59s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	11m 59s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 59s		hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 286 unchanged - 2 fixed = 286 total (was 288)
+1 💚	mvnsite	3m 23s		the patch passed
+1 💚	javadoc	3m 11s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 6s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	7m 22s		the patch passed
+1 💚	shadedclient	26m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 34s		hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 28s		hadoop-yarn-common in the patch passed.
+1 💚	unit	24m 55s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	1m 13s		The patch does not generate ASF License warnings.
		210m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4703/6/artifact/out/Dockerfile
GITHUB PR	#4703
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux 38302a3fc2e3 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `33edb8c`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4703/6/testReport/
Max. process+thread count	716 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4703/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

aajisaka

LGTM, +1

…regation enabled (#4703) Co-authored-by: Ashutosh Gupta <[email protected]> Signed-off-by: Akira Ajisaka <[email protected]> (cherry picked from commit 65a027b)

…pload Failed After YARN-11241(apache#4703).

…regation enabled (apache#4703) Co-authored-by: Ashutosh Gupta <[email protected]> Signed-off-by: Akira Ajisaka <[email protected]>

YARN-11241. Add uncleaning option for local app log file with log-agg…

fc83ec1

…regation enabled

This comment was marked as outdated.

Sign in to view

Adding configuration property to yarn-default.xml

dad54d3

This comment was marked as outdated.

Sign in to view

ashutoshcipher marked this pull request as ready for review August 5, 2022 15:59

aajisaka requested changes Aug 30, 2022

View reviewed changes

Addressing comments

7c52957

ashutoshcipher requested a review from aajisaka August 30, 2022 10:37

Removing not required while loops

0481171

This comment was marked as outdated.

Sign in to view

Addding period in javadoc

02c16cd

This comment was marked as outdated.

Sign in to view

aajisaka requested changes Sep 10, 2022

View reviewed changes

...adoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java Outdated Show resolved Hide resolved

...adoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java Outdated Show resolved Hide resolved

adressing comments related to sleep

33edb8c

ashutoshcipher requested a review from aajisaka September 12, 2022 11:02

aajisaka approved these changes Sep 12, 2022

View reviewed changes

aajisaka merged commit 65a027b into apache:trunk Sep 12, 2022

slfan1989 pushed a commit to slfan1989/hadoop that referenced this pull request Sep 15, 2022

YARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterU…

fffd3ee

…pload Failed After YARN-11241(apache#4703).

slfan1989 pushed a commit to slfan1989/hadoop that referenced this pull request Sep 15, 2022

YARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterU…

748fa05

…pload Failed After YARN-11241(apache#4703).

slfan1989 mentioned this pull request Sep 15, 2022

YARN-11305. Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After YARN-11241(#4703). #4893

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled #4703

YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled #4703

ashutoshcipher commented Aug 4, 2022 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

ashutoshcipher commented Aug 5, 2022

aajisaka left a comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

ashutoshcipher commented Aug 31, 2022

ashutoshcipher commented Sep 7, 2022

ashutoshcipher commented Sep 10, 2022

hadoop-yetus commented Sep 12, 2022

aajisaka left a comment

YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled #4703

YARN-11241. Add uncleaning option for local app log file with log-aggregation enabled #4703

Conversation

ashutoshcipher commented Aug 4, 2022 • edited Loading

Description of PR

How was this patch tested?

For code changes:

This comment was marked as outdated.

This comment was marked as outdated.

ashutoshcipher commented Aug 5, 2022

aajisaka left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

ashutoshcipher commented Aug 31, 2022

ashutoshcipher commented Sep 7, 2022

ashutoshcipher commented Sep 10, 2022

hadoop-yetus commented Sep 12, 2022

aajisaka left a comment

Choose a reason for hiding this comment

ashutoshcipher commented Aug 4, 2022 •

edited

Loading