Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im… #6262

Merged
merged 4 commits into from
Dec 1, 2023
Merged

HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im… #6262

merged 4 commits into from
Dec 1, 2023

Conversation

mccormickt12
Copy link
Contributor

…plementations (#6198)

The enclosing root path is a common ancestor that should be used for temp and staging dirs as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick

Description of PR

Cherry pick of #6198 (review)

How was this patch tested?

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@mccormickt12
HDFS-16791. Add getEnclosingRoot() API to filesystem interface and im…
82285ce 
…plementations (apache#6198)

The enclosing root path is a common ancestor that should be used for temp and staging dirs
as well as within encryption zones and other restricted directories.

Contributed by Tom McCormick
@mccormickt12
Copy link
Contributor Author

@steveloughran Nearly clean cherry pick. Just a couple of things that exist in trunk but not in 3.3, so all deletes.

Unmerged paths:
  (use "git add <file>..." to mark resolution)
        both modified:   hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
        both modified:   hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/placement/

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 5m 10s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+0 🆗 buf 0m 0s buf was not available.
+0 🆗 buf 0m 0s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 11 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 14m 3s Maven dependency ordering for branch
+1 💚 mvninstall 22m 46s branch-3.3 passed
+1 💚 compile 11m 38s branch-3.3 passed
+1 💚 checkstyle 1m 47s branch-3.3 passed
+1 💚 mvnsite 3m 59s branch-3.3 passed
+1 💚 javadoc 3m 38s branch-3.3 passed
+1 💚 spotbugs 6m 33s branch-3.3 passed
+1 💚 shadedclient 23m 28s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 19s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
-1 ❌ compile 3m 13s /patch-compile-root.txt root in the patch failed.
-1 ❌ cc 3m 13s /patch-compile-root.txt root in the patch failed.
-1 ❌ javac 3m 13s /patch-compile-root.txt root in the patch failed.
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 32s the patch passed
-1 ❌ mvnsite 0m 22s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
-1 ❌ javadoc 0m 43s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-project_hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
-1 ❌ spotbugs 0m 21s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
+1 💚 shadedclient 23m 47s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 15m 23s hadoop-common in the patch passed.
+1 💚 unit 1m 53s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 179m 50s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 0m 32s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
+1 💚 asflicense 0m 40s The patch does not generate ASF License warnings.
336m 37s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/1/artifact/out/Dockerfile
GITHUB PR #6262
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname Linux 064de127f595 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 82285ce
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/1/testReport/
Max. process+thread count 3227 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/1/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor

looks like the hadoop-rbf stuff didn't take. just did a local build of branch-3.3 and it is happy, so this probably really is a regression

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 26s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+0 🆗 buf 0m 0s buf was not available.
+0 🆗 buf 0m 0s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 11 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 14m 0s Maven dependency ordering for branch
+1 💚 mvninstall 22m 12s branch-3.3 passed
+1 💚 compile 11m 45s branch-3.3 passed
+1 💚 checkstyle 1m 49s branch-3.3 passed
+1 💚 mvnsite 3m 57s branch-3.3 passed
+1 💚 javadoc 3m 39s branch-3.3 passed
+1 💚 spotbugs 6m 32s branch-3.3 passed
+1 💚 shadedclient 23m 29s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 26s Maven dependency ordering for patch
+1 💚 mvninstall 2m 35s the patch passed
+1 💚 compile 11m 10s the patch passed
+1 💚 cc 11m 10s the patch passed
+1 💚 javac 11m 10s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 52s the patch passed
+1 💚 mvnsite 3m 56s the patch passed
+1 💚 javadoc 3m 38s the patch passed
+1 💚 spotbugs 7m 11s the patch passed
+1 💚 shadedclient 23m 38s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 15m 54s hadoop-common in the patch passed.
+1 💚 unit 2m 7s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 178m 50s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 unit 15m 14s hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
359m 19s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier
hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks
hadoop.hdfs.server.datanode.TestDirectoryScanner
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/2/artifact/out/Dockerfile
GITHUB PR #6262
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname Linux be5ac791eb92 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / d74f8cb
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/2/testReport/
Max. process+thread count 3140 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/2/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 33s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+0 🆗 buf 0m 0s buf was not available.
+0 🆗 buf 0m 0s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 11 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 13m 51s Maven dependency ordering for branch
+1 💚 mvninstall 24m 44s branch-3.3 passed
+1 💚 compile 11m 48s branch-3.3 passed
+1 💚 checkstyle 1m 47s branch-3.3 passed
+1 💚 mvnsite 3m 58s branch-3.3 passed
+1 💚 javadoc 3m 44s branch-3.3 passed
+1 💚 spotbugs 6m 40s branch-3.3 passed
+1 💚 shadedclient 23m 18s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 50s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 30s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt hadoop-hdfs-client in the patch failed.
-1 ❌ mvninstall 0m 30s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ mvninstall 0m 19s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
-1 ❌ compile 1m 28s /patch-compile-root.txt root in the patch failed.
-1 ❌ cc 1m 28s /patch-compile-root.txt root in the patch failed.
-1 ❌ javac 1m 28s /patch-compile-root.txt root in the patch failed.
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 30s the patch passed
-1 ❌ mvnsite 0m 32s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt hadoop-hdfs-client in the patch failed.
-1 ❌ mvnsite 0m 32s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ mvnsite 0m 22s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
-1 ❌ javadoc 0m 58s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project_hadoop-hdfs generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)
-1 ❌ spotbugs 0m 32s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt hadoop-hdfs-client in the patch failed.
-1 ❌ spotbugs 0m 30s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ spotbugs 0m 21s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
-1 ❌ shadedclient 7m 19s patch has errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 15m 25s hadoop-common in the patch passed.
-1 ❌ unit 0m 33s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt hadoop-hdfs-client in the patch failed.
-1 ❌ unit 0m 31s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ unit 0m 21s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
+1 💚 asflicense 0m 25s The patch does not generate ASF License warnings.
131m 51s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/3/artifact/out/Dockerfile
GITHUB PR #6262
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname Linux 729255df80a4 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19 13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 9603a9e
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/3/testReport/
Max. process+thread count 1637 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/3/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 4m 25s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+0 🆗 buf 0m 0s buf was not available.
+0 🆗 buf 0m 0s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 11 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 13m 38s Maven dependency ordering for branch
+1 💚 mvninstall 23m 4s branch-3.3 passed
+1 💚 compile 12m 4s branch-3.3 passed
+1 💚 checkstyle 1m 28s branch-3.3 passed
+1 💚 mvnsite 3m 11s branch-3.3 passed
+1 💚 javadoc 2m 48s branch-3.3 passed
+1 💚 spotbugs 5m 38s branch-3.3 passed
+1 💚 shadedclient 22m 18s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 22s Maven dependency ordering for patch
-1 ❌ mvninstall 0m 28s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt hadoop-hdfs-client in the patch failed.
-1 ❌ mvninstall 0m 28s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ mvninstall 0m 17s /patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
-1 ❌ compile 1m 34s /patch-compile-root.txt root in the patch failed.
-1 ❌ cc 1m 34s /patch-compile-root.txt root in the patch failed.
-1 ❌ javac 1m 34s /patch-compile-root.txt root in the patch failed.
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 16s the patch passed
-1 ❌ mvnsite 0m 29s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt hadoop-hdfs-client in the patch failed.
-1 ❌ mvnsite 0m 29s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ mvnsite 0m 19s /patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
-1 ❌ javadoc 0m 59s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs-project_hadoop-hdfs generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)
-1 ❌ spotbugs 0m 27s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt hadoop-hdfs-client in the patch failed.
-1 ❌ spotbugs 0m 28s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ spotbugs 0m 18s /patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
-1 ❌ shadedclient 6m 10s patch has errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 15m 52s hadoop-common in the patch passed.
-1 ❌ unit 0m 32s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt hadoop-hdfs-client in the patch failed.
-1 ❌ unit 0m 31s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch failed.
-1 ❌ unit 0m 19s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
+1 💚 asflicense 0m 22s The patch does not generate ASF License warnings.
126m 16s
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/4/artifact/out/Dockerfile
GITHUB PR #6262
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname Linux 0404f4720b88 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 9603a9e
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/4/testReport/
Max. process+thread count 3152 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/4/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@mccormickt12
Copy link
Contributor Author

@steveloughran This kind of seems like yetus issues to me now.
#6262 (comment) This check was good after I fixed the path import (you can tell from the revision)

I also locally saw one other failure locally which was the protobuf static import thats not in 3.3.
From a local build i get this (i get this vectorized failure locally even on trunk and 3.3 branch, as do others at LinkedIn)

[INFO] Reactor Summary for Apache Hadoop Main 3.3.9-SNAPSHOT:
[INFO] 
[INFO] Apache Hadoop Main ................................. SUCCESS [  2.175 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  1.702 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.783 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [  4.200 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.440 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.311 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  4.932 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.763 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 14.290 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  3.420 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 52.162 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [  5.820 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [  4.902 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [  4.801 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.221 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 27.815 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:03 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [  6.074 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [  7.388 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  8.744 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 16.316 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.110 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.115 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 16.592 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 19.048 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.151 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 18.680 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 16.008 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  5.339 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  6.741 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  5.562 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 28.425 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 12.229 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 11.407 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  6.134 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  6.123 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  0.100 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [  6.191 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [  8.402 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [  0.223 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SUCCESS [  8.135 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [  6.883 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  6.092 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [  6.275 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.202 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  6.363 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  4.921 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  3.996 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 12.177 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [  8.662 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 16.345 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 35.711 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  7.221 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 16.106 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  6.607 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.121 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [  9.997 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 11.396 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [  0.164 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ 30.373 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image  SUCCESS [  1.290 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [  0.109 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [  4.645 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.161 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  3.270 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.213 s]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [ 18.946 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 40.277 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  4.629 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  6.116 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  4.928 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  7.062 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  9.783 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  7.484 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  7.579 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  6.449 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [  5.438 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [  7.000 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [  5.220 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [ 14.487 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [  0.112 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [  5.620 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  5.841 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [  6.825 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 24.614 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 15.760 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [  6.101 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.132 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 17.450 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  3.482 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 11.015 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  4.691 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  6.107 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  4.900 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  4.188 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [  6.270 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 34.632 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  3.371 s]
[INFO] Apache Hadoop Common Benchmark ..................... FAILURE [ 19.398 s]
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Tencent COS Support .................. SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  15:36 min
[INFO] Finished at: 2023-11-18T11:43:02-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.4:single (make-assembly) on project hadoop-benchmark: Failed to create assembly: Error creating assembly archive uber: Problem creating jar: jar:file:/Users/tmccormi/projects/hadoop/hadoop-tools/hadoop-benchmark/target/hadoop-benchmark-3.3.9-SNAPSHOT.jar!/org/apache/hadoop/benchmark/generated/VectoredReadBenchmark_jmhType_B1.class -> [Help 1]
[ERROR]

Im gonna try reverting the last commit (although it seems needed) to see if yetus likes it more

@mccormickt12
Copy link
Contributor Author

After the revert I get building locally

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-hdfs-client: Compilation failure
[ERROR] /Users/tmccormi/projects/hadoop/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java:[2098,13] cannot find symbol
[ERROR]   symbol:   method getRemoteException(org.apache.hadoop.thirdparty.protobuf.ServiceException)
[ERROR]   location: class org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 20s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 detsecrets 0m 1s detect-secrets was not available.
+0 🆗 markdownlint 0m 1s markdownlint was not available.
+0 🆗 buf 0m 1s buf was not available.
+0 🆗 buf 0m 1s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 11 new or modified test files.
_ branch-3.3 Compile Tests _
+0 🆗 mvndep 13m 39s Maven dependency ordering for branch
+1 💚 mvninstall 22m 42s branch-3.3 passed
+1 💚 compile 11m 3s branch-3.3 passed
+1 💚 checkstyle 1m 31s branch-3.3 passed
+1 💚 mvnsite 3m 11s branch-3.3 passed
+1 💚 javadoc 2m 56s branch-3.3 passed
+1 💚 spotbugs 5m 51s branch-3.3 passed
+1 💚 shadedclient 21m 9s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 21s Maven dependency ordering for patch
+1 💚 mvninstall 2m 23s the patch passed
+1 💚 compile 10m 42s the patch passed
+1 💚 cc 10m 42s the patch passed
+1 💚 javac 10m 42s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 25s the patch passed
+1 💚 mvnsite 3m 11s the patch passed
+1 💚 javadoc 2m 52s the patch passed
+1 💚 spotbugs 6m 11s the patch passed
+1 💚 shadedclient 21m 12s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 15m 42s hadoop-common in the patch passed.
+1 💚 unit 1m 55s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 169m 22s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 unit 14m 43s hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 42s The patch does not generate ASF License warnings.
335m 54s
Reason Tests
Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
Subsystem Report/Notes
Docker ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/5/artifact/out/Dockerfile
GITHUB PR #6262
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint cc buflint bufcompat
uname Linux c32133654be4 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision branch-3.3 / 6a7e834
Default Java Private Build-1.8.0_362-8u372-gaus1-0ubuntu118.04-b09
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/5/testReport/
Max. process+thread count 4699 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6262/5/console
versions git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@mccormickt12
Copy link
Contributor Author

@steveloughran Let me know if theres any concerns with the current failure, it seems transient / expected to me, but not 100%
cc @simbadzina

@mccormickt12
Copy link
Contributor Author

@goiri Thoughts? Does this backport look ok. The failure doesn't seem related to my backport right?

@goiri
Copy link
Member

goiri commented Nov 30, 2023

@goiri Thoughts? Does this backport look ok. The failure doesn't seem related to my backport right?

LGTM.

@steveloughran do you mind taking a look?

@steveloughran
Copy link
Contributor

I'm going to say unrelated. HDFS-16740/#4835 addresses flakiness in trunk, but it's not been backported. as usual: volunteers welcome.

@steveloughran steveloughran merged commit e82d83e into apache:branch-3.3 Dec 1, 2023
1 check failed
@steveloughran
Copy link
Contributor

merged. If you are on common-dev you can see we've started a discussion about pushing out a branch-3.3 release...

@mccormickt12 mccormickt12 mentioned this pull request Dec 6, 2023
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@goiri goiri goiri approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mccormickt12 @hadoop-yetus @steveloughran @goiri