Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-15886. Add a way to get protected dirs from a special configuration file. #2768

Open
wants to merge 7 commits into
base: trunk
Choose a base branch
from

Conversation

Neilxzn
Copy link
Contributor

@Neilxzn Neilxzn commented Mar 12, 2021

jira

https://issues.apache.org/jira/browse/HDFS-15886

We used protected dirs to ensure that important data directories cannot be deleted by mistake. But protected dirs can only be configured in hdfs-site.xml.

For ease of management,  we add a way to get the list of protected dirs from a special configuration file.

How to use.

  1. set the config in hdfs-site.xml

<property>
<name>fs.protected.directories</name>
<value>/hdfs/path/1,/hdfs/path/2,file:///path/to/protected.dirs.config</value>
</property>

2.  add some protected dirs to the config file (file:///path/to/protected.dirs.config)


/hdfs/path/4

/hdfs/path/5

  1. use command to refresh fs.protected.directories instead of FSDirectory.setProtectedDirectories(..)

hdfs dfsadmin -refreshProtectedDirectories

 

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 9s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 buf 0m 1s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 10s Maven dependency ordering for branch
+1 💚 mvninstall 28m 3s trunk passed
+1 💚 compile 28m 42s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 compile 23m 51s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 checkstyle 5m 0s trunk passed
+1 💚 mvnsite 5m 11s trunk passed
+1 💚 javadoc 3m 52s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 5m 18s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 spotbugs 12m 23s trunk passed
+1 💚 shadedclient 17m 9s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 23s Maven dependency ordering for patch
+1 💚 mvninstall 3m 47s the patch passed
+1 💚 compile 25m 38s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
-1 ❌ cc 25m 38s /results-compile-cc-root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 29 new + 383 unchanged - 29 fixed = 412 total (was 412)
+1 💚 javac 25m 38s the patch passed
+1 💚 compile 19m 1s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
-1 ❌ cc 19m 1s /results-compile-cc-root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu120.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu120.04-b08 generated 22 new + 390 unchanged - 22 fixed = 412 total (was 412)
+1 💚 javac 19m 1s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 3m 59s /results-checkstyle-root.txt root: The patch generated 13 new + 1298 unchanged - 1 fixed = 1311 total (was 1299)
+1 💚 mvnsite 4m 39s the patch passed
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
-1 ❌ javadoc 0m 44s /patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-hdfs-client in the patch failed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.
+1 💚 javadoc 4m 45s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 spotbugs 10m 31s the patch passed
+1 💚 shadedclient 16m 55s patch has no errors when building and testing our client artifacts.
_ Other Tests _
-1 ❌ unit 17m 21s /patch-unit-hadoop-common-project_hadoop-common.txt hadoop-common in the patch passed.
+1 💚 unit 2m 25s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 213m 44s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
+1 💚 unit 17m 51s hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 0m 54s The patch does not generate ASF License warnings.
493m 14s
Reason Tests
Failed junit tests hadoop.util.TestConfTest
hadoop.hdfs.tools.TestDFSAdmin
hadoop.hdfs.TestDistributedFileSystem
hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/1/artifact/out/Dockerfile
GITHUB PR #2768
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat xml
uname Linux 3e221dc703be 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / 299eea1
Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/1/testReport/
Max. process+thread count 2441 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

2. fix TestDFSAdmin.testNameNodeGetReconfigurableProperties (set expected value to 12)
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 50s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 buf 0m 0s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 21s Maven dependency ordering for branch
+1 💚 mvninstall 22m 53s trunk passed
+1 💚 compile 22m 3s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 compile 18m 56s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 checkstyle 4m 7s trunk passed
+1 💚 mvnsite 4m 38s trunk passed
+1 💚 javadoc 3m 26s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 4m 40s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 spotbugs 9m 42s trunk passed
+1 💚 shadedclient 16m 53s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 20s Maven dependency ordering for patch
+1 💚 mvninstall 3m 25s the patch passed
+1 💚 compile 21m 40s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
-1 ❌ cc 21m 40s /results-compile-cc-root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 25 new + 387 unchanged - 25 fixed = 412 total (was 412)
+1 💚 javac 21m 40s the patch passed
+1 💚 compile 18m 53s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
-1 ❌ cc 18m 53s /results-compile-cc-root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu120.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu120.04-b08 generated 44 new + 368 unchanged - 44 fixed = 412 total (was 412)
+1 💚 javac 18m 53s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 4m 1s /results-checkstyle-root.txt root: The patch generated 2 new + 1300 unchanged - 1 fixed = 1302 total (was 1301)
+1 💚 mvnsite 4m 36s the patch passed
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
-1 ❌ javadoc 0m 45s /results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 1 new + 98 unchanged - 1 fixed = 99 total (was 99)
+1 💚 javadoc 4m 48s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 spotbugs 10m 27s the patch passed
+1 💚 shadedclient 16m 52s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 17m 9s hadoop-common in the patch passed.
+1 💚 unit 2m 23s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 332m 4s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 1m 7s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch failed.
+1 💚 asflicense 0m 55s The patch does not generate ASF License warnings.
567m 29s
Reason Tests
Failed junit tests hadoop.hdfs.tools.TestDFSZKFailoverController
hadoop.hdfs.server.datanode.TestBlockScanner
hadoop.hdfs.server.mover.TestMover
hadoop.hdfs.TestDFSShell
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor
hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
hadoop.hdfs.server.namenode.TestDecommissioningStatus
hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList
hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
hadoop.hdfs.server.namenode.TestPersistentStoragePolicySatisfier
hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.TestPersistBlocks
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/2/artifact/out/Dockerfile
GITHUB PR #2768
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat xml
uname Linux f7fb86964380 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / da0dbf1
Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/2/testReport/
Max. process+thread count 2295 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/2/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 0m 35s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 1s codespell was not available.
+0 🆗 buf 0m 1s buf was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 4 new or modified test files.
_ trunk Compile Tests _
+0 🆗 mvndep 14m 42s Maven dependency ordering for branch
+1 💚 mvninstall 20m 10s trunk passed
+1 💚 compile 20m 31s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 compile 17m 54s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 checkstyle 3m 54s trunk passed
+1 💚 mvnsite 5m 6s trunk passed
+1 💚 javadoc 3m 51s trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 5m 10s trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 spotbugs 9m 35s trunk passed
+1 💚 shadedclient 14m 46s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗 mvndep 0m 25s Maven dependency ordering for patch
+1 💚 mvninstall 3m 23s the patch passed
+1 💚 compile 19m 55s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
-1 ❌ cc 19m 55s /results-compile-cc-root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04.txt root-jdkUbuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 generated 11 new + 401 unchanged - 11 fixed = 412 total (was 412)
+1 💚 javac 19m 55s the patch passed
+1 💚 compile 17m 53s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
-1 ❌ cc 17m 53s /results-compile-cc-root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08.txt root-jdkPrivateBuild-1.8.0_282-8u282-b08-0ubuntu120.04-b08 with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu120.04-b08 generated 31 new + 381 unchanged - 31 fixed = 412 total (was 412)
+1 💚 javac 17m 53s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
-0 ⚠️ checkstyle 3m 51s /results-checkstyle-root.txt root: The patch generated 1 new + 1300 unchanged - 1 fixed = 1301 total (was 1301)
+1 💚 mvnsite 5m 6s the patch passed
+1 💚 xml 0m 1s The patch has no ill-formed XML file.
+1 💚 javadoc 3m 53s the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 5m 17s the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚 spotbugs 10m 21s the patch passed
+1 💚 shadedclient 14m 50s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 17m 40s hadoop-common in the patch passed.
+1 💚 unit 2m 45s hadoop-hdfs-client in the patch passed.
-1 ❌ unit 373m 38s /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt hadoop-hdfs in the patch passed.
-1 ❌ unit 18m 3s /patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt hadoop-hdfs-rbf in the patch passed.
+1 💚 asflicense 1m 8s The patch does not generate ASF License warnings.
618m 4s
Reason Tests
Failed junit tests hadoop.hdfs.TestDFSShell
hadoop.hdfs.TestErasureCodingPolicies
hadoop.hdfs.TestDistributedFileSystem
hadoop.hdfs.server.balancer.TestBalancer
hadoop.hdfs.TestDFSInotifyEventInputStream
hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
hadoop.hdfs.TestEncryptionZones
hadoop.hdfs.TestErasureCodingPolicyWithSnapshot
hadoop.hdfs.TestDFSStripedOutputStream
hadoop.hdfs.TestReadStripedFileWithDNFailure
hadoop.hdfs.TestFileChecksum
hadoop.hdfs.server.balancer.TestBalancerRPCDelay
hadoop.hdfs.tools.TestViewFileSystemOverloadSchemeWithFSCommands
hadoop.hdfs.TestLeaseRecovery2
hadoop.hdfs.TestBlockTokenWrappingQOP
hadoop.hdfs.TestDatanodeDeath
hadoop.hdfs.tools.TestECAdmin
hadoop.hdfs.TestFileAppend4
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy
hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor
hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy
hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy
hadoop.hdfs.TestReadStripedFileWithMissingBlocks
hadoop.hdfs.TestErasureCodingExerciseAPIs
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
hadoop.hdfs.TestErasureCodingMultipleRacks
hadoop.hdfs.TestStoragePolicyPermissionSettings
hadoop.hdfs.TestStateAlignmentContextWithHA
hadoop.hdfs.TestDistributedFileSystemWithECFile
hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
hadoop.hdfs.tools.TestWebHDFSStoragePolicyCommands
hadoop.hdfs.server.datanode.TestDirectoryScanner
hadoop.hdfs.TestDecommissionWithStriped
hadoop.fs.contract.router.web.TestRouterWebHDFSContractCreate
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/3/artifact/out/Dockerfile
GITHUB PR #2768
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell cc buflint bufcompat xml
uname Linux 7faa97e45c26 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / c045b76
Default Java Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/3/testReport/
Max. process+thread count 2854 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2768/3/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@Neilxzn
Copy link
Contributor Author

Neilxzn commented Mar 15, 2021

@sodonnel @tomscut I think these tests that failed are unrelated to this patch according to Jenkins logs.
Can you please take a look for this? Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants