HDFS-16457.Make fs.getspaceused.classname reconfigurable #4069
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Hello @jojochuang @ferhui @tasanuma @goiri,PTAL.Thanks a lot. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
The failed check has nothing to do with my changes, the last check passed. @jojochuang ,As can be seen from the generated log, my ut passed. |
|
Can you review my code, thanks a lot. @jojochuang |
|
Thanks for working on this, @singer-bin. I'd like to review your PR after #3863. I hope you will rebase this PR after merging #3863. |
|
@tasanuma Thanks for your comment, I will. |
|
I merged #3863 just now. |
|
💔 -1 overall
This message was automatically generated. |
|
@tasanuma Hello, in the method startDFCluster, I set it as follows: |
tasanuma
reviewed
Mar 31, 2022
Comment on lines
774
to
790
| @Test | ||
| public void testDfsUsageKlass() throws IOException, ReconfigurationException, | ||
| InterruptedException { | ||
|
|
||
| long lastCounter = counter; | ||
| Thread.sleep(5000); | ||
| assertTrue(counter > lastCounter); | ||
|
|
||
| for (int i = 0; i < NUM_DATA_NODE; i++) { | ||
| DataNode dn = cluster.getDataNodes().get(i); | ||
| dn.reconfigurePropertyImpl(FS_GETSPACEUSED_CLASSNAME, null); | ||
| } | ||
|
|
||
| lastCounter = counter; | ||
| Thread.sleep(5000); | ||
| assertEquals(lastCounter, counter); | ||
| } |
There was a problem hiding this comment.
@singer-bin Thanks for updating the PR.
How about we stop using conf.setClass in startDFCluster, and reconfigure DummyCachingGetSpaceUsed in the unit test, and replace the order of the assertions? (But after this change, somehow the unit test failed.)
Suggested change
| @Test | |
| public void testDfsUsageKlass() throws IOException, ReconfigurationException, | |
| InterruptedException { | |
| long lastCounter = counter; | |
| Thread.sleep(5000); | |
| assertTrue(counter > lastCounter); | |
| for (int i = 0; i < NUM_DATA_NODE; i++) { | |
| DataNode dn = cluster.getDataNodes().get(i); | |
| dn.reconfigurePropertyImpl(FS_GETSPACEUSED_CLASSNAME, null); | |
| } | |
| lastCounter = counter; | |
| Thread.sleep(5000); | |
| assertEquals(lastCounter, counter); | |
| } | |
| @Test | |
| public void testDfsUsageKlass() throws IOException, ReconfigurationException, | |
| InterruptedException { | |
| long lastCounter = counter; | |
| Thread.sleep(5000); | |
| assertEquals(lastCounter, counter); | |
| for (int i = 0; i < NUM_DATA_NODE; i++) { | |
| DataNode dn = cluster.getDataNodes().get(i); | |
| dn.reconfigurePropertyImpl(FS_GETSPACEUSED_CLASSNAME, DummyCachingGetSpaceUsed.class.getName()); | |
| } | |
| lastCounter = counter; | |
| Thread.sleep(5000); | |
| assertTrue(counter > lastCounter); | |
| } |
There was a problem hiding this comment.
Hi, thanks for your comment. My UT ideas come from HDFS-15120 ,you can take a look to help understand. My conf comes from the getNewConf() method as follows:
private void reconfSpaceUsedKlass(){
List<FsVolumeImpl> volumeList = data.getVolumeList();
for (FsVolumeImpl fsVolume : volumeList) {
Map<String, BlockPoolSlice> blockPoolSlices = fsVolume.getBlockPoolSlices();
for (BlockPoolSlice bp : blockPoolSlices.values()) {
try {
bp.refreshSpaceUsedKlass(getNewConf());
} catch (IOException e) {
e.printStackTrace();
}
}
}
}
so , passing in DummyCachingGetSpaceUsed.class.toString() in
dn.reconfigurePropertyImpl(FS_GETSPACEUSED_CLASSNAME, DummyCachingGetSpaceUsed.class.toString());
doesn't make sense to me.
I need to set it to my own custom class when the service starts, then wait for the counter to change, and then use dn.reconfigurePropertyImpl(FS_GETSPACEUSED_CLASSNAME, null); to refresh the value to the default,and then the counter is not changing.
There was a problem hiding this comment.
Looking forward to your comments and suggestions. @tasanuma
There was a problem hiding this comment.
- As well as other reconfiguring properties of DataNode, I prefer to be able to update parameters by calling
reconf.. .Parameters(property, newVal). - I think this PR is similar to HDFS-16413. Reconfig dfs usage parameters for datanode #3863. Instead of creating the new method of
reconfSpaceUsedKlass(), how about extendingreconfDfsUsageParametersto updatefs.getspaceused.classname?
There was a problem hiding this comment.
Thank you very much for your comment, I will consider it. @tasanuma
|
💔 -1 overall
This message was automatically generated. |
tasanuma
reviewed
Apr 6, 2022
Comment on lines
904
to
905
| Class<? extends GetSpaceUsed> klass = (newVal == null ? DU.class : | ||
| Class.forName(newVal).asSubclass(GetSpaceUsed.class)); |
There was a problem hiding this comment.
@singer-bin Thanks for updating the PR.
We may need to use WindowsGetSpaceUsed.class if it is windows os.
There was a problem hiding this comment.
Thank you very much for your review, I will revise it right away. @tasanuma
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
I ran it several times, and the error has nothing to do with my code. Can you help me, thank you. @tasanuma |
|
@singer-bin It almost looks good to me. But, I have one more request. Please arrange the order of the import sentences alphabetically as much as possible. If you use IntelliJ, it will do it by pressing Alt + Enter key. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Hello, thank you for your reminder, I have completed the revision, can you take a look at it again? @tasanuma |
tasanuma
approved these changes
Apr 8, 2022
|
Merged it. Thanks for your contribution, @singer-bin! |
|
@singer-bin Could you create the same PR for branch-3.3? There are some conflicts. |
|
@tasanuma Ok, I'll try to commit to 3.3, thanks for the review and merge. |
singer-bin
added a commit
to singer-bin/hadoop
that referenced
this pull request
Apr 10, 2022
tasanuma
pushed a commit
that referenced
this pull request
Apr 11, 2022
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: HDFS-16457.
Make fs.getspaceused.classname reconfigurable