Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDFS-17565. EC: dfs.datanode.ec.reconstruction.threads should support… #6928

Open
wants to merge 7 commits into
base: trunk
Choose a base branch
from
+70 −2
Open

HDFS-17565. EC: dfs.datanode.ec.reconstruction.threads should support… #6928

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
14911e8
HDFS-17565. EC: dfs.datanode.ec.reconstruction.threads should support…
zhengchenyu Jul 8, 2024
68d39fd
remove necessary if condition
zhengchenyu Jul 9, 2024
e770648
fix
zhengchenyu Jul 12, 2024
07aeb55
fix
zhengchenyu Jul 12, 2024
ff31d28
fix
zhengchenyu Oct 30, 2024
20d6b9f
trigger ci
zhengchenyu Oct 31, 2024
12cd104
trigger ci
zhengchenyu Oct 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion ...fs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DISK_BALANCER_ENABLED_DEFAULT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DISK_BALANCER_PLAN_VALID_INTERVAL;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DISK_BALANCER_PLAN_VALID_INTERVAL_DEFAULT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_DEFAULT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_MAX_NUM_BLOCKS_TO_LOG_DEFAULT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_MAX_NUM_BLOCKS_TO_LOG_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_METRICS_LOGGER_PERIOD_SECONDS_DEFAULT;
Expand Down Expand Up @@ -374,7 +376,8 @@ public class DataNode extends ReconfigurableBase
DFS_DATANODE_DATA_TRANSFER_BANDWIDTHPERSEC_KEY,
DFS_DATANODE_DATA_WRITE_BANDWIDTHPERSEC_KEY,
DFS_DATANODE_DATA_READ_BANDWIDTHPERSEC_KEY,
DFS_DATANODE_SLOW_IO_WARNING_THRESHOLD_KEY));
DFS_DATANODE_SLOW_IO_WARNING_THRESHOLD_KEY,
DFS_DN_EC_RECONSTRUCTION_THREADS_KEY));

public static final String METRICS_LOG_NAME = "DataNodeMetricsLog";

Expand Down Expand Up @@ -740,6 +743,8 @@ public String reconfigurePropertyImpl(String property, String newVal)
return reconfDiskBalancerParameters(property, newVal);
case DFS_DATANODE_SLOW_IO_WARNING_THRESHOLD_KEY:
return reconfSlowIoWarningThresholdParameters(property, newVal);
case DFS_DN_EC_RECONSTRUCTION_THREADS_KEY:
return reconfStripedReconstructionParameters(property, newVal);
default:
break;
}
Expand Down Expand Up @@ -1079,6 +1084,22 @@ private String reconfSlowIoWarningThresholdParameters(String property, String ne
}
}

private String reconfStripedReconstructionParameters(String property, String newVal)
throws ReconfigurationException {
String result = null;
try {
if (property.equals(DFS_DN_EC_RECONSTRUCTION_THREADS_KEY)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this if condition? it would be always true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need this if condition? it would be always true?

For now, this if condition is not necessary. I wanted to use this method to handle all about the striped block configuration refresh. Sine only one configuration, I will remove this if condition.

int size = (newVal == null ? DFS_DN_EC_RECONSTRUCTION_THREADS_DEFAULT :
Integer.parseInt(newVal));
result = Long.toString(size);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are we doing here? converting newVal to Integer & then again converting into string? shouldn't we just check null & directly pass the string, rather than this conversion

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is special case, if the string passed is null, the default value will be used. So there's an extra conversion from int to string.

ecWorker.setStripedReconstructionPoolSize(size);
}
return result;
} catch (IllegalArgumentException e) {
zhengchenyu marked this conversation as resolved.
Show resolved Hide resolved
throw new ReconfigurationException(property, newVal, getConf().get(property), e);
}
}

/**
* Get a list of the keys of the re-configurable properties in configuration.
*/
Expand Down
19 changes: 19 additions & 0 deletions ...src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
*/
package org.apache.hadoop.hdfs.server.datanode.erasurecode;

import org.apache.hadoop.classification.VisibleForTesting;
import org.apache.hadoop.util.Preconditions;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.conf.Configuration;
Expand All @@ -37,6 +38,9 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;

import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_SLOW_IO_WARNING_THRESHOLD_KEY;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove unused import.

import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_KEY;

/**
* ErasureCodingWorker handles the erasure coding reconstruction work commands.
* These commands would be issued from Namenode as part of Datanode's heart beat
Expand Down Expand Up @@ -172,4 +176,19 @@ public void shutDown() {
public float getXmitWeight() {
return xmitWeight;
}

public void setStripedReconstructionPoolSize(int size) {
Preconditions.checkArgument(size > 0,
DFS_DN_EC_RECONSTRUCTION_THREADS_KEY + " should be greater than 0");
this.stripedReconstructionPool.setCorePoolSize(size);
this.stripedReconstructionPool.setMaximumPoolSize(size);
}

@VisibleForTesting
public int getStripedReconstructionPoolSize() {
int poolSize = this.stripedReconstructionPool.getCorePoolSize();
Preconditions.checkArgument(poolSize == this.stripedReconstructionPool.getMaximumPoolSize(),
"The maximum pool size should be equal to core pool size");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this isn't required, this looks like validating the ThreadPoolExecutor, that ain't a hadoop thing & isn't required here in the scope either

return poolSize;
}
}
33 changes: 33 additions & 0 deletions ...dfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeReconfiguration.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DISK_BALANCER_ENABLED_DEFAULT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DISK_BALANCER_PLAN_VALID_INTERVAL;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DISK_BALANCER_PLAN_VALID_INTERVAL_DEFAULT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_DEFAULT;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DN_EC_RECONSTRUCTION_THREADS_KEY;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNull;
Expand Down Expand Up @@ -948,4 +950,35 @@ public void testSlowIoWarningThresholdReconfiguration() throws Exception {
}
}

@Test
public void testDataNodeECReconstructionThreads() throws Exception {
for (int i = 0; i < NUM_DATA_NODE; i++) {
DataNode dn = cluster.getDataNodes().get(i);

// Verify DFS_DN_EC_RECONSTRUCTION_THREADS_KEY.
// Try invalid values.
LambdaTestUtils.intercept(ReconfigurationException.class,
"Could not change property dfs.datanode.ec.reconstruction.threads from "
+ "'8' to 'text'",
() -> dn.reconfigureProperty(DFS_DN_EC_RECONSTRUCTION_THREADS_KEY, "text"));
LambdaTestUtils.intercept(ReconfigurationException.class,
"Could not change property dfs.datanode.ec.reconstruction.threads from "
+ "'8' to '-1'",
() -> dn.reconfigureProperty(DFS_DN_EC_RECONSTRUCTION_THREADS_KEY, "-1"));
LambdaTestUtils.intercept(ReconfigurationException.class,
"Could not change property dfs.datanode.ec.reconstruction.threads from "
+ "'8' to '0'",
() -> dn.reconfigureProperty(DFS_DN_EC_RECONSTRUCTION_THREADS_KEY, "0"));

// Set value is 10.
dn.reconfigureProperty(DFS_DN_EC_RECONSTRUCTION_THREADS_KEY,
String.valueOf(10));
assertEquals(10, dn.getErasureCodingWorker().getStripedReconstructionPoolSize());

// Set default value.
dn.reconfigureProperty(DFS_DN_EC_RECONSTRUCTION_THREADS_KEY, null);
assertEquals(DFS_DN_EC_RECONSTRUCTION_THREADS_DEFAULT,
dn.getErasureCodingWorker().getStripedReconstructionPoolSize());
}
}
}
2 changes: 1 addition & 1 deletion hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSAdmin.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,7 @@ public void testDataNodeGetReconfigurableProperties() throws IOException, Interr
final List<String> outs = Lists.newArrayList();
final List<String> errs = Lists.newArrayList();
getReconfigurableProperties("datanode", address, outs, errs);
assertEquals(26, outs.size());
assertEquals(27, outs.size());
assertEquals(DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY, outs.get(1));
}

Expand Down
Loading