Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support accessing OSS with proxy #18139

Merged
merged 2 commits into from
Sep 13, 2023
Merged

Support accessing OSS with proxy #18139

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b74a054
support accessing OSS with proxy
Jackson-Wang-7 Sep 13, 2023
3d5c21a
support accessing OSS with proxy
Jackson-Wang-7 Sep 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions dora/core/common/src/main/java/alluxio/conf/PropertyKey.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1411,6 +1411,29 @@ public String toString() {
.setConsistencyCheckLevel(ConsistencyCheckLevel.WARN)
.setScope(Scope.SERVER)
.build();
public static final PropertyKey UNDERFS_OSS_PROTOCOL =
stringBuilder(Name.UNDERFS_OSS_PROTOCOL)
.setAlias("alluxio.underfs.oss.protocol")
.setDefaultValue("http")
.setDescription("The protocol for OSS endpoint, by default http.")
.setConsistencyCheckLevel(ConsistencyCheckLevel.WARN)
.setScope(Scope.SERVER)
.build();
public static final PropertyKey UNDERFS_OSS_PROXY_HOST =
stringBuilder(Name.UNDERFS_OSS_PROXY_HOST)
.setAlias("alluxio.underfs.oss.proxy.host")
.setDescription("The proxy host address for OSS connection, if any.")
.setConsistencyCheckLevel(ConsistencyCheckLevel.WARN)
.setScope(Scope.SERVER)
.build();
public static final PropertyKey UNDERFS_OSS_PROXY_PORT =
intBuilder(Name.UNDERFS_OSS_PROXY_PORT)
.setAlias("alluxio.underfs.oss.proxy.port")
.setDefaultValue(0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it need a default? port 0 means assign dynamically right? So I'm afraid it's not the best default to use? I think you will need to configure this anyway (there's an existing port the proxy uses, instead of for the alluxio proc to allocate). So maybe no need for a default?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the port is 0(or not > 0), we think the proxy configuration is not enabled, so we don't set proxy in the oss config.

.setDescription("The proxy port for OSS connection, if any.")
.setConsistencyCheckLevel(ConsistencyCheckLevel.WARN)
.setScope(Scope.SERVER)
.build();
public static final PropertyKey UNDERFS_OSS_STS_ECS_METADATA_SERVICE_ENDPOINT =
stringBuilder(Name.UNDERFS_OSS_STS_ECS_METADATA_SERVICE_ENDPOINT)
.setAlias("alluxio.underfs.oss.sts.ecs.metadata.service.endpoint")
Expand Down Expand Up @@ -7191,6 +7214,9 @@ public static final class Name {
public static final String UNDERFS_OSS_SOCKET_TIMEOUT = "alluxio.underfs.oss.socket.timeout";
public static final String UNDERFS_OSS_ECS_RAM_ROLE = "alluxio.underfs.oss.ecs.ram.role";
public static final String UNDERFS_OSS_RETRY_MAX = "alluxio.underfs.oss.retry.max";
public static final String UNDERFS_OSS_PROTOCOL = "alluxio.underfs.oss.protocol";
public static final String UNDERFS_OSS_PROXY_HOST = "alluxio.underfs.oss.proxy.host";
public static final String UNDERFS_OSS_PROXY_PORT = "alluxio.underfs.oss.proxy.port";
public static final String UNDERFS_OSS_STS_ECS_METADATA_SERVICE_ENDPOINT =
"alluxio.underfs.oss.sts.ecs.metadata.service.endpoint";
public static final String UNDERFS_OSS_STS_ENABLED = "alluxio.underfs.oss.sts.enabled";
Expand Down
73 changes: 73 additions & 0 deletions dora/underfs/oss/src/main/java/alluxio/underfs/oss/OSSUnderFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import alluxio.Constants;
import alluxio.PositionReader;
import alluxio.conf.AlluxioConfiguration;
import alluxio.conf.Configuration;
import alluxio.conf.PropertyKey;
import alluxio.retry.RetryPolicy;
import alluxio.underfs.ObjectUnderFileSystem;
Expand All @@ -29,6 +30,7 @@
import com.aliyun.oss.OSS;
import com.aliyun.oss.OSSClientBuilder;
import com.aliyun.oss.ServiceException;
import com.aliyun.oss.common.comm.Protocol;
import com.aliyun.oss.model.AbortMultipartUploadRequest;
import com.aliyun.oss.model.DeleteObjectsRequest;
import com.aliyun.oss.model.DeleteObjectsResult;
Expand Down Expand Up @@ -380,9 +382,80 @@ public static ClientBuilderConfiguration initializeOSSClientConfig(
ossClientConf.setConnectionTTL(alluxioConf.getMs(PropertyKey.UNDERFS_OSS_CONNECT_TTL));
ossClientConf.setMaxConnections(alluxioConf.getInt(PropertyKey.UNDERFS_OSS_CONNECT_MAX));
ossClientConf.setMaxErrorRetry(alluxioConf.getInt(PropertyKey.UNDERFS_OSS_RETRY_MAX));
if (isProxyEnabled(alluxioConf)) {
String proxyHost = getProxyHost(alluxioConf);
int proxyPort = getProxyPort(alluxioConf);
ossClientConf.setProxyHost(proxyHost);
ossClientConf.setProxyPort(proxyPort);
ossClientConf.setProtocol(getProtocol());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a log here if proxy is used (so you know)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea, done

LOG.info("the proxy for OSS is enabled, the proxy endpoint is: {}:{}", proxyHost, proxyPort);
}
return ossClientConf;
}

private static boolean isProxyEnabled(AlluxioConfiguration alluxioConf) {
return getProxyHost(alluxioConf) != null && getProxyPort(alluxioConf) > 0;
}

private static int getProxyPort(AlluxioConfiguration alluxioConf) {
int proxyPort = alluxioConf.getInt(PropertyKey.UNDERFS_OSS_PROXY_PORT);
if (proxyPort >= 0) {
return proxyPort;
} else {
try {
return getProxyPortFromSystemProperty();
} catch (NumberFormatException e) {
return proxyPort;
}
}
}

private static String getProxyHost(AlluxioConfiguration alluxioConf) {
String proxyHost = alluxioConf.getString(PropertyKey.UNDERFS_OSS_PROXY_HOST);
if (proxyHost != null) {
return proxyHost;
} else {
return getProxyHostFromSystemProperty();
}
}

private static Protocol getProtocol() {
String protocol = Configuration.getString(PropertyKey.UNDERFS_OSS_PROTOCOL);
return protocol.equals(Protocol.HTTPS.toString()) ? Protocol.HTTPS : Protocol.HTTP;
}

/**
* Returns the Java system property for proxy port depending on
* {@link #getProtocol()}: i.e. if protocol is https, returns
* the value of the system property https.proxyPort, otherwise
* returns value of http.proxyPort. Defaults to {@link this.proxyPort}
* if the system property is not set with a valid port number.
*/
private static int getProxyPortFromSystemProperty() {
return getProtocol() == Protocol.HTTPS
? Integer.parseInt(getSystemProperty("https.proxyPort"))
: Integer.parseInt(getSystemProperty("http.proxyPort"));
}

/**
* Returns the Java system property for proxy host depending on
* {@link #getProtocol()}: i.e. if protocol is https, returns
* the value of the system property https.proxyHost, otherwise
* returns value of http.proxyHost.
*/
private static String getProxyHostFromSystemProperty() {
return getProtocol() == Protocol.HTTPS
? getSystemProperty("https.proxyHost")
: getSystemProperty("http.proxyHost");
}

/**
* Returns the value for the given system property.
*/
private static String getSystemProperty(String property) {
return System.getProperty(property);
}

@Override
protected InputStream openObject(String key, OpenOptions options, RetryPolicy retryPolicy)
throws IOException {
Expand Down
Loading