-
Notifications
You must be signed in to change notification settings - Fork 8.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HADOOP-13144. Enhancing IPC client throughput via multiple connections per user #4542
Conversation
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
client = ProtobufRpcEngine2.getClient(newConf); | ||
assertEquals(2, client.getConnectionIds().size()); | ||
} catch (ServiceException e) { | ||
e.printStackTrace(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably want to surface this.
null); | ||
|
||
// create a server with two handlers | ||
server = setupTestServer(newConf, 2); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could create it here:
Server server = setupTestServer(newConf, 2);
firstProxy.ping(null, newEmptyRequest()); | ||
secondProxy.ping(null, newEmptyRequest()); | ||
|
||
client = ProtobufRpcEngine2.getClient(newConf); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For readability maybe we can do:
Client client2 = ProtobufRpcEngine2.getClient(newConf);
assertEquals(2, client2.getConnectionIds().size());
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRpcBase.java
Show resolved
Hide resolved
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRpcBase.java
Show resolved
Hide resolved
Thanks @goiri for your review. I have updated this patch, please help me reivew it. Thanks And @Hexiaoqiao @ayushtkn Can you help me to review this patch? |
🎊 +1 overall
This message was automatically generated. |
@ZanderXu Thanks for pushing is forward. It makes sense. |
Description of PR
JIRA ID: HADOOP-13144
The generic IPC client (org.apache.hadoop.ipc.Client) utilizes a single connection thread for each ConnectionId. The ConnectionId is unique to the connection's remote address, ticket and protocol. Each ConnectionId is 1:1 mapped to a connection thread by the client via a map cache.
The result is to serialize all IPC read/write activity through a single thread for a each user/ticket + address. If a single user makes repeated calls (1k-100k/sec) to the same destination, the IPC client becomes a bottleneck.