Improve rpc_soak and channel_soak test to cover streaming #11687
Conversation
zbilun
commented
Nov 13, 2024
•
zbilun
commented
- PTAL @apolcyn
- The clientCompressedStreaming and serverCompressedStreaming cases are not included in this version of the PR but will be added if necessary.
- In this design, the large_unary soak test case will continue to use the blocking API, while the streaming test cases will use the non-blocking API.
- A new test type parameter has been introduced to the performSoakTest function.
- Please review the POC code, and we can discuss the design further, thanks!
…ing soak_num_threads Flag
…, channel creation logic, and refactor thread body for performSoakTest
…d simplify thread result aggregation
…edException. Update the ThreadResults data type.
| return new SoakIterationResult(TimeUnit.NANOSECONDS.toMillis(elapsedNs), status); | ||
| } | ||
|
|
||
| private SoakIterationResult performOneSoakIterationPingPong( |
There was a problem hiding this comment.
I'm not sure how much benefit there will really be to running the different RPC types (client streaming, server streaming, etc.) in a loop.
The code paths and behaviors exercised are going to be very similar to the unary based soak tests we already have.
Here is a straw man idea for what I think might be useful here:
- using a long-lived stream
- start the stream once (per thread). On each soak iteration, send one message and receive one message.
- If/when the stream fails, indicate it in the log, but otherwise restart the stream and continue on the new one.
This would provide us a new dimension of test coverage that we don't currently have much of (long lived RPCs).
There was a problem hiding this comment.
Alex, thanks for the suggestion! I have discussed with Feng. He said, “For completeness, we should cover all these variations of RPCs, but I don’t see how sending a message out and back can construct a long-lived stream. Usually, long-lived streams last for hours, and covering long-lived streams is not part of the plan.”
So, seems he definitely wants to ensure we cover all RPC types. My thought is we can handle the long-lived RPCs in a separate set of tests, not as part of the current soak tests. We can definitely consider this in more detail and plan it out for a future PR. Let me know your thoughts, and happy to discuss further!
There was a problem hiding this comment.
“For completeness, we should cover all these variations of RPCs, but I don’t see how sending a message out and back can construct a long-lived stream. Usually, long-lived streams last for hours, and covering long-lived streams is not part of the plan.”
My idea here is not to send a message out and back once. Instead, it's to keep sending messages out and back on the same stream for as long as possible. By setting soak iterations and soak_min_time_ms_between_rpcs, you can make these clients do a fixed QPS for a fixed time (e.g. 10 QPS for 1 hour).
So, seems he definitely wants to ensure we cover all RPC types.
Our integration test matrix is already huge, and these tests are expensive to maintain generally speaking. I'm not excited about adding this for the sake of completeness, unless there's a very strong reason that I'm missing. It seems like these tests will overlap a lot with the existing unary based tests, so it doesn't seem like there would be much bang for the buck with these additional tests.
