Adding timeouts to wait methods, to never allow for infinite wait() state. #198
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
If you configure the TCP stack of your Jenkins Controller and Jenkins Agents to a proper value for TIME_WAIT, The connection will die and the threads. The default value in Linux is 7200 seconds, I usually recommend to set it to 120 seconds. |
|
BTW magic numbers are bad idea |
Add timeouts to StreamGobbler methods
Use existing DEFAULT_WAIT_TIMEOUT for timeout
Add DEFAULT_WAIT_TIMEOUT, and use it for wait() calls
|
Updated the all values to default 20mins of wait. Overridable by variable as was already used in the previous cases. |
|
The CI do not pass |
|
Fixed issues with DM_BOXED_PRIMITIVE_FOR_PARSING |
|
was root cause why this was needed related to: |
|
Anything that causes half closed connections cause the issue, this would mitigate some cases, probably not all. |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In a couple of important methods the Object.wait() is invoked without any timeout, which in some cases produces threads waiting forever. I've added default timeout of 30mins, in one case of 12h. I'm open to any other timeouts as long as it's not measured in days. It'd be better to have them as some variable as is already done in some cases with e.g.: DEFAULT_WAIT_TIMEOUT, but I left it as just plain values for now, to discuss about the value for the wait(), and then set it as variable.
See JENKINS-73575.
I believe that when something is wrong with the process it should communicate and not stay in the background wasting resources never to do anything again. This update may create some new issues with regards to problems with agents connectivity, where before Jenkins would just forever try to reconnect, even though the agent stopped existing, and now it would just raise error, as in any other case.
The tests for this would be incredibly hard to perform as the situations where such infinite wait() happens are very specific and I was not able to replicate them so far.
Submitter checklist