Openvds azure manager thread count #285
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In 3.4.4 open-vds introduced a way to choose number of threads used to download data with Azure. From experiments it might be better to decide on number of threads per file, but - number of threads is set through openvds environment variable OPENVDS_AZURE_IOMANAGER_THREAD_COUNT, so one can't really define this value per request - there is some fluctuation in the results - there doesn't seem to be extremely significant difference when larger-than-desired number of threads is chosen - whatever formula we come up with would be heuristics anyway So for simplicity variable is defined for the whole server instance. Experiments have shown that best value lies somewhere between ncores and ncores x 4. It seems like with x4 cores we get best speed, but random slower times start to pop up.
For some reason openvds fails to build in debug mode when curl and azure presigned io managers are disabled. In order to keep compilation arguments the same for Release and Debug, enable curl and azure presigned builds in all cases. libuv library is dependency of newly reactivated openvds dependencies.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Master failing on memory tests: tried to investigate why debug build fails myself, also failed, so submitted issue to openvds. They don't have resources to work on it, so reverting.