Update mp for fast pull #201
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
commit caaee36 Merge: b4eb584 63183d1 Author: Kenneth <[email protected]> Date: Sat Mar 9 13:55:30 2024 -0800 Merge pull request #203 from kyocum/fix/new-lib-versions Update library dependencies isort / black commit b4eb584 Merge: 7d6a1d5 39cb9a5 Author: Kenneth <[email protected]> Date: Fri Mar 8 17:04:49 2024 -0800 Merge pull request #202 from pmdaly/restrict-python-version Restrict python version commit 39cb9a5 Author: pmdaly <[email protected]> Date: Fri Mar 1 12:23:48 2024 -0800 Update tox.ini commit d96fdc8 Author: pmdaly <[email protected]> Date: Fri Mar 1 12:23:06 2024 -0800 enforce a range in setup
* enforce a range in setup * Update tox.ini * removing six * bumping to pandas 2.x * fix comment * isort / black --------- Co-authored-by: pmdaly <[email protected]> Co-authored-by: Ken Yocum <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR removes the use of multiprocessing in favor of multithreading. There were instances where using multiprocessing was problematic in containers (e.g., getting an accurate cpu count as well as memory issues). In all cases we were more interested in concurrency than parallelism (using multithreading for I/O for managing keys on s3) .
This is a relatively major change internally for many of the s3 related functions, but it should not affect common users. Users can still set the env variable
DISDAT_CPU_COUNTand the system will create thread pools with exactly that number of threads. Otherwise it will use the system cpu count and set threads = 2* that amount.In addition, this removes
six.moves.urllibdependencies and removes a number of print statements in lieu of logger.info() calls (useful for those using the API instead of the CLI).