This repository has been archived by the owner on Dec 4, 2024. It is now read-only.
Use requests directly, instead of sseclient. #114
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brndnmtthws
force-pushed
the
drop-sseclient
branch
2 times, most recently
from
218cd8d to
3905f17
Compare
This will hopefully resolve some of the issues seen with SSE mode (see #35).
brndnmtthws
force-pushed
the
drop-sseclient
branch
from
3905f17 to
bb4ae4b
Compare
toofishes
pushed a commit
to toofishes/marathon-lb
that referenced
this pull request
Dec 14, 2016
Earlier issues d2iq-archive#35 and d2iq-archive#114 made things better in this department. However, when running marathon-lb with a large (400+ applications) marathon instance, there are still problems. These problems can be traced back to Python itself, unfortunately: http://stackoverflow.com/questions/21797753/efficiently-reading-lines-from-compressed-chunked-http-stream-as-they-arrive Python requests uses urllib under the covers, and there are implicit issues when 7.5 MB of JSON comes back on a single line, as we're seeing when certain events are emitted. These events are deployment_info and deployment_success at a minimum, there may be more. By switching to PycURL, as noted in the Stack Overflow post, we bypass this whole issue. We use an HTTP library that handles this particular edge case well, reducing CPU usage dramatically when a large event comes in. It also handles gzip compression, which means any 7.5 MB JSON dumps should shrink significantly. One unsolved problem remains here: the addition of DC/OS authentication support in d2iq-archive#285 is extremely tightly coupled to internal implementation details of the python requests module. This simply won't work with this code, and I have zero ability to fix or test it as we don't use DC/OS.
toofishes
pushed a commit
to toofishes/marathon-lb
that referenced
this pull request
Dec 28, 2016
Earlier issues d2iq-archive#35 and d2iq-archive#114 made things better in this department. However, when running marathon-lb with a large (400+ applications) marathon instance, there are still problems. These problems can be traced back to Python itself, unfortunately: http://stackoverflow.com/questions/21797753/efficiently-reading-lines-from-compressed-chunked-http-stream-as-they-arrive Python requests uses urllib under the covers, and there are implicit issues when 7.5 MB of JSON comes back on a single line, as we're seeing when certain events are emitted. These events are deployment_info and deployment_success at a minimum, there may be more. By switching to PycURL, as noted in the Stack Overflow post, we bypass this whole issue. We use an HTTP library that handles this particular edge case well, reducing CPU usage dramatically when a large event comes in. It also handles gzip compression, which means any 7.5 MB JSON dumps should shrink significantly.
toofishes
pushed a commit
to toofishes/marathon-lb
that referenced
this pull request
Jan 20, 2017
Earlier issues d2iq-archive#35 and d2iq-archive#114 made things better in this department. However, when running marathon-lb with a large (400+ applications) marathon instance, there are still problems. These problems can be traced back to Python itself, unfortunately: http://stackoverflow.com/questions/21797753/efficiently-reading-lines-from-compressed-chunked-http-stream-as-they-arrive Python requests uses urllib under the covers, and there are implicit issues when 7.5 MB of JSON comes back on a single line, as we're seeing when certain events are emitted. These events are deployment_info and deployment_success at a minimum, there may be more. By switching to PycURL, as noted in the Stack Overflow post, we bypass this whole issue. We use an HTTP library that handles this particular edge case well, reducing CPU usage dramatically when a large event comes in. It also handles gzip compression, which means any 7.5 MB JSON dumps should shrink significantly.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To address #35.