Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix HTTP performance regression #1265

Merged
merged 1 commit into from
Oct 19, 2024
Merged

Conversation

reidsunderland
Copy link
Member

This fixes the performance regression in the HTTP transfer class.

Instead of calling build_opener (which is very slow) for every __open__, it is only done in connect now.

I also made another change, which removes the setting of self.connected = False in the __open__ method. With that there, it forces a "re-connect" (and build_opener call) every time __open__ was called. Now connect gets called only when check_is_connected returns False, which will mainly happen if there is a change to the "sendTo" URL (if it was downloading from google.com, then we switch to downloading from example.com, it will re-connect).

If we have a feed that downloads from many different domains/sources, we would still see the performance issue, because of the re-connecting. I left a note in the comments saying that I think build_opener could actually only be done even less frequently, in __init__. But I was worried that that was too large of a change to make.

This change is just one commit, so it should be easily cherry-pickable if we want to make a 3.00.55post2 release

Copy link

Test Results

249 tests   248 ✅  1m 29s ⏱️
  1 suites    1 💤
  1 files      0 ❌

Results for commit 0fe61aa.

@petersilva petersilva merged commit d4729aa into development Oct 19, 2024
46 of 59 checks passed
@reidsunderland reidsunderland deleted the issue1261_http_cpu_usage branch November 20, 2024 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants