Session requests are ~70% slower in version 2.26.0

[omermizr]

When using a session, requests are slower than in previous versions.
My metrics indicate a performance degradation, and after some testing I suspect this is the root cause.

Expected Result

Same performance as in version 2.25

Actual Result

When I use a session, requests are about 70% slower than in previous versions.
This seems to be the PR that caused the regression: #5681
I removed the rebuild_proxies call locally and performance shot back up.
I ran a benchmark on 1000 requests making a real-world request with the same session.
In version 2.25.1 it took ~13s, in version 2.26.0 it takes ~23s.

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "charset_normalizer": {
    "version": "2.0.3"
  },
  "cryptography": {
    "version": "3.4.7"
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.10"
  },
  "platform": {
    "release": "5.4.0-1055-azure",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "101010bf",
    "version": "19.1.0"
  },
  "requests": {
    "version": "2.26.0"
  },
  "system_ssl": {
    "version": "101000cf"
  },
  "urllib3": {
    "version": "1.26.6"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

[omermizr]

I dug a little deeper and it looks like all the time is spent inside getproxies_environment, here: requests.utils.proxy_bypass:781.
We are deployed in a k8s cluster and have a lot of environment variables (around ~530 ATM), so it makes sense that iterating these would take time.
I cached getproxies_environment's result and performance is back up.

Note: My OS is Ubuntu 18.04.

[bityob]

Thanks @omermizr for finding this issue.

In this gist I have a full repro for this latency issue, comparing requests 2.26 vs 2.25.1 packages.

We can see that when we have 500 environment variables, there is a huge difference between the versions.

But without any variables, there is no difference at all.

See here the full details and how to run it by yourself -
https://gist.github.com/bityob/02b1a823b478ad74a8440b86f95e13a6/

[olka]

Any updates on this one?
Some hints how to workaround it?
Our application spends 80% of time in decoding huge JSON payloads and rebuilding list of proxies 😅

[ykharko]

Hi guys. Do you know when in theory this issues might be solved and when new version of the library might be released?
Do you have any plans for that or smth like that?

[nateprewitt]

We expect the current fix to go out with the 2.27.0 release which is currently scheduled for this coming Monday. There is still an edge case where we may see a slow down in environments with a large number of proxies available. The long term recommendation for this is passing the proxies parameter directly with the call to circumvent this look up. We unfortunately don't have a better way to handle this since it's a fundamental problem of the search functionality in the standard library.

Resolving now that the fixes have been merged for release.