Manually set Proxy-Authorization header is always removed in 2.26

[chris-sawarzynski]

In versions 2.25 it was possible to add Proxy-Authorization header manually.
Starting from requests 2.26 this header is removed from request if set manually.
When this could be a problem (real life example):
If you use requests library to programatically access service hidden behind Google Identity Aware Proxy that is protected with Basic Auth you have to provide token in Proxy-Authorization header:
https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_proxy-authorization_header

Expected Result

Manually provided Proxy-Authorization header should be kept intact, or there should be an obvious way to make this work without hackng library internals

Actual Result

Proxy-Authorization header is removed from request

Reproduction Steps

import requests

s = requests.Session()
s.headers.update({"Accept": "application/json", "Proxy-Authorization":"Bearer XXX"})
s.auth = ("login", "password")

resp = s.get('http://localhost:3000/answer')
print(resp.request.headers)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.3"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.2"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.8"
  },
  "platform": {
    "release": "3.10.0-957.1.3.el7.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.26.0"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.26.6"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

[nateprewitt]

Hey @chris-sawarzynski, thanks for the report. I believe this is a byproduct of #5681. This change is now forcing a rebuild on the proxies to resolve parameters from both the Session and Request. Part of the rebuild involves stripping existing Proxy-Authorization headers and attempting to rebuild them from the url.

I think this was potentially an oversight in the recent change. I'm still trying to work out why we're stripping this off in the first place instead of only updating it pending contradictory information in the URL. I couldn't find an answer in the original PR (#1951) but we're going pretty far back in time. We've co-opted what was originally a method intended for redirects into the default flow. We're stripping general auth info to avoid potentially leaking sensitive information to a new domain, but that's not relevant in the first call. We may need to break up rebuild_proxies into some smaller components to allow us this to work correctly, or consider a different approach for merging.

@sigmavirus24 I don't know if you have any thoughts about this. I'm thinking we consider this a regression, though, and it may need some follow up work to untangle the auth sanitizing from the configuration resolution.

[sigmavirus24]

Yeah this part of the code is a bit of a mess frankly. We'll definitely need some careful refactoring here

[Tribunny]

When will it be settled?

[stanislavlevin]

The self-test of mod_auth_gssapi is also affected by this issue:
https://github.com/gssapi/mod_auth_gssapi/blob/master/tests/t_spnego_proxy.py

[nateprewitt]

We've merged #5924 which should address this in the next minor release of Requests. I'm currently tracking for a date before EoY for the 2.27.x milestone. Resolving this for now, thanks everyone.