401 "Request did not validate missing authorization header" when header present

[choiRyan]

This might be a StackOverflow-type question but I'm constantly getting 401 Unauthorized, errcode 109 (Invalid authentication) and message: "Request did not validate missing authorization header". I'm using VAPID headers to a Mozilla push endpoint as suggested in #30

I use this line to send the notification:
x = pywebpush.WebPusher(subscription_info).send(data, vapid_headers, 60*60)

The authorization header appears to be present (logged x.request.headers):
{ 'Accept': '*/*', 'authorization': 'WebPush eyJ0eXAiOi...', 'crypto-key': "p256ecdsa=b'Nz2H1lIDt...7b7OAfMA-bwZP6qk_WonOnahzw7iDx4rA';keyid=p256dh;dh=BCvGo_YbulU_xUnT-U...", 'ttl': '3600', 'Accept-Encoding': 'gzip, deflate', 'content-encoding': 'aesgcm', 'encryption': 'keyid=p256dh;salt=azMk5vK0...', 'User-Agent': 'python-requests/2.11.1', 'Connection': 'keep-alive', 'Content-Length': '370' }

The lengthy details (sorry) below show what I do:
subscription_info = {'endpoint': 'https://updates.push.services.mozilla.com/wpush/v2/gAA...', 'keys': {'p256dh': 'BJv7nI...p31z4Qvt86+oeQMfZk5Nb/fhko...wmPXvo99A=', 'auth': '2Y4...+WA=='}}

data = json.loads({'abc': 'def'})

vapid_headers = _make_vapid_headers(endpoint) (endpoint is from subscription_info)

Relevant definitions below (and possibly where an error lies, as I attempted to fix an issue I got in python3 running the original code from https://blog.mozilla.org/services/2016/04/04/using-vapid-with-webpush/ )

# generated with openssl ecparam -name prime256v1 -genkey -noout -out vapid_private.pem
VAPID_PRIVATE_KEY_PATH = 'vapid_private.pem' 
priv_key = open(VAPID_PRIVATE_KEY_PATH, 'r', encoding='utf-8').read()
VAPID_PRIVATE_KEY_CONTENTS = SigningKey.from_pem(priv_key)  # from ecdsa import SigningKey

# As seen in https://blog.mozilla.org/services/2016/04/04/using-vapid-with-webpush/
def _make_jwt(header, claims, key):
    vk = key.get_verifying_key()
    jwt = jws.sign(
        claims,
        key,
        algorithm=header.get('alg', 'ES256')).strip('=')
    raw_public_key = b'\x04' + vk.to_string()  # need to work with bytes
    public_key = base64.urlsafe_b64encode(raw_public_key).strip(b'=')
    return (jwt, public_key)

def _make_vapid_headers(endpoint):
    header = {'typ': 'JWT', 'alg': 'ES256'}
    origin = endpoint.split('.com')[0] + '.com'
    claims = {
        'aud': origin,
        'exp': '{}'.format(int(time()) + 60*60*12),  # import time from time
        'sub': 'mailto:[email protected]'
    }

    my_key = settings.VAPID_PRIVATE_KEY_CONTENTS
    (jwt, public_key) = _make_jwt(header, claims, my_key)

    headers = {
        'Authorization': 'WebPush {}'.format(jwt),
        'Crypto-Key': 'p256ecdsa={}'.format(public_key)
    }
    return headers

I followed the instructions for getting the public key to send to the frontend from https://blog.mozilla.org/services/2016/08/23/sending-vapid-identified-webpush-notifications-via-mozillas-push-service/ where I removed -----BEGIN PUBLIC KEY, made the key urlsafe, etc.

[jrconlin]

Hi,

Looks like one of the problems is that a printed byte string is being sent as the crypto-key.
From above I see:

"p256ecdsa=b'Nz2H1lIDt...7b7OAfMA-bwZP6qk_WonOnahzw7iDx4rA';keyid=p256dh;dh=BCvGo_YbulU_xUnT-U...",

(Note the b'...')
I'm not sure if that's your code or mine that's doing it. You're running on Python3, may I ask what subversion? I'd like to see if I can replicate it and add a test to see if I can catch that.

I'll take a look at the library code as well and see if I broke something with the latest push.

[jrconlin]

AH! I think I see the problem, and it's in the py_vapid library. THANK YOU!

[choiRyan]

Thanks for the quick response, that does look like a problem - note that I did change the code given in the mozilla link from raw_public_key = "\x04" + vk.to_string() to raw_public_key = b'\x04' + vk.to_string(), I think that is why that b'...' showed up.

Also, I'm on Python 3.5.2, and now that you mention it it does seem like a problem on my end instead of the library (hopefully!)

Changed public_key = base64.urlsafe_b64encode(raw_public_key).strip(b'=') to public_key = base64.urlsafe_b64encode(raw_public_key).strip(b'=').decode('ascii'), still getting the same error (headers below again)

{'encryption': 'keyid=p256dh;salt=cd4...', 'content-encoding': 'aesgcm', 'Accept': '*/*', 'User-Agent': 'python-requests/2.11.1', 'Content-Length': '370', 'ttl': '3600', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'crypto-key': 'p256ecdsa=BDc....;keyid=p256dh;dh=BMf1...', 'authorization': 'WebPush eyJh...'}

I get the feeling something is up with my keys, though. Compared to the ones generated in the node library, the keys I have aren't the same length, and when trying to use keys generated by node in the python libraries, I get a wrong key length error.

edit: Oops didn't see your update above. No problem (?) :)

[jrconlin]

Yeah, I've seen that before. The problem is that some frameworks use different encodings for the public key types.

(Probably more info than you want: The ECDSA public key is a pair of 32byte octets, thus the 64 bytes. It's prefixed by a "RAW" key flag (the \x04) Some frameworks use a different key storage mechanism that encodes the key into a totally different format that's a pain in the butt to extract, unless you run it though a ASN1 library. Basically, if your key isn't 64 bytes, or 65 with the first byte being \x04, it's probably ASN1 and needs to be transcoded. Sadly: not super happy fun times, but possible. I'll try to add some notes tomorrow if that makes things easier for you.)

The spec requires that the public key be in RAW DER format (`\x04' + 2*32 bytes).

More fun news! The VAPID format is changing soon. Draft02 moves stuff as a prefix for the AUTH key.

No, really, feel free to use a library to deal with this nonsense.

You have a life. You should spend more time enjoying it.

(Well, unless you find this sort of sadness fun, in which case, feel free to pitch in over on the VAPID libs)

[choiRyan]

Sounds good (and yeah, "excited" to hear Vapid 02 is on its way soon, ha).

I'll check out py_vapid's releases to sign that header (noticed you've already pushed some changes regarding this - can confirm python3 is busted). Thanks for all your work! Closing this issue now.

[choiRyan]

@jrconlin Sorry to bump this again, but my existing problem wasn't resolved with latest version of py_vapid (0.7.1).

The issue I have currently is how to use the public key generated by py_vapid as the client's applicationServerKey, which is supposed to be a length 65 UInt8Array after converting from the original base64 form.
(if true, that is - I thought I had a pretty good grasp of what's needed, but despite many hours, nothing has worked).

I've been using the latest py_vapid to generate my keys and sign claims (and pywebpush to send notifications). It seems straightforward and I'm following the steps, but the public key always comes up as invalid on Chrome. (Fails the public key UInt8Array length check for validity)

I tried this for changing the public key:
I noted your last comment and looked at the tests, found T_PUBLIC_RAW, and likewise removed the 36 character prefix from the middle content from public_key.pem generated by vapid.save_public_key(). I did not make the urlsafe changes since I just pasted the code into a constant on the frontend. I figured that the result was the required 2*32 byte key, and prepended 0x04 to the length 64 UInt8Array to round out the length=65 requirement on the browser before registering with the key. But I ended up getting the same authorization error when I sent an update to the Push Service later from the backend (same error as first post).

I generated a key for example, to go over the steps from the beginning:
ex: the full public key MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEfLyLOsmUiEFalpQ3C4IvTV7SPLT2k8/Fzx16nChKP+9K+7bUg6pUKEySFHA4n7pDRYygQDUYBtUupMWfgnwJuw==

Becomes
fLyLOsmUiEFalpQ3C4IvTV7SPLT2k8/Fzx16nChKP+9K+7bUg6pUKEySFHA4n7pDRYygQDUYBtUupMWfgnwJuw==.

I subscribe from the client as below:

const PUBLIC_KEY = 'fLyLOsmUiEFalpQ3C4IvTV7SPLT2k8/Fzx16nChKP+9K+7bUg6pUKEySFHA4n7pDRYygQDUYBtUupMWfgnwJuw==';
function b64ToUint8ArrayPrepend04(base64String) {
        const rawData = window.atob(base64String);
        const outputArray = new Uint8Array(65);
        outputArray[0] = 0x04;
        for (let i = 1; i < rawData.length; ++i) {
            outputArray[i] = rawData.charCodeAt(i);
        }
        console.log(outputArray); //  result: Uint8Array [ 4, 188, 139, 58... ]; length 65
        return outputArray;
    }
let applicationServerKey = b64ToUint8ArrayPrepend04(PUBLIC_KEY); 
...
registration.pushManager.subscribe({
    userVisibleOnly: true,
    applicationServerKey: applicationServerKey
}); // browser: OK, sends to server to store subscription details.

Then on the backend I send an update to the push API providers as below:

def send_web_notification(user_web_app_session, data):
    """
    :param user_web_app_session: subscription details from frontend
    :param data: string
    :return: -
    """
    data = 'abc'   # for testing
    endpoint = user_web_app_session.get('endpoint')
    auth = user_web_app_session.get('auth_secret')
    p256 = user_web_app_session.get('key')
    aud = user_web_app_session.get('audience')  # I checked, all these fields are filled

    subscription_info = {
        'endpoint': endpoint,
        'keys': {
            'auth': auth,
            'p256dh': p256
        }
    }

    claims = {
        'aud': aud,
        'sub': settings.VAPID_CLAIMS_SUB
    }

    vapid_lib = py_vapid.Vapid(private_key_file=settings.VAPID_PRIVATE_KEY_PATH)
    vapid_headers = vapid_lib.sign(claims)
    x = WebPusher(subscription_info).send(data, vapid_headers, ttl=60*60*12)

Log of x.request.headers:

{
'Content-Length': '21',
'crypto-key': 'p256ecdsa=fLyLOsmUiEFalpQ3C4IvTV7SPLT2k8_Fzx16nChKP-9K-7bUg6pUKEySFHA4n7pDRYygQDUYBtUupMWfgnwJuw;keyid=p256dh;dh=BJANMmQ[...]Yduj_QJtK8yYF6NJ148', 
'content-encoding': 'aesgcm', 
'ttl': '43200', 
'encryption': 'keyid=p256dh;salt=BpD5X_88-Ip[...]', 
'authorization': 'WebPush eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJz[...]NDN9.kG_fU8n[...]Q-lvWBpf_UDGmezfsbboKE60AvlANBTZJPg'
}

The response from a Chrome/FCM endpoint:
Error code 400, UnauthorizedRegistration, reason: UnauthorizedRegistration

The response from a Firefox endpoint:
Error code 401, Unauthorized, errno 109, message: "Request did not validate missing authorization header"

[jrconlin]

Unfortunately, converting between a public PEM and a raw key can involve more than just stripping data. Because encryption, there are several different formats that keys can be stored in. Just clipping out the last 64 octets off the key is usually "OK" in the same way that using duct tape to fix your rear bumper is "OK". It's really far better to decode the PEM using ASN1 to get the values, but that's overkill for now.

That said, there are a few things a play here that may be tripping you up. Chrome's WebPush requires a few more steps than Firefox does, and has a different auth mechanism.

In short you don't use VAPID with Chrome's WebPush, you use FCM.
https://github.com/GoogleChrome/samples/tree/gh-pages/push-messaging-and-notifications
provides an example of how to send messages via Chrome's Webpush/FCM

So, for your example, you'd need to create a manifest.json file including the "gcm_sender_id" that matches whatever your Chrome FCM project's Sender ID is, and then send the notification with

curl --header "Authorization: key=_YourLongFCMServerKey_" \
--header "Content-Type: application/json" \
-d "{\"registration_ids\":[\"_RegistrationIds_\"]}"

The RegistrationIds are provided from the last segment of the endpointURL (Probably should note the extra conditions for older versions of Chrome)

I've tried out the code sample above and have been able to send a message through to Chrome. I'm a bit sad that they're taking this approach rather than VAPID, but then VAPID isn't yet formalized as a standard and FCM already works, so I'm not going to ding them about it.

Now, as for the 401/109 you're seeing for firefox, I'm not sure I know why that failed. It may be that you're not properly encoding the public key (again, just stripping off octets may not do what you think).

[choiRyan]

Got it, I'll check out more formal procedures than removing octets.

I thought the FCM-proprietary method was for older versions (Chrome 53 and below) and that recent versions of chrome supported Web Push just like Firefox, from their post here: https://developers.google.com/web/updates/2016/07/web-push-interop-wins
But this not being the case would explain FCM issues completely, though.

With VAPID you no longer need to sign up for an account with GCM to use push in Chrome and you can use the same code path for subscribing a user and sending a message to a user in both Chrome and Firefox. Both are following the standards.

What you need to bear in mind is that in Chrome 51 and before, Opera for Android and Samsung browser you'll still need to define the gcm_sender_id in your web app manifest and you'll need to add the Authorization header to the FCM endpoint that will be returned.

VAPID provides an off ramp from these proprietary requirements. If you implement VAPID it'll work in all browsers that support web push. As more browsers support VAPID you can decide when to drop the gcm_sender_id from your manifest.

EDIT: I think there may be an issue with python-jose when working with jws on python 3.5. Looking into it...

[jrconlin]

Huh, I had not seen that post. Ok, let me experiment a bit and see if I can figure out what's going on there.

thanks!

[jrconlin]

Welp, if it's any help, I'm now getting the ever helpful 400 UNAUTHORIZED errors when I try to send VAPID requests through on chrome.

I pulled the public key from my py_vapid library and was able to request a restricted endpoint, but when I try to submit a vapid header that is signed using that key, it bounces.

[choiRyan]

Yeah, it seems like some other people had this problem, reading this is where I thought something could be up with the signature: http://stackoverflow.com/questions/39336523/webpushvapid-request-fails-with-400-unauthorizedregistration

Edit: I tried my code with the example keys in test_vapid.py copied over and it didn't work with same errors, which is highly confusing and makes me think there's something wrong on my end. Not sure what could be going wrong - All tests passed on py_vapid on my setup too

[jrconlin]

OBE by #53 (Thanks!)