get_stream_url gives 403 with 'DEVICE_NOT_AUTHORIZED' for Mobileclient.FROM_MAC_ADDRESS

[stegzilla]

Been using everything fine until a few hours ago, when any call to get_stream_url returns 403 with 'X-Rejected-Reason': 'DEVICE_NOT_AUTHORIZED' when using Mobileclient.FROM_MAC_ADDRESS

I'm able to do everything but get_stream_url. Using an actual Android ID seems to work, so Google may have changed something to stop us using MAC addresses? Makes no difference if 2FA is enabled or not.

This is probably a dupe of #584, sorry.

[simon-weber]

Huh, that's the first I've heard of them actually introspecting device ids. Sounds like we may need to deprecate FROM_MAC_ADDRESS and make it easier for people to look up their existing device ids before logging in.

[stegzilla]

I just tried it again, and FROM_MAC_ADDRESS seems to be working now, so I don't really know what's happening.

I guess deprecating is probably a good idea, no idea when it will just stop working completely.

[simon-weber]

Weird. Could be that they're in the process of rolling the new change out (though in the past I've usually seen that around Fridays, not the start of a week).

[thebigmunch]

Or could just be Google servers being flaky as they have been so many times in the past : P

[fergyfresh]

I just got a similar problem to this.

stack trace is basically this

[DEBUG]	2018-05-25T00:23:41.272Z	e02cc2ce-5fb1-11e8-a4dc-637d90e1a294	https://mclients.googleapis.com:443 "GET /music/mplay?opt=hi&net=mob&pt=e&slt=1527207821033&sig=OMuDy_lx-OjkiflstlKqbnb1k9s&mjck=Tbm4qc5ka36wvdcifyfryjalqx4&hl=en_US&dv=0&tier=aa HTTP/1.1" 403 None
[2018-05-25 00:23:41,274] ERROR in app: Exception on /alexa/stream/Tbm4qc5ka36wvdcifyfryjalqx4 [GET]
Traceback (most recent call last):
File "/var/task/gmusicapi/protocol/shared.py", line 218, in perform
response.raise_for_status()
File "/var/task/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://mclients.googleapis.com/music/mplay?opt=hi&net=mob&pt=e&slt=1527207821033&sig=OMuDy_lx-OjkiflstlKqbnb1k9s&mjck=Tbm4qc5ka36wvdcifyfryjalqx4&hl=en_US&dv=0&tier=aa

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/task/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/var/task/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/var/task/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/var/task/flask/_compat.py", line 33, in reraise
raise value
File "/var/task/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/var/task/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/var/task/geemusic/controllers.py", line 40, in redirect_to_stream
stream_url = api.get_google_stream_url(song_id)
File "/var/task/geemusic/utils/music.py", line 148, in get_google_stream_url
return self._api.get_stream_url(song_id)
File "<decorator-gen-102>", line 2, in get_stream_url
File "/var/task/gmusicapi/utils/utils.py", line 293, in wrapper
return function(*args, **kw)
File "/var/task/gmusicapi/clients/mobileclient.py", line 388, in get_stream_url
return self._make_call(mobileclient.GetStreamUrl, song_id, device_id, quality)
File "/var/task/gmusicapi/clients/shared.py", line 84, in _make_call
return protocol.perform(self.session, self.validate, *args, **kwargs)
File "/var/task/gmusicapi/protocol/shared.py", line 226, in perform
raise CallFailure(err_msg, call_name)
gmusicapi.exceptions.CallFailure: GetStreamUrl: 403 Client Error: Forbidden for url: https://mclients.googleapis.com/music/mplay?opt=hi&net=mob&pt=e&slt=1527207821033&sig=OMuDy_lx-OjkiflstlKqbnb1k9s&mjck=Tbm4qc5ka36wvdcifyfryjalqx4&hl=en_US&dv=0&tier=aa

So I work on an Alexa app that is built off of this, which is awesome by the way and suddenly my deployment stopped working. So I have a serverless setup that depends on this that has been working for about two months now and it randomly stopped working today. It gets the song info but fails to stream the song. The only thing I can think of is that I'm using S3 to redirect (I've used it for almost 200 songs at this point.) the stream through because I'm using a serverless setup (ApiGateway and Lambda) cause its just so damn cheap.

Long story short, does anything glaringly wrong jump out in my stack trace? Like I said I haven't changed the code at all, it seems Google is blocking me, thats my only idea. Maybe I should try rotating the app password every so often? Maybe this is their way of telling me? I'm not sure.

[fergyfresh]

In order to use ApiGateway + Lambda with this skill we need to manually login to the api in a python shell from a computer with a good mac address and then use Mobileclient.get_registered_devices() to get the list of valid devices and then use one of those device ids hardcoded as the device id.

[simon-weber]

Ah, yeah. I think geemusic defaults to from_mac_address, which doesn't work well in paas setups: stevenleeg/geemusic#183 (comment).

[fergyfresh]

Any idea why it worked for about a month and then they finally cut me off? I did notice the stream taking longer and longer to play even from the native app.

[simon-weber]

Hard to say. Could have been that you were creating lots of new devices without knowing, Google blacklisted the lambda ips, aws switched to new ip range that Google already had blocked, etc.

[fergyfresh]

True. I just didn't know how much you knew about the inner workings of this stuff. It appeared to only block me because I wasn't using a device id, or does gmusicapi randomly generate one if you don't supply one. The reason why I'm asking is that it logged in on the PAAS, but couldn't download from get_stream_url.

[simon-weber]

gmusicapi can generate one from the machine's mac address if requested, though it's not recommended. geemusic defaults to this behavior.

[fergyfresh]

Ah ok, and since Lambda hits a different machine every time which has a different MAC Address, but has the same ARN, Google blocks it because I'm coming from the same address but with multiple different device ids.

I now see what you were saying about the creating lots of new devices.

[GallantJR]

I was using the same MAC address for each call for a month now... Today my main MAC address that i pass trough the API as an hardcoded string isn't working ! So i don't think the only issue is about using multiple Mac addresses.

[robertocock]

I have also been using the same MAC address for each call for well over a month, something was changed on Googles end. I even have tried using a valid android device_id that is registered with the account, seems the only way I am able to get Mobileclient.get_stream_url() to work is being logged into the account on the mobile app with the device of the device_id at the same time. Otherwise it is throwing a 403 error. I am able to reproduce this with 2 different accounts using 2 different devices and device_id's on 2 completely separate networks; so it is not blocking or anything strange on Googles end other than the changes they made this last week.

[robertocock]

After some testing I have found out that using the MAC address no longer will work. You need a valid android device_id that has been linked (authorized) to the account. You also can no longer use your regular login/password, you must use an app password after setting up 2FA. Once I met these two requirements everything started working again.

[dnikkel]

@fizzybunk I'm having this problem on my Raspberry Pi. Do you know what I need to do for a valid device_id?

[dnikkel]

@fizzybunk Actually, I tried plugging in one authorized Android device ID's and I'm back to streaming GMusic!

[fergyfresh]

That's right baby.

…

On Sun, May 27, 2018, 3:31 PM Dave Nikkel ***@***.***> wrote: @fizzybunk <https://github.com/fizzybunk> Actually, I tried plugging in one authorized Android device ID's and I'm back to streaming GMusic! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#590 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJN8OJwu_lyXlUbhtgXr6haZm2sA22dpks5t2v77gaJpZM4RGSpw> .

[butty2017]

@fergyfresh

Any idea why it worked for about a month and then they finally cut me off?

I think this problem has something to do with Google Play Music's integration to YouTube Music. Because I had the same problem just after the change (night May 22 UTC).

[simon-weber]

My CI broke too. I'll try to make some time for this over the weekend.

[fergyfresh]

I'll try to help out too. I think I have a workaround, if you use FROM_MAC_ADDRESS to log in and get the valid device IDs and then logout and re-log with the valid device id it works. I know its a run around, but that's the quickest solution I could think of off the top of my head. Do you have any idea what way you'd go?

[simon-weber]

If it's just the format of the id that they care about, we might be able to fix the mac address generation. Otherwise, we can just make it easier to pick an existing id (I don't think one is needed during login, so we can pick it afterwards).

[fergyfresh]

Yeah and if they actually care about the id we kinda have to use the already existent device ids right?

…

On Wed, May 30, 2018, 10:35 AM Simon Weber ***@***.***> wrote: If it's just the format of the id that they care about, we might be able to fix the mac address generation. Otherwise, we can just make it easier to pick an existing id (I don't think one is needed during login, so we can pick it afterwards). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#590 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJN8ODjOu8Ny3chLvMaDpJLshXlgNPJqks5t3q5HgaJpZM4RGSpw> .

[thebigmunch]

Just an FYI: when I was poking at this in the IRC channel with someone, I noticed that my Pixel 2 XL's associated ID wasn't its Android Device ID, but rather its Google Service Framework ID. Seems that it is the same kinda format, digits and uppercase letters. But they are different.

[thebigmunch]

If it's just the format of the id that they care about, we might be able to fix the mac address generation.

From my testing, this is not the case : (

Appears to require the ID of a linked Android device.

[simon-weber]

An interesting finding: I can get stream urls with a device id that's been registered to another account. There were a few hangups, though:

• on my first try, I got a NotLoggedIn when calling get_all_songs. I also got a security alert to my recovery account notifying me of a login from an unrecognized device.
• on my second try with the same args, I got an InvalidDeviceId since the device didn't show as registered.
• after disabling that check, I was able to get a stream url. Interestingly, the device still doesn't show in my registered list afterwards. The same id can still be used on the original account.

Trying a fake id generated here results in the same login notification, but I'm not able to get a stream url.

So, my guess is that they're verifying that the android device has been connected to GSF, but not necessarily that it's registered to the account. That means we could potentially hardcode ids into gmusicapi, though that doesn't seem like a good idea.

Unless someone comes up with something better, my proposal is:

• remove from_mac_address support
• add docs that streaming requires having previously installed the app
• provide a command line way to list registered device ids (probably just a python -m gmusicapi...)

[simon-weber]

Oh, and checking how ios devices are handled is a good idea if someone gets a chance. They're not as tightly linked to Google as android devices are, so they might be a potential way to fix from_mac_address.

[ndg63276]

Both ANDROID and DESKTOP_APP in the list. Android ones are 16 digit hex, and Desktop ones are 64 chars. My current phone is on the list, and it's not its MAC address, maybe it is the Google Service Framework ID but I have no way to check that.

[fergyfresh]

@ndg63276 what did you try and what does didn't work mean. You omitted the 0x off the front right and you have a subscription to google play, not the free version?

[ndg63276]

I only have the free version, up until a couple of weeks ago that worked fine, using

api=Mobileclient()
api.login(email, password, "0123456789abcdef")

Now that logs in, but any call to api.get_stream_url(song_id) returns a 403. I get song_id from my uploaded songs, eg song_id = api.get_all_songs()[0]['id']

I have also tried:

for device in api.get_registered_devices():
    device_id = device['id']
    api.login(email, password, device_id)
    try:
        url=api.get_stream_url(song_id)
        print url
    except:
        api.logout()

This started happening when I got a new phone, so I thought maybe I'd hit a limit on number of devices (I had 9), but even after deauthorising an old device through the website, the problem remains.

[fergyfresh]

What's the description of the 403? They return 403 for a few different reasons. Could it be that you're trying them too quickly and you're getting 403 forbiddened by rate limits and not by invalid device?

[ndg63276]

Traceback (most recent call last):
File "", line 1, in
File "", line 2, in get_stream_url
File "/usr/local/lib/python2.7/dist-packages/gmusicapi/utils/utils.py", line 293, in wrapper
return function(*args, **kw)
File "/usr/local/lib/python2.7/dist-packages/gmusicapi/clients/mobileclient.py", line 372, in get_stream_url
return self._make_call(mobileclient.GetStreamUrl, song_id, device_id, quality)
File "/usr/local/lib/python2.7/dist-packages/gmusicapi/clients/shared.py", line 84, in _make_call
return protocol.perform(self.session, self.validate, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/gmusicapi/protocol/shared.py", line 226, in perform
raise CallFailure(err_msg, call_name)
gmusicapi.exceptions.CallFailure: GetStreamUrl: 403 Client Error: Forbidden for url: https://mclients.googleapis.com/music/mplay?opt=hi&songid=dada4a40-1e5a-3710-8109-8120fddf38e6&pt=e&slt=1528311299883&tier=fr&sig=9CUUKbYP1-GgiccDhu-qkS7l8CU&hl=en_US&dv=0&net=mob
(requests kwargs: {u'url': u'https://mclients.googleapis.com/music/mplay', u'headers': {u'X-Device-ID': '1311768467294899695', u'Authorization': u''}, u'allow_redirects': False, u'params': {u'opt': u'hi', u'songid': u'dada4a40-1e5a-3710-8109-8120fddf38e6', u'pt': u'e', u'slt': '1528311299883', u'tier': u'fr', u'sig': '9CUUKbYP1-GgiccDhu-qkS7l8CU', u'hl': u'en_US', u'dv': 0, u'net': u'mob'}, u'method': u'GET'})

[thebigmunch]

Both ANDROID and DESKTOP_APP in the list. Android ones are 16 digit hex, and Desktop ones are 64 chars. My current phone is on the list, and it's not its MAC address, maybe it is the Google Service Framework ID but I have no way to check that.

The device ID was never the MAC address of the Android Device. You can check the GSF ID in the Settings of your phone. You can also download apps to display that and other information (e.g. the Android device ID). Just search for 'device id' on the Play Store.

The device ID in your failed request posted has a length of 19. Valid Android device and GSF IDs should be 16.

[fergyfresh]

Mine is also 16 characters.

[ndg63276]

The device ID in your failed request posted has a length of 19. Valid Android device and GSF IDs should be 16.

Mine is length 16, but it is converted to a base 10 int by the method _ensure_device_id()

>>> api.android_id
'1234567890abcdef'
>>> api._ensure_device_id()
'1311768467294899695'

But, using my Google service framework id as my device id does work! I made a stupid mistake, in my comment 2 days ago, I looped through the registered devices and tried using the id from each of them, but without accounting for the fact that they are reported with '0x' prefixes, which I failed to strip. D'oh!

[fergyfresh]

That's why I asked the question about the 0x prefix buster brown!

…

On Thu, Jun 7, 2018, 4:26 AM ndg63276 ***@***.***> wrote: The device ID in your failed request posted has a length of 19. Valid Android device and GSF IDs should be 16. Mine is length 16, but it is converted to a base 10 int by the method _ensure_device_id() >>> api.android_id '1234567890abcdef' >>> api._ensure_device_id() '1311768467294899695' But, using my Google service framework id as my device id does work! I made a stupid mistake, in my comment 2 days ago, I looped through the registered devices and tried using the id from each of them, but without accounting for the fact that they are reported with '0x' prefixes, which I failed to strip. D'oh! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#590 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJN8OIa1pNALI47eOUKmjCGRqZ8U2mHsks5t6OPQgaJpZM4RGSpw> .

[fergyfresh]

I tried this on my computer and using Mobileclient.FROM_MAC_ADDRESS and the device id created by the Mac address worked with get_stream_url. I haven't tried it back on my PAAS code, but I am confused again. So </3

[roadriverrail]

Joining the thread because I just ran into this issue myself while trying to pick back up my Play Music skill for Mycroft. I can confirm that I can use the FROM_MAC_ADDRESS to login, get my registered device list, and then hard-code an id out of there to make things work again. It'd be awesome to have an alternate solution, of course, since it feels like I'd be inviting the wrath of Google to go around spoofing my phone's ID on my Mycroft skill.

[budowski]

What worked for me was using @ndg63276's answer, just stripping the 0x in the beginning:

mobile_client = Mobileclient()
mobile_client.login('[email protected]', 'mypass', mobile_client.FROM_MAC_ADDRESS)

device = mobile_client.get_registered_devices()[0]
device_id = device['id'][2:]
mobile_client = Mobileclient()
mobile_client.login('[email protected]', 'mypass', device_id)

# Continue as normal

[fergyfresh]

@budowski do you know if it works with every ID type or just the ANDROID_ID type? I have a couple devices registered to my account so I was just checking if maybe I should also check for the device ID type that works for me.

For example, the iPhone's device ID doesn't have a 0x in front of it, as was the case with a few others when I looked at them a few weeks ago.

[roadriverrail]

@budowski I agree that this works personally and for now, but I am really leery of distributing code to others saying "This works only if you have an Android device already." and "You're technically abusing your Android device's ID and Google might take action against you." I can certainly keep doing things that way for personal development, but it feels like an unreasonable requirement and risk to give any users of my code.

[thebigmunch]

@roadriverrail By using gmusicapi at all, you're using private API calls that Google might already take action against you for using. None of this is supposed to be used by 3rd parties. And there are actually some things possible with this that would REALLY whip Google into a frenzy if they were abused.

That being said, I certainly wouldn't distribute one of my device IDs in code. Google now requires a valid Android (probably iOS, too) device ID for streaming. There is nothing to be done about it. If your application doesn't allow for users to provide their own Android device ID in some kind of configuration or hard-code it pre-install, then you're out of luck.

@fergyfresh We don't know. Earlier in this thread, Simon asked if someone could test with an iOS device ID. No responses were given.

[budowski]

I use Androids only.
But I have an iPhone laying around and can test this - how do I detect the ID for that device? Do I just need to install Play Music for it to work?

[fergyfresh]

yeah, install play music and auth to that device with your account.

then you call the same function get_registered_devices() on your computer or whatever and then that new device (iPhone) should be in that list as type iPhone or something. I don't know as I don't have an iPhone.

[budowski]

Tested this with an iPhone (IOS 10.3.3): The device ID is indeed different, needs a different mechanism for building the ID to be legal:

mobile_client = Mobileclient()
mobile_client.login('[email protected]', 'mypass', mobile_client.FROM_MAC_ADDRESS)

device = mobile_client.get_registered_devices()[0]
if device['type'] == 'ANDROID':
    device_id = device['id'][2:]
elif device['type'] == 'IOS':
    device_id = 'ios' + device['id'][4:]

mobile_client = Mobileclient()
mobile_client.login('[email protected]', 'mypass', device_id)

However, even though calling an API function like get_all_user_playlist_contents and add_store_tracks works for both iOS and Android clients, calling get_stream_url with the iOS client, returns a 403 error (while it works for the Android client, with the same params).

[thebigmunch]

@budowski I'm not quite sure how you came up with 'ios' + device['id'][4:]. The returned ID previously was of the form: ios:01234567-0123-0123-0123-0123456789AB. If the returned ID is still in that form, then you wouldn't need to process it (and you missed the colon with what you did). If it's no longer in that form, then I'd be interested in seeing what form it takes. And if that doesn't work, you can try it without the ios: prefix.

PS Almost none of the calls ever required any kind of device ID. It was just the streaming calls and the podcast listing calls. That's why it doesn't matter for most calls.

[fergyfresh]

So my snapchat got pwned somehow and posted a stream url of showtunes to every one of my snapchat contacts. Not sure if its a contaminated url as the url was deleted. I changed all my passwords and made everything 2 factor auth, but I was just checking to see if anyone else got pwned to see if this was related to our ANDROID_ID hack.

[budowski]

@thebigmunch - I came up with that format, since when I tried to login using the iPhone ID I received (as is, from device['id']), I got the following error:

gmusicapi.exceptions.InvalidDeviceId: Invalid device_id ios:1234ABCD-AAAA-BBBB-CCCC-1234567890AB. Your valid device IDs are:
* 123456789ABCDEF0
* ios1234ABCD-AAAA-BBBB-CCCC-1234567890AB

Where the first ID is the Android, and the second one is the iPhone device.

It seems like that's the logic the gmusicapi library uses to validate the device ID while doing a login (@simon-weber):

gmusicapi/gmusicapi/clients/mobileclient.py

Lines 46 to 58 in 7264dd8

	def _validate_device_id(self, device_id, is_mac=False):
	"""Ensure that a given device_id belongs to the user supplying it."""
	if is_mac: # Always allow logins with MAC address.
	return device_id
	devices = [
	d['id'][2:] if d['id'].startswith('0x') else d['id'].replace(':', '')
	for d in self.get_registered_devices()
	]
	if device_id in devices:
	return device_id
	else:
	self.logout()
	raise InvalidDeviceId('Invalid device_id %s.' % device_id, devices)

[simon-weber]

Heads up that the ios:... format ids are now accepted as of #610.

[shivasiddharth]

After authentication, using the Mac address, we can get the list of device ids, then logout and login again using one of those device ids. Its working for me. Here is what the piece of code looks like. Hope it helps someone.

api = Mobileclient()
logged_in=api.oauth_login(Mobileclient.FROM_MAC_ADDRESS)

for device in api.get_registered_devices():
    if device['type']=='ANDROID':
        deviceid=device['id'][2:]
        print(deviceid)
        break
api.logout()
logged_in=api.oauth_login(deviceid)

[Walkman100]

Alright, so combining #590 (comment) and #590 (comment), and the fix for iOS ID's, I've got:

mc = Mobileclient()
mc.__init__(debug_logging=False, validate=True, verify_ssl=True)
mc.login(username, password, mc.FROM_MAC_ADDRESS)

# Pick a device_id
device_id = None
for device in mc.get_registered_devices():
    if device['type'] == 'ANDROID':
        device_id = device['id'][2:]
        break
    elif device['type'] == 'IOS':
        device_id = device['id']
        break

if not device_id:
    print "No Android or iOS device linked to account!"
    exit()

mc = Mobileclient()
mc.login(username, password, device_id)

This works for me, I've got a few desktop clients and an android client on my account, however I would like to verify that this works on iOS - can someone with an iOS device check that this currently works? (I'm using it here)