New Authentication Flow Discussion

[ABaumher]

This is a place for us to bounce ideas and code stuff to fix #159.

[slo]

:)
You were refering that Steam suggests using OAuth2. Is this the page you are refering to: https://partner.steamgames.com/doc/webapi_overview/OAuth ?
I feel kind of lost when digging through Steam documentation. Maybe you know of better place to find Steam documentation, especially related to websockets implementation ?

[ABaumher]

Well, we're not doing Web Requests, but that's the basic principal. And yeah, digging through the documentation is not fun.

My idea is to basically steal whatever we can. So i'm using SteamKit as my basis. You'll probably want to clone SteamKit https://github.com/SteamRE/SteamKit so you can see what they do. Same for ValvePython, as we are basically redoing everything they already do, except probably worse.

[ABaumher]

OAuth2 is a fairly common authentication workflow. I know how it works, but don't really implement it ever.

We basically need to port everything in https://github.com/SteamRE/SteamKit/tree/master/SteamKit2/SteamKit2/Steam/Authentication to python. Fortunately a good portion is simple OOP classes, but the Task stuff will be a bit more work.

One thing that might give us a challenge is they use polling to check and see if the authentication went through. We do have multiple threads but i'm not sure which would do the polling on.

[slo]

Ok, I'll take a look, but in the evening probably.

[ABaumher]

sounds good.

A quick note on how SteamKit does it: BeginAuthSessionViaCredentialsAsync is our starting point. Well, technically, GetPasswordRSAPublicKeyAsync, but that's just so we can RSA the password. It returns an Authentication Session. In Authentication Session, we use the polling call PollingWaitForResultAsync (essentially, wait for Steam Guard to finish) to get our tokens. Getting that integrated into our code's design is the biggest issue

[ABaumher]

What's weird is we use a website (index.html) to get your user and password, but for the life of me i can't figure out how it sends that to steam. I don't see anywhere in that web form where we call steam's API with our user and password. unless it's all via the code-behind and that's a glorified enter user and password page. As i'm typing this, i'm begining to think that's the case. Hoo, boy, that's hacky af.

[slo]

Hmmm, index.html is used mostly for rendering part. The rest is sent through websockets.

[ABaumher]

It's a glorified Enter User and Password prompt. It's supposed to be used for Web Auth, but evidently they aren't doing that. The problem with WebAuth is then Steam can limit our calls per day, while the network is largely hidden.

[ABaumher]

hence the hacky comment

[ABaumher]

In our code, it looks like we can change _prepare_log_on_msg, log_on_token and log_on_password so that instead of using ClientLogon as our message, we use BeginAuthSessionViaCredentials. The response changes as well

[slo]

I'm trying to call it using https://github.com/ValvePython/steam code. It's simpler to understand the concept.

[ABaumher]

I don't blame you lol. Playnite does an anonymous login, which is what we currently do. Anonymous logins only persist as long as they are logged in, they don't give you any access token or refresh token. I have no idea how PlayNite stays logged in.

[slo]

There is also this "sentryfile" - received along with CMsgClientUpdateMachineAuth message - sending it allows omitting 2FA

[ABaumher]

Huh. Where's that at?

[slo]

Just search for CMsgClientUpdateMachineAuth in protobuf_client.py (I'm detached from source)

[ABaumher]

I really wish SteamKit's wiki was finished. RIP. Regardless, their authentication example is relatively straightforward. https://github.com/SteamRE/SteamKit/tree/master/Samples/1a.Authentication

They perform the login via those two calls, then actually use the client login message we're already using. Which is great, because it means a lot of our code doesn't need to change. basically, all we need to do is make 2 calls before we call client login.
https://github.com/SteamRE/SteamKit/blob/fe2a34eac1b58aef039e1d8bff96e7d90a535d21/SteamKit2/SteamKit2/Steam/Handlers/SteamUser/SteamUser.cs#L326

[ABaumher]

We just need to port this

// Begin authenticating via credentials
var authSession = await steamClient.Authentication.BeginAuthSessionViaCredentialsAsync( new AuthSessionDetails
{
    Username = user,
    Password = pass,
    IsPersistentSession = false,
    Authenticator = new UserConsoleAuthenticator(), //This authenticator needs to change since we aren't using command line
} );
// Starting polling Steam for authentication response
var pollResponse = await authSession.PollingWaitForResultAsync();

// Logon to Steam with the access token we have received
// Note that we are using RefreshToken for logging on here
steamUser.LogOn( new SteamUser.LogOnDetails
{
    Username = pollResponse.AccountName,
    AccessToken = pollResponse.RefreshToken,
} );

[ABaumher]

As for SteamUser.LogOn:

    public void LogOn( LogOnDetails details )
    {
       //... Error Checking... Omitted for brevity.

        var logon = new ClientMsgProtobuf<CMsgClientLogon>( EMsg.ClientLogon );

Is a call we already do via log_on_token

[ABaumher]

I'm going to rebase to your repo so i can actually commit changes. If i have permissions i can just push it to a branch on your repo, or i can fork it and do a PR. Mine has mixed Tabs and Spaces so it's a mess anyway. I am really not a fan of Python forcing soft tabs on everyone. A bracket is literally one character and is infinitely less error-prone

I'm looking at the code, and since I've never gotten to the login with token option (we don't have the token, that's been our whole problem), i have no idea if that will still work but i think it will and we can leave it alone.

[slo]

I gave you access to it. Please check and let me know if it works fine

[ABaumher]

Github says i have push access. Gonna create a fork and push just to see if i can.

[ABaumher]

👍

[ABaumher]

I pushed a small change to the gitignore file because i'm using Visual Studio and it creates a .vs folder with a bunch of garbage. If it causes any conflicts on your end, you can ignore it. merging it should not pose an issue but lmk.

Edit: nevermind. It'd require doing the PR with approvals and it's not worth the effort lol

[ABaumher]

We're going to need RSA for the password. I've found rsa and crpytodome. I'm guessing your python-fu is stronger than mine, so if you have a preference either way, let me know. https://stackoverflow.com/questions/30056762/rsa-encryption-and-decryption-in-python shows how to do it with cryptodome, potentially, and https://www.section.io/engineering-education/rsa-encryption-and-decryption-in-python/ uses regular rsa.

Also, GOG requires us deploy any third-party python modules we use alongside the plugin. Since we're adding a RSA module, we'll need to do it The instructions are here https://github.com/gogcom/galaxy-integrations-python-api#deploy-structure-example. It's gibberish to me, because i almost never work with python. lmk if it makes sense to you, if not, we'll deal with it later.

I'm going to log off for the night, and try to get the RSA and other two calls we need implemented tomorrow morning.

Also, apparently Achievement Sync is broken if you want to take a look for it. Not the top priority but mentioned on the main issue page

[ABaumher]

Oh, you are not going to believe what i had to do to track down the Retrieve RSA Public Key call to Steam's server. There's no dedicated EMsg for it, and SteamKit's code is a question wrapped in an enigma. The Tl;Dr: version: they do a LOT of meta programming. I can walk you through it to see if another set of eyes caught any mistakes i may have made, but I'm guessing you'd rather not. Regardless, I think i have it, but i haven't tested it. I'd need to include the RSA module in GoG Galaxy as i noted in previous post, but haven't figured it out yet.

Does ValvePython retrieve that RSA Key anywhere? I didn't look too hard but if it does i can steal that code or confirm what i have will work.

I'm going to push what i have right now, so you can look at what i've been doing. I have no idea if it will compile, but afaik it should. I'm not pushing my python environment, but you'll need to install rsa if you want to t

Honestly, it might be easier to test this stuff in a dedicated python environment with all the GOG Galaxy stuff stripped out. A simple command line style thing that sees if we can get the 3 messages we need to work, then integrate back in.

If i'm really lucky, the Auth methods we'll need for the Polling Authentication function are in EMsg, but i'm not expecting that. I'm hoping it will at least be the same as the Retrieve RSA Key call so i can reuse that.

[ABaumher]

New Auth flow is brutal, not gonna lie. Pushed what i have so far. Changes are gonna propegate all the way up to backend_steam_network.py, unless i can hijack the steamguard string in the message. It's a string, but the format for that string is not stated. I have no idea if i need some sort of header to tell it which type of steam guard i am using or not. I'm guessing it's just the steam guard raw data and that'd be great.

[ABaumher]

Created a new project that strips out all (or nearly all) of the GOG Galaxy integration so i can just worry about getting the auth working. It'll make debuggin significantly easier. once it's done i can copypaste the code into the corresponding files in the regular project. https://github.com/ABaumher/IntegrationMinimal

[nakkamarra]

@ABaumher I'm somewhat familiar with python, let me know if I can help out somehow

[ABaumher]

@nakkamarra Just saw this now, lol. could have used that 4 days ago. I'm still going to try to do some code cleanup, but i quickly realized how integrated GOG galaxy api calls and errors are, so i've deprioritized it. If i haven't pushed the latest, i'm going to do that now. If you want, you can give it a look over and see if there's anything you'd do different or if there's a python on-liner that i could use that doesn't sacrifice readability.

As of a few 5/5/23, i've got the plugin into a rough state, and if you want to help debug this shit, i would absolutely welcome it. If you clone the authfix branch on the repo (https://github.com/slo/galaxy-integration-steam/tree/AuthFix), then copypaste all the files from the repo into their respective places, you can almost get to where we are.
you'll also need to dump two python modules into their self-contained form and then copy-paste the folders into the plugin as well.

replace the placeholder with a temporary file of your choice (must exist), then copy the contents of that temporary folder to the install directory of the plugin
python -m pip install pyasn1 --target <path\to\temporary dump folder\no\braces> --implementation cp --python-version 37 --no-deps python -m pip install rsa --target <path\to\temporary dump folder\no\braces> --implementation cp --python-version 37 --no-deps

On Windows, you'd put them in %localappdata%\GOG.com\Galaxy\plugins\installed\steam_ca27391f-2675-49b1-92c0-896d43afa4f8

I will need to automate this so i can get everyone to be able to test it before going through making it available through the plugin manager on GOG, so if you want me to do that now, i can. I could do it as either a shell script or batch/ps1)

[ABaumher]

Alternatively, are you any good with PyTest? If so, i'm going to need a lot of test cases when this all works. Most can be repurposed from the old code, as ~90% hasn't changed, but some stuff is being essentially reverted to pre 1.0 release of this plugin

[slo]

I still try to figure out how do websocket work. And where is the documentation for it. This is new thing for me. I wonder why this was not implemented using Web API. If would be way easier and straightforward.
Unfortunately I don't have enough time for it. I only have small portion in the evenings :/

I tried adopting this https://github.com/SteamRE/SteamKit/tree/master/Samples/1a.Authentication to python, but with no luck.

[slo]

Yup, I also noticed that there is no dedicated EMsg for it, but from SteamKit I extracted that they send: ServiceMethodCallFromClientNonAuthed. Althoutgh I didn't manged to make it work, i also found: https://steamwebapi.azurewebsites.net/ and this HTTP call just works:
https://api.steampowered.com/IAuthenticationService/GetPasswordRSAPublicKey/v1?account_name=your_name

I'm lost what is Valve and what is not and where is official documentation documenting IAuthenticationService.

[slo]

Oh, I guess I'm not the only one lost:
"I created this documentation to make up for the abysmal lack of any official documentation on the part of Valve. Sure, they maintain various wikis, but they're manually maintained, rarely updated, and often don't reflect the current nature of the Steam Web API. To solve that problem, this page will update every 24 hours with the latest state of the Steam Web API public endpoints. In addition, I will regularly update the various How To sections in case something more detailed changes."
This is from https://steamwebapi.azurewebsites.net/

[slo]

I focused on getting old "flow" working but with tokens. I managed to get to the point where I have to fill in SteamGuard code.
But now I know that I actually don't know how galaxy AND plugin work altogether.

[ABaumher]

It's confusing as all hell. The code design is absolutely terrible. I plan to refactor it all to something simpler that's well documented and easier to maintain, but for now getting it working takes priority.

I have a refresh token, i need to get it to work in the old flow. If i'm logged in through the new flow and i just got the token, i can call the old client login method with just the username and refresh token and it works. If i save the token and username, but don't go through the new flow, it errors. If i can just get the token login from the cache to work, i have the entire login process done. We'd just need to hook back up all the sync calls. But i can't figure it out right now. If you have an idea, let me know

In the meantime, i'm trying to hook back up the sync calls. I have to manually delete my cache every time it fails, (EDIT: or disable reading from the cache, which i've done for now), which is annoying, but if i get it working again, it'll be one less thing to fix.

[ABaumher]

So this happened

I don't know if i get all the data - Friends, etc. But it did sync my games

I doubt i can stay logged in if i close it though

[ABaumher]

I get like 95% of what we used to. This is awesome. I removed a couple calls that were added to a job list, so when it tries to write to that list, it errors, but i can rewire them in, hopefully.
plugin-steam-ca27391f-2675-49b1-92c0-896d43afa4f8.log

[ABaumher]

I don't stay logged in. I'm waiting on SteamKit for a response at SteamRE/SteamKit#1220

[ABaumher]

@slo The Achievement issue seems to be that when you get achievements in a steam, they don't update. Or, maybe it's they don't appear at all. idk.

Honestly, i really need a break from this, but unfortunately, i've done a terrible job explaining what i've done and a lot of the progress is my work, so I can't just hand it off. Tears of the Kingdom is out tomorrow, i might play that for a few days, sorry, lol.

[ABaumher]

Currently Testing https://github.com/SteamRE/DepotDownloader/tree/auth/accesstoken. If i'm lucky, i can find out how it works when the sample on steamkit does not. Port the message to Python, and et voilà! I can freeze the repo i'm working on so you all have time to read it before 17 other things change (which makes it hard to understand, sorry y'all), and then step back while the testers find bugs we need to fix.

[ABaumher]

Unfortunately CSGO is absolutely massive and that's the depot i'm apparently downloading. oops.

[ABaumher]

Afk rn, but i think there's also an asyncio wait_for which you can specify "all" for that'd work as well. I think we want those jobs because they give us some information on the games that we used before. I want to refactor the whole thing lol. The new protos have rpc options you can get with grpcio, which should let us simplify some calls but we're on an extremely old protoc

…

On Fri, May 12, 2023, 8:34 AM don-de-marco ***@***.***> wrote: yap, the job list thing is a bit hacky indeed. But after I uncommented the code it seemed to work and I didn't really think about it much more. There are other places in the code that I'd want to direct at my refactoring attempts first. I don't think one needs to loop over awaitables. There is asyncio.as_completed which returns awaitables as there are results. https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed — Reply to this email directly, view it on GitHub <#164 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASX2DPBERW7OMJGWW3AKEITXFYU4JANCNFSM6AAAAAAXQILGPE> . You are receiving this because you were mentioned.Message ID: <FriendsOfGalaxy/galaxy-integration-steam/repo-discussions/164/comments/5884591 @github.com>

[ABaumher]

So here's where i'm at. We go through the whole new auth flow as follows:

1. Say hello, so the websocket doesn't immediately close. We aren't authorized yet, and a call from a non-authorized will immediately close unless we announce ourselves first.
2. Retrieve the RSA public key for a username
3. Begin the Credentials Session, giving username and RSA-enciphered password
4. Update our Credentials Session with any Steam Guard results (if enabled and code is required)
5. Poll (ask) steam if the Credentials are Valid. If step 4 succeeds, this will as well (except mobile confirm, if you lie about completing it)
6. Retrieve the Refresh Token.
7. Login to the client version using this refresh token.
8. Save the refresh token for later logins.

That all works. To log in again, we should just be able to

1. EDIT: Say hello. We were always doing this, but i forgot to mention it. (works)
2. Retrieve the refresh token and username from the cache. (Works).
3. Log in to the client using the saved refresh token, like we do in 7 above. (Fails).

I do not know why. I cannot figure out why. Apparently this works at DepotDownloader (linked above) but i can't replicate it. At this point, i will pay someone to figure it out for me, i don't even care. Hell, if you can figure out depot downloader well enough for me to know what i need to input on it to test the login token stuff, i will pay you for that (you need to use -ugc or -pubfile, as it won't do auth for normal depot pull)

In the meantime, i'm going to play Zelda. Feel free to ask any questions about how the code works; i'll see the stuff on my email and respond here. i'm intimately familiar with pretty much all of the code structure by now. it's not how i'd have designed it but it's what was there and i'm just going with it

[quangkieu]

I just do some search. I wonder if this lib good to use? If it fit then you might not need to redo the work https://github.com/ValvePython/steam

[woj1993]

The only think I can think of is that you don't say hello when using refresh token or maybe we need to repeat steps 1,2,3 (only username),7?

[ABaumher]

We do say hello, but i missed writing it in. Thanks for catching that

[ABaumher]

We looked into it. Unfortunately they're in the same boat. The dev there expects the new auth flow to be up by the end of the month, hopefully

…

On Sat, May 13, 2023, 2:20 AM Quang Kieu ***@***.***> wrote: I just do some search. I wonder if this lib good to use? If it fit then you might not need to redo the work https://github.com/ValvePython/steam — Reply to this email directly, view it on GitHub <#164 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASX2DPBOOBVMLNNO3REL3RLXF4R33ANCNFSM6AAAAAAXQILGPE> . You are receiving this because you were mentioned.Message ID: <FriendsOfGalaxy/galaxy-integration-steam/repo-discussions/164/comments/5890394 @github.com>

[ABaumher]

Because it frustrates me, i'm back. Still mostly doing Zelda, but not as much as i'd like. I'm in the process of stripping out all the depot-related stuff in https://github.com/SteamRE/DepotDownloader/tree/auth/accesstoken, which apparently works with the new auth and remembers logins. I'm in the process of stripping it down so all it does is log in, and then store the login info. If it works like they say, i can see what the hell magic they are doing and reverse-engineer it (basically, just record their messages and then send the same stuff). I'm guessing they have a service_id, steam_id, client_id or something similar set or unset somewhere, and the lack of it (or the inclusion of it) is breaking both our code and the sample code from SteamKit.

In the interim, the latest auth_fix branch hasn't changed in a good while, and can be used for testing. At this point, you'll have to log in every time you boot up gog but the sync stuff should work.

[slo]

Isn't it here:
https://github.com/SteamRE/DepotDownloader/blob/auth/accesstoken/DepotDownloader/ContentDownloader.cs#L342-L360
and tokens are stored in depot.config holding AccountSettingsStore structure ?

[slo]

So it looks like they are using password for first attempt and for next ones, password is omitted and loginKey is used. But they seem to use "pure" Login method. They also use sentry file for some purpose.

[ABaumher]

It's also here: https://github.com/SteamRE/DepotDownloader/blob/auth/accesstoken/DepotDownloader/Steam3Session.cs#L94C13-L99
That part is important because it differs from the sample at https://github.com/SteamRE/SteamKit/blob/master/Samples/1a.Authentication/Program.cs#L17

The sample fails to log back in, this succeeds. I think that configuration has something to do with it. I didn't really look today (i took the day off from this), but when i get a chance i'll see what that config does in SteamKit and how it changes the message sent

[ABaumher]

I've debugged the sample code and depot downloader messages sent during login. They are identical, minus the obfuscated private ip, which is normal. Using the refresh token obtained in depot downloader works in the sample. At this point, i'm all the way down to where they serialize the message to bytes. There are one or two things that we might be doing in python we shouldn't, but i'm beginning to think the login auth messages we send has some slight difference. If the refresh token has some secret flag that prevents subsequent use, it'd be screwing us.

I'm also wondering if the SteamKit i originally built had a slight error. At the time the latest build (2.5 Beta) was not out yet, and i had to build it from source code. If that was the case, the AuthDenied error i have been getting there might've just been a bug. That's another thing i'll get to checking. Still errors. Which means the login auth calls differ somehow. Time to find out how.

[ABaumher]

Found it. Porting to python now

[ABaumher]

Anyone here, can you get the latest branch at authfix and try it out? I just remained connected on a restart and want to make sure it's not a fluke before i let everyone on the main issue board know

[don-de-marco]

Ah, nice. I didn't yet take the time to look into it. The authentication workflow used to ignore AccessDenied and AuthenticationStateWrong error messages which lead to unexpected situations while testing.

[don-de-marco]

Here's the log you were asking for: log.txt
I cut it once authentication succeeded and sync started

[ABaumher]

We might still ignore AuthenticationStateWrong because i've never seen it thrown. If we get an unexpected error, the old behavior was to just straight up fail and log the error. For the most part we still do that, so that might still break things

[ABaumher]

Also, for the confirm method, iirc you let them fallback to mobile code. can you check if that works, too? looking at the log you just sent, the end uri regex only goes to the confirm option, but i think the javascript for mobile code sends you to the code option, so that page might never close

Edit: can you fall back to mobile code on the http page? is that something the mobile app offers/allows? if not, ignore me. I have no idea how it works lol

[don-de-marco]

I've seen this message when I was playing around with the fallback you mentioned. Need to figure the conditions for that to appear though, can't remember how it happened right now. Will look into that tomorrow.

[ABaumher]

In order to push the stuff to Nebula, i've rebased my development branch to their fork. I am now actively developing at
https://github.com/ABaumher/galaxy-integration-steam/tree/master

[woj1993]

Hi how to install this fork? Dev env setup doesn't work for me.

[ABaumher]

hop over to #169 and i'll help you get set up. If there's some mistakes in there, it will probably affect everyone doing the testing so they will want to know

[ABaumher]

@don-de-marco do you think we could write a tool that could strip out the duplicates from either common.proto and common_base.proto or steammessages_base.proto? Would it even be worth it?

[don-de-marco]

See my comments on ABaumher#1
tl;dr: imo not really

[ABaumher]

man alive, it is good to have this thing mostly done. I really do appreciate all the help as well

[ABaumher]

Removing public profiles also slims down our package includes, which makes our plugin smaller. That's helpful. Hopefully i can simplify the install process even more. @woj1993 provided a nice powershell script that automates the various commands necesary to go from installing python 3.7.9 to installing the plugin with just one command. There's some minor tweaks i need to make, but hopefully i can get a shell script up shortly for our MacOS users.

[ABaumher]

@don-de-marco @slo I updated the app.txt to remove the stuff from public profiles. It does cause an issue with user_profile.py, but that seems to be unused, so i removed it and all references to it as well. I don't see any other references to these removed modules, but i could use another set of eyes. lmk if you catch any other references that are now missing

[ABaumher]

At this point, we're basically done, minus some bugfixes and QoL changes. If anyone here can figure out pytest so i don't have to and fix all the shit we broke, that'd be amazing.