-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Non-Steam Game Categories Not Working #949
Comments
Have you tried using the latest SteamTinkerLaunch (SteamTinkerLaunch-git from ProtonUp-Qt with Advanced Mode enabled)? You're using v12.12 which is very old. Also, Add Non-Steam Game got a lot of useful improvements lately. It also correctly sets the Steam AppID now, which may resolve the problem. I'll take a look and see if tags are working, perhaps how we write them out is wrong or something. This worked before a long time ago, but I'll take a look when I have some time.
That program is written using custom libraries in TypeScript, SteamTinkerLaunch writes out directly to the binary VDF file and is written in Bash, so it is not super helpful unfortunately. |
I've confirmed the issue with the latest master (93ec382), but I haven't figured out the cause. I made a few casing changes, and after that, byte-for-byte I can generate the same shortcuts.vdf file from Steam as I can with STL. Tthe file contents are identical, making sure categories are selected in the same order, naming and paths are the same (including quotes and no trailing space), so I can't figure out why the tags in the My hunch then is that Steam sets the collections elsewhere, maybe in the same place it uses for Steam games now, but even when editing categories from Steam, Steam itself updates Also:
I'm pretty happy to hear this. This is how I use the |
Confirmed that writing categories out to the Here is an example script: import vdf
user_id=''
vdf_path = f'/home/user/.local/share/Steam/userdata/{user_id}/config/shortcuts.vdf'
vdf_dict = vdf.binary_load(open(vdf_path, 'rb'))
# Set first category for first shortcut to 'Valve'
vdf_dict['shortcuts']['0']['tags']['0'] = Valve
print(vdf.binary_dumps(vdf_dict['shortcuts']['0']['tags'])) # Verify new tag was added
# Write out new binary content
with open(vdf_path, 'wb') as new_vdf:
new_vdf.write(vdf.binary_dumps(vdf_dict)) After opening Steam, the new shortcut is not present, despite it being in the VDF file. Here's a screenshot from Okteta inspecting the raw VDF file of my own Steam must store the categories elsewhere for Non-Steam Games. |
|
It looks like information on which tags a game is using is stored in I discovered this when I noticed
This is an escaped JSON string, and controls which Collections a Shortcut shows up in, as well as the hidden status! I don't understand yet how collections are inserted or updated, and how SteamTinkerLaunch needs to write out to this file just yet, but it was interesting. I mentioned that the Hidden status of a game is stored in this file as well. If you recall, even though Steam doesn't read collections from In other words, Steam uses
For AllowOverlay, it actually stores this in a separate block to
Therefore, on the SteamTinkerLaunch side, we need to make the following changes:
That'll be a pretty significant refactor, but hopefully doable. |
It looks like the {
"uc-1QwE3sdxpu7J": {
"id": "uc-1QwE3sdxpu7J",
"added": [
<appid>
],
"removed": []
},
"uc-y5nlF+rzO*+fR": {
"id": "uc-y5nlF+rzO*+fR",
"added": [
<appid>
],
"removed": []
}
} I assume It's worth noting too that However even if these invalid entries are removed (i.e. collections added for shortcuts no longer in the library), Steam will remember them and re-insert them. So it must track it somewhere else, perhaps in a LevelDB file. So this makes me wonder if inserting a new collection here will actually work, or if this is just a "cache" that actually pulls from LevelDB... I hope not, because updating information under "Apps" for the Overlay and VR settings does actually work, so I hope the same applies for collections. |
Another complication: Not all tags use this encoded format. We sometimes have to use a different format, where we can insert an entry for a new tag like this: "from-tag-tagname": {
"id": "from-tag-tagname", // Must match object name
"added": [
<appid>
],
"removed": [] // This can always be blank
} I noticed this because for some collections, Steam uses this format, but not for others. For example, my "ATLUS" tag uses I don't know if it's possible to know which tag will use which format. Perhaps whatever say SteamTinkerLaunch parses information about available tags will give us a clue, but I haven't investigated deeply yet. My guess is tools like Steam-ROM-Manager don't have this problem because they create new categories, and expect the category names they provide to not exist. Despite all of this, it seems like we can insert collections this way. We just need to figure out the logic to insert, and also figure out a way to know which tag format to use (the encoded |
It looks like we can find out if a tag uses the old-style or new-style tag naming convention by parsing If we can figure out how Steam encodes the collection names, we could take an input collection name - say "Shortcuts" - and convert it to this format. Then we can check if the file has either Depending on which we get a match for, we can then infer which name to use when writing out to The tricky part now is figuring out how these strings are encoded. |
Actually, for which tag name is used, it seems to be the other way around: Still no lads for how these are encoded, I can't discern a pattern because even tags starting with the same letters look entirely different. For example "A" and "AA" are both entirely different tag patterns. Instead, it seems to be some form of ID. For example, the following are two collections created about 15 seconds apart. The first one is "A", and the second one is "B" (despite how the client displays collections, they are stored case-sensitive, which can be seen in {
"uc-9DEqXUO*+qs06": {
"id": "uc-9DEqXUO*+qs06", // "A"
"added": [
<id>
],
"removed": []
},
"uc-38QEEdaMgmw0": {
"id": "uc-38QEEdaMgmw0", // "B"
"added": [
<id>
],
"removed": []
}
} We may have to resort to grepping this from some LevelDB file or some other file... Possibly relevant Steam-ROM-Manager file: https://github.com/SteamGridDB/steam-rom-manager/blob/cef009ef9b3bd742052a4c968747b180d89ea564/src/lib/category-manager.ts (although it looks like they write directly to LevelDB) |
There is one LevelDB file that has the structure we need. It's a binary file but it has a list of lots of information including information about Steam collections. We can find this file by grepping for the known JSON start hex bytes from all files in the This is a quick-and-nasty command I used in my terminal to write the grepped JSON out:
This will produce the JSON we need. Sadly, some characters appear to be invalid, especially once we go beyond the user collections. We'll need to figure out a way to resolve the invalid characters when converting back to decimal with Once the JSON is sanitised though, we can parse it with something like this |
The contents of this file may not always be reliable, for example if it is parsed while Steam is writing into it, or while Steam is closing. However, in a happy-path scenario, the following script can fetch the collection ID by collection name: #!/usr/bin/env bash
## get_steam_collection_id.sh
##
## Example usage: bash get_steam_collection_id.sh "Danganronpa"
## Example output: Danganronpa -> uc-k4365+6f3
startbytes_nobrackets="636c6f75642d73746f726167652d6e616d6573706163652d31.*?"
startbytes_withbrackets="${startbytes_nobrackets}5b5b"
endbytes="01495f68747470733a2f2f737465616d6c6f6f706261636b2e686f73740001"
steam_leveldb_path="$HOME/.steam/root/config/htmlcache/Local Storage/leveldb"
jsonfilename="${steam_leveldb_path}/collectionsjson.json"
findname="$1"
if [ -f "${jsonfilename}" ]; then
rm "${jsonfilename}"
fi
# grep on hex representation of each file in the leveldb dir
# for known byte sequence representing collections json
for f in "$(find "${steam_leveldb_path}" -name "*.log" -type f)"; do
filebytes="$( xxd -p -c 0 "$f" )"
if grep -qoP "${startbytes_withbrackets}" <<< "${filebytes}"; then
# Parse the JSON bytes out based on known start and end bytes,
# convert to char which should be valid JSON, and write out to file
grep -oP "${startbytes_nobrackets}\K.*?(?=${endbytes})" <<< "${filebytes}" | xxd -r -p | strings | tr -d '\n' > "${jsonfilename}"
break
fi
done
# If JSON file wasn't created, no file matching bytes above was found
if [ ! -f "${jsonfilename}" ]; then
echo "Could not find LevelDB file with JSON collection information, try closing and re-opening Steam"
exit
fi
# Use JQ to parse out the 'value' property from each JSON object starting with 'user-collections'
# Each collection entry has a 'value' property which is a JSON string that we can parse out
# and get the 'id' and 'name' properties from
while read -r CATENTRY; do
CATNAME="$( echo "${CATENTRY}" | jq -r '.name' )"
CATID="$( echo "${CATENTRY}" | jq -r '.id' )"
if [ "${CATNAME}" = "${findname}" ]; then
echo "${CATNAME} -> ${CATID}"
break
fi
done <<< "$( jq -r '.[][1] | select(.key | startswith("user-collections")) | select (.value != null) | .value | fromjson | tojson' "${jsonfilename}" )" This is really fragile though, and if parsed at the wrong time, will miss categories or be entirely unreadable by |
Hmm, in testing, this seems to only be inaccurate while Steam is starting up or shutting down. Occasionally it breaks when Steam is running. I will need to do much more testing but this could be incorporated into STL with the caveat that Steam must be closed before we can write out collections. I considered storing and parsing collections, but I think fetching from Steam is probably generally ok if we don't try to add a collection while Steam is running or shut down. |
Tinkering around with making a tiny C++ program that can parse the LevelDB and interestingly I'm running up against a lot of the same formatting challenges as I was with parsing the log file. The data is reliably able to be extracted (unless Steam is running, because the DB is locked). I can sanitise it with Bash mostly I think (JQ doesn't like it but I'm still not seeing any errors) but I'm not sure how to sanitise it with C++... |
I should note that there are still various challenges with this approach:
I think the main way to get around this is just transparency on the wiki. |
Note to self: If we can't statically link against LevelDB, we could get it from the Arch mirrors for SteamOS: https://archive.archlinux.org/packages/l/leveldb/ Though I'd prefer something more self-contained. |
Created the little C++ program, it's up on GitHub at sonic2kk/DumpSteamCollections. There are no releases, and maybe just dynamically linking to EDIT: Nope, SteamOS doesn't have LevelDB. It doesn't come as a standard binary, like It is somewhat unfortunate to introduce this as a dependency for adding Non-Steam Games, but |
The overall binary size using |
Did some tinkering and removed some headers, binary size is down to just over 30kb. Pretty happy with that tbh, even with LevelDB and its Speaking of this dependency, I am unsure if the Steam Deck has it. If it doesn't, we will need to download that, which will be a real pain... I'm wondering if we could build an AppImage for this little program that includes these two dependencies. I'm unsure with how that would play with licensing though. Perhaps it's just something to document on the program's readme, and then on the STL wiki. I have no idea how to create an AppImage though, or what that might mean for file size implications (hopefully very little, since we only need to include a couple of dependencies for these executables). |
Steam Deck does indeed come with |
Using DumpSteamCollections, we can probably attempt to store a cache of collections somewhere. This means we always have some collections available. Since we can't read them when Steam is running, we'll just store what we can find and display those as available collections. |
System Information
Issue Description
Hello! I'm using steamtinkerlaunch for it's ability to add non-steam games to the steam library via command line options.
I am uncertain whether the 'tags' option is currently supposed to be working. This issue from several years ago mentioned that it wasn't working, but then the following comments were unclear as to whether that was resolved or not.
To be clear, I am running the script, closing Steam, then reopening Steam.
SteamTinkerLaunch successfully added the game to the library, but it was not added to any Collections.
I also tried the linked solution of running
steam://resetcollections
. I rebuilt my collections and tried again, and it still doesn't seem to be working.Is this intended to be working?
If it's not, perhaps looking at Steam Rom Manager would be helpful - they are somehow successfully adding items to the Steam Library and adding them to collections.
Thanks for reading!
Logs
steamtinkerlaunch.log
The text was updated successfully, but these errors were encountered: