Force prefill of specific app in event of prefill failure.

[cybersteel8]

In reference to this discussion: #216

---

There is a potential use case for a new (optional) parameter to be added to the --force command that will allow a specific app to be force prefilled, rather than --force only applying to the entire selected list.

The problem is when the entire LANCache is emptied, and a force prefill may not successfully download every item in the selected list to repopulate the cache correctly. For example, if during this force prefill, this message appears ("1 failed downloads"):

[2:28:39 PM] Starting Tiny Tina's Wonderlands
[2:28:51 PM] Downloading 48.33 GiB
1 failed downloads
[3:36:35 PM] Finished in 1:07:43.64 - 102.16 Mbit/s

It is likely that the download for this item is not complete and it may be required to perform another force prefill on this specific app. I have a list with 50 items selected, so it is infeasible to perform a full prefill of the entire selection list for the sake of resolving a single issue.

It seems that the application does not detect that the failed download occurred for this product, so a normal (un-forced) prefill considers this product up-to-date and the application does not make any attempt to check or acquire any data missed during the force prefill.

# ./SteamPrefill prefill
[5:33:31 PM] Starting login!
[5:33:33 PM] Logged into Steam
[5:33:33 PM] Steam session initialization complete!

[5:33:36 PM] Starting Phasmophobia
[5:33:39 PM] Downloading 8 GiB
[5:33:39 PM] Detected Lancache server at lancache.steamcontent.com [<IP REDACTED>]
[5:37:24 PM] Finished in 03:45.47 - 304.82 Mbit/s

[5:37:24 PM] Prefill complete!
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  Prefilled 50 apps in 03:53.83

   Updated │ Up To Date
  ━━━━━━━━━┿━━━━━━━━━━━━
      1    │     49

[5:37:24 PM] Disconnected from Steam!

Phasmophobia had no issue during the force prefill, there was simply an update to the product. All other products, including the failed one from the force prefill, was considered up-to-date.

---

There may be both a bug and a feature suggestion out of this scenario.

The bug may be that a product that experienced a failed download is considered up-to-date.
The expected outcome is: if a prefill of a product results in a failed download, the next prefill will attempt to prefill the product again, making no assumptions about the validity of the cache data (ie. a force prefill of that particular product only).
The actual outcome is that despite the failed download, the product was considered up-to-date and no attempt was made to correct any failed cache data.

The feature suggestion I have for this scenario is that a parameter could be passed to the --force flag, such as the App ID, that will force a prefill of that particular product, instead of the entire selection list. This would be for the user to manually attempt to resolve any problems they find with the cache data for a product.

An alternative may be to provide a means to "reset" SteamPrefill so it assumes no product is up-to-date, so that the next standard prefill will perform the equivalent of a forced prefill on all products. This could also support the optional parameter, as aforementioned for the --force command, to reset only a particular product.
This command would be used in the scenario where the user is emptying their cache and requires all tracking of all products to be forgotten. However, this may not address the above bug, as the tracking of failed downloads must still be accurate for this to function correctly. Furthermore, its behaviour is similar to the existing --force command and may lead to confusion.

---

Ultimately, the goal is for the cache to either automatically corrected in the event of a download failure, or tools introduced for the user so they may manually attempt to resolve cache issues they experience or observe through using LANCache or reported through SteamPrefill.

[tpill90]

One other issue to mention from our other thread :

---

[5:24:53 PM] Prefill complete!
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  Prefilled 50 apps in 15:24:31.64

   Updated │ Up To Date
  ━━━━━━━━━┿━━━━━━━━━━━━
     48    │     0

[5:24:53 PM] Disconnected from Steam!

The forced prefill finally completed. I noticed immediately that it says 50 apps were prefilled, but only 48 are updated. I believe the other app might be Euro Truck Simulator 2 that also shows "1 failed downloads".
The game Teardown had the unexpected error I mentioned in an earlier comment:
An unexpected error (System.Net.Http.HttpRequestException) occurred while downloading manifests. Retrying...

---

SteamPrefill is reporting that only 48 apps were updated, despite you selecting 50 total to prefill. This may or may not be related to the main problem reported on this issue, but its certainly worth keeping in mind.

[tpill90]

I've done some thinking on how I'm planning on implementing this, here's what I'm thinking of doing:

• Add a new command reset that does the following:
  • Deletes successfullyDownloadedDepots.json, effectively forgetting the current status of each app. The next prefill run will then start at the beginning, and run through every app again.
    • The user will be prompted with a message explaining what will happen once successfullyDownloadedDepots.json has been deleted.
  • Deletes all cached manifests in the Cache directory.
• The current clear-cache command will be merged with the new reset command.
  • This likely makes more sense as for 99% of the time users won't likely need to worry about clearing up space via clear-cache. Having this extra command adds noise to the list of available commands.
  • Additionally, clear-cache can be slightly misleading in that it doesn't affect the 'cache/Lancache' at all. It is instead clearing out SteamPrefill's locally cached copies of manifests

@cybersteel8 Thoughts on this? @cholzer79 I know that you ran into a similar issue when you cleared your cache, thoughts on this?

[cybersteel8]

The next prefill run will then start at the beginning, and run through every app again.

This sounds like it would be replacing the prefill --force functionality, as what you described is functionally identical. Have I understood you right?

clear-cache can be slightly misleading in that it doesn't affect the 'cache/Lancache' at all.

I agree with your sentiment here, so merging the two commands makes sense to me.

When I run a reset and a particular app shows the 1 failed downloads error like it did originally, would I be able to reset that particular app only or will I have to do the reset against the whole list? In my original post, I suggested that an optional argument containing the app ID could be provided to only force (now, reset) that app to re-attempt the prefill - instead of doing an entire reset.

I certainly like the suggestion, bringing --force out of being a prefill flag and becoming its own command as reset. It makes sense. I don't know what else your proposed implementation changes - I should tell you that I don't know what successfullyDownloadedDepots.json contains and how it impacts the next prefill. It looks like it has a map of app IDs to depots, so removing a particular app ID might be feasible. Being able to single out an app to reset is important to me.

[tpill90]

I didn't consider removing the --force flag, but now that you mention it it is something that maybe I could consider.

You are correct in that removing a specific app from successfullyDownloadedDepots.json isn't necessarily going to be possible, as it is a list of the depots only without any mapping back to the owning app ids. I am wondering though how necessary it would be to even need that at all. Suppose that in your original scenario you were able to use reset rather than running through again with --force. You would have needed to run through every app again since your cache was emptied out completely. You would have possibly still ran into the 1 failed downloads on a single app, and that app would have been marked as failing. Thus in the next prefill run that would be the only app that would be retried.

Also do you happen to have a Discord account? I'd be glad to talk about this more in chat, so we can hash out all the details like this without having to wait so long in between messages.

[cybersteel8]

Hey so I have been messing about with my LANCache today and I remembered this thread. I had to fix an issue and clear the whole cache (again, sadly), and I loaded up your app to prefill all my selected apps again. As the cache and this app were out of sync, obviously not all of the games were downloaded again.

I came to the realisation that I could use the benchmark functionality to redownload the entire select-apps list again. In fact, because it contains the setup --appid flag, I could have used this specifically for the apps that I wanted to ensure were cached correctly, but I didn't as I wanted to ensure the entire cache was correct (I used setup --use-selected). The benchmark functionality effectively ensures this during its warmup phase during its run.

In my original issue, after I saw the error message, I could have used benchmark setup --appid to specifically do a benchmark against that failed app, and the warmup would ensure it is cached correctly.

I hope these thoughts are helpful :)

[tpill90]

I appreciate the update on this, always good to hear some additional thoughts on this.

Something I've been playing around with is to just roll this "reset" functionality into clear-cache, which is the way that it originally worked. The only thing I'm not sure about is that clear-cache is a terribly named command and doesn't completely reflect that it would be resetting the downloaded app tracking.