borestad: PoC of streaming output from cached invocations

[dimo414]

@borestad posted this in #29 but I'm moving it out since it's not really related to fzf

Hi!

I wrote a Proof-of-Concept (that works) here for live output

https://github.com/borestad/ci-utils/blob/main/memo

Example

memo --ttl=60s -- wget http://speedtest.tele2.net/1MB.zip

Did a short screen recorder of how I use it.

Kapture.2023-05-03.at.21.00.44.mp4

As a notice. This PoC don't use the return value, but more as a invocation orchestrator. We essentially just cache the base64-encoded value beforehand and eval the code if not cached by bkt.

Todo: add a trap if the command is cancelled early... and probably clean up the script a bit 😃

[dimo414]

@borestad thanks for sharing your script. Looking over it, I don't actually know that this script benefits from using bkt at all. It's basically caching the downloaded file*, not the subprocess call (notably the cached invocations just print a "cached" message, the output of the user's command is not cached).

As you say, the script is basically using bkt as an "invocation orchestrator" or like a kind of mutex, which really isn't a use case bkt is intended for. bkt is being used to limit how frequently the .ttl file is written to, which could be done much more directly by manually touching the file and using stat -c %Y to check its age, or using a dedicated mutex mechanism such as flock.

Part of why I've avoided adding features like the ability to inspect the cache's state is exactly because I worry people will use that information :) I'm definitely open to exploring introspection use cases but I suspect many of them would be better served by custom logic or other tools.

If #31 is implemented would you be able to replace your script with bkt --ttl=60s -- wget ...? Or do you have other use cases for inspecting or modifying the cache's state?

---

* Worth noting if you're specifically interested in caching wget you might simply want the --timestamping flag, which causes wget to skip downloading a file that hasn't been changed on the server.

[borestad]

Hi!
Thanks for the reply. To clarify some things first :)

I'm using bkt for multiple things, but this in particular is for downloading multiple blocklists in different cronjobs.
Some of these blocklists are hosted on servers that only allow 1-2 request per 24/hours.

In this particular case - the memo script is only caching the result of an invocation (the exit code), not the file or stdout (even if one could use do that as well ofc)

This line just creates an invocation to a temporary file in a temporary directory
https://github.com/borestad/ci-utils/blob/f046c3b4fd62959656b2aef51312cfd6ee332930/memo#L69

https://github.com/borestad/ci-utils/blob/f046c3b4fd62959656b2aef51312cfd6ee332930/memo#L72
This line just checks if the file was ever created. If it was, then the TTL was expired.

Another thing is that I use often use git as a database, and touch/stat doesn't get carried over with git. In a ci-pipeline (I'm just using @action/cache to cache the bkt cache folder & restore it. Since it's only storing the status of invocations, it's usually just a couple hundred bytes to restore => instant.

wget --timestamping has proven to be extremely unreliable for me in real-world-scenarios. Proxys/CDN's not updating it properly, or some servers not support it at all (same with with etags or content-disposition). Too unreliable to use in production. I've had cases where the file on disk was set to the wrong year in the future, due to misconfigured servers, and in that case, the file was never update and had to be flushed manually.

The reason for live output was to be able to more easily debug really slow requests and see their progress.

All in all, this is experimental, but I wanted to show you how I'm personally are using bkt 🙂

Also, normally - the scripts looks something like this (with a download helper/curl wrapper)

memo --ttl=1800s -- timeout 120s dl http://someserver.com/file.txt

in pseudocode this might have looked like this

bkt --ttl=1800s --discard-failures --exit-direct-if-failed --delete-cache-if-fail --timeout 120s --expose-stats --dont-cache-stdout --stream-to-stdout -- dl http://someserver.com/file.txt

All the best
Johan

[dimo414]

And I'm glad you're finding bkt useful and appreciate the input, to be clear! :)

That said I do think some of these usages are better served by other approaches - at least as-implemented there's no real need to use in your memo script. I'm not trying to say you can't use bkt, just that I think there may be simpler options for that task.

Of course the goal "hit a web server no more than once every twelve hours" is very well suited for bkt, and I'd love to better understand why you need the wrapper script at all for that task. I'd think bkt --ttl=12h -- wget ... and done.

Going through the flags in your pseduocode:

• --ttl=1800s - works today (though you could say 30m if you wanted, IMO that's more readable)
• --discard-failures - works today, though I'd be concerned about passing this when hitting a server that requires traffic throttling since it could lead to an increase in hits if there's a failure for some other reason than the server being unreachable
• --exit-direct-if-failed - what does this mean?
• --delete-cache-if-fail - same here, what would this be for? Hopefully Mechanism for invalidating (parts of) the cache #5 can address this, though I'd probably offload the exact invalidate-if-fail behavior to the caller
• --timeout 120s - I don't think this should be implemented by bkt, consider timeout (and cache that call with bkt)
• --expose-stats - I assume this is Record cache statistics #19 - unless you're asking to have them logged to stdout?
• --dont-cache-stdout - what's the motivation for this? I don't think we'd add this to bkt because you can discard stdout via a shell, e.g. bkt ... -- bash -c 'foo >/dev/null'
• --stream-to-stdout - This is of course Feature request: stream the subprocess' stdout and stderr while it's running #31 and will be the only behavior once implemented, it won't be an option

[borestad]

Haha, I should not write this much when tired and feverish (tonsillitis is horrible) because when I read the post again, it's clear that my thoughts wasn't clearly explained or even any good.

The pseudocode above was clearly not a good example, because as you said - i.e --timeout (as I'm using it as well), should not be a part of bkt, 100% agree! As is the ability to just pipe to /dev/null. Sorry for the confusion. Also the use of --discard-failures was just to show how I'm using it today (and it works great). Same with --delete-cache-if-fail and --exit-direct-if-failed - they were just there for emulation. I wish I just had never written that pseudocode, because it just confused things. Sorry about that.

So....now. To clearify again (hopefully) 😂

1. bkt is extremely useful for me
2. I wanted to show some of my use cases
3. The already existing issues Mechanism for invalidating (parts of) the cache #5 , Feature request: stream the subprocess' stdout and stderr while it's running #31 and Record cache statistics #19 would solve a lot for me without the need of a wrapper (which in the end is what I'm trying to avoid btw 🥰 )
4. For ci/scripting purposes, I'm lacking a way to be able to retrieve the cached result, without risking to execute the command again if the ttl has expired. (dry-run?)

[dimo414]

For ci/scripting purposes, I'm lacking a way to be able to retrieve the cached result, without risking to execute the command again if the ttl has expired. (dry-run?)

Ok that sounds like #24 - the CI use case is an interesting one, it helps me understand why behavior like that might be useful.