Please provide a basic progress / transfer speed output ("CI compatible") on copy #658
Comments
|
Thanks for your report. Reporting the recent transfer speed should be reasonably possible using |
|
Thanks for the quick response. The issue I am having is to see the current progress (transfer speed per second) rather than the total process. In short: I simply want to see how quickly things are progressing currently. |
|
This got notably more complex in the meantime, because up to 6 blobs are now copied simultaneously; so, the concept of “current progress” does not make sense any more, without some sort of aggregated view. |
|
@mtrmac urgh, too bad :-( As cool as it is when Skopeo does sync / copy multiple gigabytes of Docker layers in seconds on a fast local network, when you are dealing with potentially slow registries which are accessed via the internet across half the globe any indication if things are still moving or (almost) stalled would help. |
|
@vrothberg Can we use the progress bars that we use in Podman for this? |
Yes, that's possible. There is an example on GitHub (https://github.com/vbauerster/mpb#bytes-counters) that indicates the download speed. I am currently busy with other things but the change should be straight forward. We need to change the decorators in |
|
A friendly reminder that this issue had no activity for 30 days. |
|
unstale |
As already mentioned by @mtrmac, copying layers is happening in parallel which makes it challenging to have a single indicator of the download speed. But I want to revive the conversation on how we could get closer. @frittentheke, would the upper example be of any help? Each layer would have the IO printed before the progress bar. |
|
@vrothberg absolutely would this help, even be more detailed than a summed up data rate. Just please also consider the usage of Skopeo in CI pipelines which do not like constant updates to the same few lines all the time. Maybe an option to refresh the output only so often is sensible here? |
It seems possible in principle to sum the speeds of the individual items; of course actually doing that, in concurrent code, and figuring out the relevant heuristics (smoothing / moving averages, and making sure the data for all streams covers the same time range, so that if 6 items “take turns” on a 100 MB/s link, 5 report 0 speed and 1 reports 100 MB/s at the time it is receiving data, we don’t sum that up to 600 MB/s) might end up pretty complex. But we are getting a bit into the weeds… do I understand correctly that the core need is to:
and it’s not very important what the actual data reported is, beyond the two concerns above? |
Yes. Maybe continue to print the single line that was introduced with containers/image#558 and just add some progress info like |
|
A friendly reminder that this issue had no activity for 30 days. |
|
This is not stale - was just talking to @vrothberg and @mtrmac about this ;-) |
|
A friendly reminder that this issue had no activity for 30 days. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
@mtrmac @vrothberg @frittentheke Any movement on this? |
|
Not to my knowledge. To summarize the above conversation:
@mtrmac @frittentheke does that sound right to you? |
|
The progress bars already include (per-layer) speed, so that’s a no-op. I understand this as an opt-in, periodic, report, only in the non-interactive case. |
|
Compare #1477 ; it’s not quite the same thing but it might need computing similar data. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
@vrothberg I hope you don't mind me keeping this from staling out .... |
|
We don't just close stale issues, We use it as an opportunity to take a fresh look. |
|
It would be super useful to have some simple progress reporting that could be captured from stdout or stderr to provide some progress indications. We use skopeo in ScanCode.io to fetch images and it would help a lot to report some progress when we have larger images. For FWIW our code is at: https://github.com/nexB/scancode.io/blob/00bf2545436ebcfc5e94f45f9a29a4b2abfe2131/scanpipe/pipes/fetch.py#L92 and is a CLI wrapper using "docker://" URLs as inputs which are accepted in the UI of scancode.io to fetch and scan whole docker images for origin, license and more. See also aboutcode-org/scancode.io#372 |
|
A friendly reminder that this issue had no activity for 30 days. |
|
Gentle ping... I am still looking forward to this! |
This should not stale away .... @rhatdan @vrothberg |
|
Note that I have made some crude tests using This get us some output from mpb with escape sequences: Then using @vbauerster this project uses your excellent mpb https://github.com/vbauerster/mpb 🙇 .... would there be a way to get progress provided optionally in a non-interactive mode without terminal escape sequence decoration? @ cco3 FYI, this is the underlying issue making it hard(er?) to report progress when fetching images in aboutcode-org/scancode.io#372 |
|
Actually progress is rather in the image module? https://github.com/containers/image/blob/bd97b5826c0729bea56b0af8297a38ef85fe0380/copy/copy.go#L924 |
|
A friendly reminder that this issue had no activity for 30 days. |
It's useful to see in the progress output how many layers there are to fetch. This is similar to containers/bootc@6eb5718 which ended up being totally reworked in a nicer way in containers/bootc@d8b5df2 But doing the latter would require nontrivial changes to our DBus API around status and progress reporting...and I'd like to think about how we tackle that more generally in e.g. containers/skopeo#658 Closes: coreos#5024
It's useful to see in the progress output how many layers there are to fetch. This is similar to containers/bootc@6eb5718 which ended up being totally reworked in a nicer way in containers/bootc@d8b5df2 But doing the latter would require nontrivial changes to our DBus API around status and progress reporting...and I'd like to think about how we tackle that more generally in e.g. containers/skopeo#658 Closes: coreos#5024
|
Progress bar is absolutely needed. |
First of all, thank you massively for Skopeo and its ability to efficiently copy images between registries. Especially when having to use proxies and other special setups Skopeo is gold!
Just like @baracoder in issue #597, we are using Skopeo within Gitlab CI jobs.
While having Skopeo not print and update the otherwise nice progress indicators during a CI run is an improvement - not having any indication of the current transfer speed is an issue for our use case though. With copy being silent until it either succeeds or fails makes it hard to spot if there is a (speed) issue with the transfer.
In our case having some feedback on the transfer speed / likely success is crucial. May I kindly suggest to introduce a setting (or by checking for a tty to be present) to simply report the current speed / progress along with a timestamp in front about once a minute or every 30 seconds as a single line? This should not clutter ones output in CI and with the --quite option still available can still be switched off. This would also be nice for cron jobs to have some sort of "logging" of the progress maybe?
The text was updated successfully, but these errors were encountered: