WIP - detect copy operations by other processes

[vrothberg]

containers/image#607 and containers/storage#307 must be merged before this PR.

Requires to vendor containers/image, storage and the latest
version of the progress bar library.

Signed-off-by: Valentin Rothberg [email protected]

[vrothberg]

Here's an example how to test it:

[edsantiago]

I'm obviously doing something wrong; I can't get my skopeo to recognize a simultaneous pull:

$ git show
commit dd70b2a9f846f45f4381d45d4df36e7d77d4db9c (HEAD -> pr/619)
...
$ make binary-local
go build "-buildmode=pie" -ldflags "-X main.gitCommit=dd70b2a9f846f45f4381d45d4df36e7d77d4db9c" -gcflags "" -tags "   ostree " -o skopeo ./cmd/skopeo
$ sudo ./skopeo copy docker://fedora:29 containers-storage:fedora:esm1   [and, in other window, esm2]

In all my attempts, both skopeos pull the entirety of 01eb078129a0. Do you see any obvious problems with my setup?

[vrothberg]

@edsantiago my apologies! As @mtrmac pointed out here containers/image#607 (comment), it's the code not you. I simply missed committing the last s/false/true/ to the PR at containers/image. Sorry! I shouldn't commit at the end of the day 🙉

[edsantiago]

That works better, thank you!

The word 'stopped' is pretty strong; would you consider 'paused' instead? And perhaps 'being copied' to indicate ongoing activity?

[vrothberg]

That works better, thank you!

The word 'stopped' is pretty strong; would you consider 'paused' instead? And perhaps 'being copied' to indicate ongoing activity?

Sounds great, thanks!

[vrothberg]

@rhatdan @mtrmac @nalind @edsantiago, please have another look:

• storage: store: add digest locking storage#307
• image: Do not merge - Copy detection image#607

Besides several cleanups of the code, I performed the following changes:

• Storage does not expose digest-specific Lock() - Unlock() API anymore but has a GetDigestLock(path string) (Locker, error) API that we use in storageImageDestination.
• File descriptors do not leak anymore as the lock files are now opened and closed on demand. We're already reference counting in the lockfile implementation. When Unlocking the last remaining reference on a lock we're closing the file descriptor.
• The digest locks are being tracked in storageImageDestination and all locked locks (but not yet unlocked ones) are unlocked in storageImageDestination.Close() which is being invoked in a deferred call in copy.Image() and effectively avoids deadlocks in the precense of errors or panics.

[vrothberg]

Closing in favour of containers/image#611.