Refactor the optimizer based on containerd

[ktock]

#223
#230
#238

This refactors optimizer based on continerd and fanotify. Options of ctr-remote image optimize are unchanged except it now accepts --oci option and multiple --platform options. Now ctr-remote image optimize requires containerd daemon up-and-running but doesn't require FUSE. CAP_SYS_ADMIN is needed for fanotify and cloning a mount namespace.

The design is the following:

1. *analyzer.Analize() creates a mount namespace and monitor "/" mountpoint of that namespace with fanotify-based notifier process (ctr-remote notify).
2. Then *analyzer.Analyze preapres snapshots and container spec based on the specified option. Then it runs a container in that mount namespace created in step 1 (notifier process is invisible to the container). Notifier sends all accessed files to the optimizer over stdio.
3. nativeconverter converts layers based on these accesssed files (= prioritized files).

NOTEs:

• The reason we pre-create mount namespace and let the container use it is that fanotify doesn't notify file accesses across namespaces.
• Notifier monitors a single flat rootfs so it notifies accessed files to optimizer without layer index. Here, *analyzer.imageRecorder takes responsibility to find out the layer that the file notified belongs to.
  • Initially I thought if we can do fanotify for each lower layer of the overlay mount but fanotify doesn't seem to notify anything even accesses occur on the merged directory.

TODOs:

• Update tests
• Refactor codes
  • Step 2 can be refactored using containerd lib's options (containerd.WithXXXX) instead of analyzer.runtimeOpts.
• --wait-on-signal doesn't seem to work yet.
• add & update docs
• Measure the benchmark with eStargz optimized by this optimizer
• Add "Optimizer In Container" (following-up)
• Come up with better communication protocol between optimizer and notifier (following-up?).
• Enable nativeconverter (estargz) to specify per-layer option indexed by layer index (not by layer digest) because it's possible that the same two layers in a single image have same digests (following-up?).

cc @AkihiroSuda

[AkihiroSuda]

s/notify/fanotify/ might have more clarity

[AkihiroSuda]

Thanks, design SGTM 👍

[AkihiroSuda]

nit: CentOS is deprecated

[ktock]

@mc256 Sorry for the slow reply. I implemented --wait-on-signal for non-terminal mode. PTAL.

[ktock]

Benchmarking result with images generated by this new optimizer seems fine.

https://github.com/ktock/stargz-snapshotter/runs/1789973644

[AkihiroSuda]

Merging, but test script should show error on non-amd64