Fix long container startup times #763
Conversation
The use of a uid map leads to a new layer with all files chowned. This takes several seconds due to the size of the instructlab container (26GB). Normally this would be a one time cost where the idmap layer is cached and reusued accross container creations; however, since the container is stored on a read-only additional image store, no caching is performed. Address the problem by creating a derived empty contianer in mutable container storage. This allows the 1k idmap layer to be created in the smae area, yet reuses the layers in additional image store. Signed-off-by: Jason T. Greene <[email protected]>
n1hility
requested review from
MichaelClifford,
rhatdan,
sallyom,
lmilbaum,
cgwalters and
Gregory-Pereira
as code owners
|
PTAL @Gregory-Pereira @lmilbaum , another high priority fix // cc @rhatdan |
| if [ "$?" != "0" ]; then | ||
| echo "Initializing ilab container..." | ||
| id=$(sudo podman create "$SOURCE_IMAGE") | ||
| sudo podman commit "$id" "$IMAGE_NAME" |
There was a problem hiding this comment.
This is creating a new layer in /var/lib/containers that references the base layers, but those layers will go away across an upgrade when the base ilab image changes, right?
IOW won't this break across upgrades?
There was a problem hiding this comment.
@cgwalters As long as we are updating our tag (1.1 today -> 1.2 tomorrow) this should be ok, it will then just create a new container at that point. We do need to add code to cleanup old images though (1.2 will need to delete the old 1.1 container)
|
Backing up a second, we should support kernel native idmapping and shouldn't need to chown I think... |
|
I don't think idmapping was supported in RHEL for 9.4. |
|
Right yeah we are on 5.14 and this was added to overlay in 5.19 right? |
|
Not sure of the numbers but sounds right. |
|
OK yes I guess overlayfs idmapping is only going to be for RHEL10 so far unless further backports happen. |
The use of a uid map leads to a new layer with all files chowned. This takes several seconds due to the size of the instructlab container (26GB). Normally this would be a one time cost where the idmap layer is cached and reusued accross container creations; however, since the container is stored on a read-only additional image store, no caching is performed.
Address the problem by creating a derived empty contianer in mutable container storage. This allows the 1k idmap layer to be created in the smae area, yet reuses the layers in additional image store.