Skip to content

Commit

Permalink
make layer-tree lookup errors non-fatal
Browse files Browse the repository at this point in the history
Internally, Podman constructs a tree of layers in containers/storage to
quickly compute relations among layers and hence images.  To compute the
tree, we intersect all local layers with all local images.  So far,
lookup errors have been fatal which has turned out to be a mistake since
it seems fairly easy to cause storage corruptions, for instance, when
killing builds.  In that case, a (partial) image may list a layer which
does not exist (anymore).  Since the errors were fatal, there was no
easy way to clean up and many commands were erroring out.

To improve usability, turn the fatal errors into warnings that guide the
user into resolving the issue.  In this case, a `podman system reset`
may be the approriate way for now.

[NO TESTS NEEDED] because I have no reliable way to force it.

[1] containers#8148 (comment)

Signed-off-by: Valentin Rothberg <[email protected]>
  • Loading branch information
vrothberg committed Feb 12, 2021
1 parent 745fa4a commit 6140369
Showing 1 changed file with 13 additions and 2 deletions.
15 changes: 13 additions & 2 deletions libpod/image/layer_tree.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (

ociv1 "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
)

// layerTree is an internal representation of local layers.
Expand Down Expand Up @@ -84,7 +85,12 @@ func (ir *Runtime) layerTree() (*layerTree, error) {
}
node, exists := tree.nodes[topLayer]
if !exists {
return nil, errors.Errorf("top layer %s of image %s not found in layer tree", img.TopLayer(), img.ID())
// Note: erroring out in this case has turned out having been a
// mistake. Users may not be able to recover, so we're now
// throwing a warning to guide them to resolve the issue and
// turn the errors non-fatal.
logrus.Warnf("Top layer %s of image %s not found in layer tree. The storage may be corrupted, consider running `podman system reset`.", topLayer, img.ID())
continue
}
node.images = append(node.images, img)
}
Expand All @@ -107,7 +113,12 @@ func (t *layerTree) children(ctx context.Context, parent *Image, all bool) ([]st

parentNode, exists := t.nodes[parent.TopLayer()]
if !exists {
return nil, errors.Errorf("layer not found in layer tree: %q", parent.TopLayer())
// Note: erroring out in this case has turned out having been a
// mistake. Users may not be able to recover, so we're now
// throwing a warning to guide them to resolve the issue and
// turn the errors non-fatal.
logrus.Warnf("Layer %s not found in layer. The storage may be corrupted, consider running `podman system reset`.", parent.TopLayer())
return children, nil
}

parentID := parent.ID()
Expand Down

0 comments on commit 6140369

Please sign in to comment.