Layer metadata not getting cached after upgradiing to v2

[Hemant28codes]

I am migrating from libcnb from v1(1.30.4)->v2(2.0.0). Following the migration doc we have removed layerContributor from our codebase.

Now we are trying to do pack build 2 times. Second time we are not passing "--clear-cache" flag and expects layer metadata that would have been cached to get reused when we ran it for the second time. But what we see is when we ran for the first time the metadata did got set in the layer but when we tried to retrieve it in the second run we were getting empty metedata.

We have only made changes corresponding to layercontributor removal and also upgraded buildpacks api version to 0.9.

Tasks

Preview Give feedback

No tasks being tracked yet.

[dmikusa]

I'm not sure that I totally follow the issue that you're having, but previously the LayerContributor was typically where you applied layer caching, so the removal of that may be causing the caching problem you're seeing.

Aside from that, I would suggest putting together a small sample that demonstrates the issue. I can take a look at that and try to give you further advise.

[Hemant28codes]

func (ctx *Context) Layer(name string, opts ...layerOption) (*libcnb.Layer, error) {
	if strings.Contains(name, "/") {
		return nil, buildererror.Errorf(buildererror.StatusInternal, "%v is an invalid layer name; layer names may not contain '/'", name)
	}
	l, err := ctx.buildContext.Layers.Layer(name)
	if err != nil {
		return nil, buildererror.Errorf(buildererror.StatusInternal, err.Error())
	}
	if err := ctx.MkdirAll(l.Path, layerMode); err != nil {
		return nil, buildererror.Errorf(buildererror.StatusInternal, "creating %s: %v", l.Path, err)
	}
	for _, o := range opts {
		if err := o(ctx, &l); err != nil {
			return nil, err
		}
	}
	if l.Metadata == nil {
		l.Metadata = make(map[string]interface{})
	}
	ctx.buildResult.Layers = append(ctx.buildResult.Layers, layerContributor{&l})
	return &l, nil
}

The above mentioned function is what we are using for creating our layer. In options it is - build, cache or launch options. Now as a part of migration we have changed this

ctx.buildResult.Layers = append(ctx.buildResult.Layers, layerContributor{&l})
to
ctx.buildResult.Layers = append(ctx.buildResult.Layers, l) .

Now when we do pack build for the first time layer metadata is getting set through:

func (ctx *Context) SetMetadata(l *libcnb.Layer, key, value string) {
	l.Metadata[key] = value
}

and during second run we expect that we should get this cached metadata. This is the corresponding logic:

func (ctx *Context) GetMetadata(l *libcnb.Layer, key string) string {
	v, ok := l.Metadata[key]
	if !ok {
		return ""
	}
	s, ok := v.(string)
        return s
}

But we see that metadata on the second execution is coming empty.

We tried logging layer after first execution of pack build and before second execution of pack build.

After first execution we see the following layer details which shows metadata getting set:

{"Build":true,"Cache":true,"Launch":false,"Metadata":{"stack":"xyz","version":"1.22.10"},"Name":"go","Path":"/layers/runtime/go","BuildEnvironment":{},"LaunchEnvironment":{},"SharedEnvironment":{},"Exec":{"Path":"/layers/runtime/go/exec.d"}}

And before second execution we see metadata empty:

{"Build":true,"Cache":true,"Launch":false,"Metadata":{},"Name":"go","Path":"/layers/runtime/go","BuildEnvironment":{},"LaunchEnvironment":{},"SharedEnvironment":{},"Exec":{"Path":"/layers/runtime/go/exec.d"}}

[Hemant28codes]

Hi @dmikusa is there an update on how should i proceed further with this?

[dmikusa]

Sorry, had not had a chance to look at this.

l, err := ctx.buildContext.Layers.Layer(name)

If metadata exists for a layer, then this will load it (see https://github.com/buildpacks/libcnb/blob/main/layer.go#L179-L182). Since it's not presently, I can only assume that the file doesn't have any metadata.

The lifecycle is responsible for restoring metadata, so you'd have to look at that more closely to see why it might not be restoring it. You have build + cache, so I think that would typically be correct to get layer metadata restored. I'm only seeing part of the puzzle, so I could be missing something.

What I would suggest is to do some runs with pack build -v -e BP_LOG_LEVEL=debug ... This will turn up debugging to the max from libcnb and should give you more helpful information about the state of things as the buildpacks run.

[Hemant28codes]

I investigated and found that layer metadata writing step is writing metadata as empty -

libcnb/build.go

Line 291 in 986aa1f

config.logger.Debugf("Writing layer metadata: %s <= %+v", file, layer)

With libcnb v1 we were getting this in logs while writing metadata for a layer-

Writing layer metadata: /layers/runtime/go.toml <= {LayerTypes:{Build:true Cache:true Launch:false} Metadata:map[stack:22 version:1.22.10] Name:go Path:/layers/runtime/go BuildEnvironment:map[] LaunchEnvironment:map[] SharedEnvironment:map[] Profile:map[] Exec:{Path:/layers/runtime/go/exec.d}}

With libcnb v2 we were getting this in logs while writing metadata for a layer-

Writing layer metadata: /layers/runtime/go.toml <= {LayerTypes:{Build:true Cache:true Launch:false} Metadata:map[] Name:go Path:/layers/runtime/go BuildEnvironment:map[] LaunchEnvironment:map[] SharedEnvironment:map[] Exec:{Path:/layers/runtime/go/exec.d}} 

We see that if we are just printing the layer after call our SetMetadata function we see that metadata is present in the layer -

{"Build":true,"Cache":true,"Launch":false,"Metadata":{"stack":"xyz","version":"1.22.10"},"Name":"go","Path":"/layers/runtime/go","BuildEnvironment":{},"LaunchEnvironment":{},"SharedEnvironment":{},"Exec":{"Path":"/layers/runtime/go/exec.d"}}

But at the same time we see that this Writing metadata step mentioned here - link shows metadata is empty.

[Hemant28codes]

This comment - link looks similar to this issue. Could you please provide some additional details mentioned in this comment?

[jkutner]

Would it be possible to create a simple repo that reproduces the problem so that I can run it locally with the Google buildpacks that use v2?

[Hemant28codes]

@jkutner Here is the GoogleCloudPlatform Buildpacks with libcnb v2 changes -
https://github.com/Hemant28codes/buildpacks.

[dmikusa]

Ok, summarizing a bit and adding some questions.

1. You mentioned earlier that the first build works OK. The image is generated correctly & metadata is being handled as expected. Did you confirm this with verbose logging turned on?

2. You mentioned earlier that it was on the second build that you saw the metadata not behaving as expected. Your comment here, is this from the first run or second run? If this is dependent on the build number and it is at all possible, please include the full build logs from both the first and second build with the verbose logging enabled.

3. If you are setting metadata on a layer and then it's writing the metadata out empty that should have nothing to do with the state of the metadata being restored or the number of times you run/rerun a build. When you set the metadata on the layer, it should stick and be written out. Can you confirm that you can reproduce the issue in that way? Independent from first/second build, and by simply setting metadata on a layer and writing that layer out.

4. I looked at the repo you linked, but there's too much going on there for me to go through (unless you're looking for a consultant 😄). What I need is a concise reproduction case or unit test that I can run to reproduce the problem. I don't mind if it's a few steps, just provide the steps and what I need to do with the test case.

Thanks

[Hemant28codes]

So the error was with removing the layercontributor from our codebase, there was a reference issue. So the layer on which libcnb was operating and the one where we were trying to set metadata was pointing to different layer as layercontibutor used to expect &layer and now it accepts layer directly. Its fixed now.

[dmikusa]

Fantastic! Thanks for following up.