Do we need digests for Lima templates

[jandubois]

In #3009 (comment) @AkihiroSuda wrote:

[This has to support specifying sha256 digest]

base:
  location: "https://example.com/a.yaml"
  digest: "sha256:deadbeef"

As a side topic, limactl start https://... should also have a flag like --template-digest=sha256:deadbeef

I find this cumbersome to use. I think if you want a locator for an immutable template, you should use a content-addressable store to fetch it from instead of requiring an additional digest.

Trust assumptions:

• DNS will provide the correct IP address for github.com
• TLS will verify that https://github.com content originates from GitHub
• Git commit ids cannot be forged

Then you can use a raw commit id to point to a specific version of the template, e.g. to run the Lima 1.0.2 release version of template://fedora:

$ limactl start --tty=false https://raw.githubusercontent.com/lima-vm/lima/7cea6f851656ee9aac606d02ea055d49d1c41cf2/templates/fedora.yaml

We could add another scheme to download templates from OCI registries, but I would not spend any effort on it unless somebody really needs it.

These URLs will work fine with turning relative locators into absolute ones.

So I don't think we need any explicit support for checksumming template files.

---

As for using checksums for individual basedOn or file references, I think this is totally impractical. If this is important, then you would run limactl template assemble (or whatever we end up calling it) to embed everything into a single file, and then store a checksum for that.

[AkihiroSuda]

Then you can use a raw commit id to point to a specific version of the template,

Not every template is located on GitHub, so the digest should still make a sense for non-GitHub URLs

[afbjorklund]

You could use ipfs://bafybeib2irmeecuu74t3znqkvitir7yqt3tfv5l4527eisiklgslarzo6q/fedora.yaml

But then you would still have to fall back on a gateway, so same problem as with OCI references*.
http://127.0.0.1:8080/ipfs/bafybeib2irmeecuu74t3znqkvitir7yqt3tfv5l4527eisiklgslarzo6q/fedora.yaml

* suddenly we can't curl anymore, so have to wrap json parsers and whatnot (or run an oras gateway)?

[afbjorklund]

This seems to be an interesting take on the "curl pipe sh" debate, currently stuck in "curl + review + sh" limbo

https://www.tumblr.com/curlpipesh

https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script

I suppose one could use IPFS, but that is as cumbersome as ORAS. Some people use anchors as a workaround.

[afbjorklund]

Then again it already has too many (unused) flags.

• [feature] Checksum when downloading file curl/curl#1399

[AkihiroSuda]

Sent a PR to GNU coreutils instead of curl:

curl https://example.com/install.sh | sha256sum --check-stream deadbeef | sh

https://lists.gnu.org/archive/html/coreutils/2024-12/msg00005.html

[afbjorklund]

This feature was actually available in the BSD sha256, with the -c flag (no long opts).

$ sha256 -c dd0cdb334df053a5016a8665d9f52279a6550400cc7e466b6a6a9dabc6b8cd70 fedora.yaml
SHA256 (fedora.yaml) = dd0cdb334df053a5016a8665d9f52279a6550400cc7e466b6a6a9dabc6b8cd70

https://github.com/afbjorklund/bsdsum

[afbjorklund]

Currently I was using separate files for verification, which worked for everything but BuildKit (SBOM, sigh)

https://github.com/afbjorklund/lima-the-hard-way/blob/main/sha256sums.txt (as in sha256sum -c)

     -c string
             If the program was called with a name that does not end in sum,
             compare the digest of the file against this string.  If combined
             with the -q option, the calculated digest is printed in addition
             to the exit status being set.  (Note that this option is not yet
             useful if multiple files are specified.)

     -c file
             If the program was called with a name that does end in sum, the
             file passed as argument must contain digest lines generated by
             the same digest algorithm with or without the -r option (i.e., in
             either classical BSD format or in GNU coreutils format).  A line
             with the file name followed by a colon “:” and either OK or
             FAILED is written for each well-formed line in the digest file.
             If applicable, the number of failed comparisons and the number of
             lines that were skipped since they were not well-formed are
             printed at the end.  The -q option can be used to quiesce the
             output unless there are mismatched entries in the digest.

[afbjorklund]

Also implemented different output formats, which was another one of those rejected features...

Known encodings: none, base16, base32, base32hex, base58btc

[jandubois]

Then again it already has too many (unused) flags.

That is my concern: it is just another option that clutters up the --help output that nobody will ever use.

And it is not that the issue cannot be solved easily with a few lines of shell script, so I don't feel the need to provide a shortcut for a thing that nobody asked for.

[jandubois]

I think I figured it out: the checksums should be an optional suffix on the template locator:

$ limactl create https://example.com/templates/our.yaml@deadbeef

This means we don't need extra parameters in the UI, and no extra fields in the implementation; the checksums ride for free in the locator string.

And while I still don't think anybody will use this, it automatically provides for this:

basedOn: template://fedora@feedc0de
provision:
- file: script.sh@decafbad

If a locator ending in a @digest is passed to limatmpl.Read() then it will return an error if the fetched content doesn't match the checksum.

Things to bike-shed:

• do we need to specify the checksum algorithm, even though we will only support sha256, e.g. script.sh@sha256:decafbad?
• do we allow abbreviated checksums, or do we require the full length?

My bike-shed is blue with pink polka dots:

• we don't include the redundant sha256: part
• we allow checksums to be abbreviated down to 7 hex characters, but no less

Having a compact format makes things easier to read/write without line wrapping, and 7 characters are plenty protection against accidental collision. People worrying about malicious changes can use the full checksum.

@AkihiroSuda WDYT?

[jandubois]

I've implemented this in c288fcb. It does allow you to optionally specify an algorithm, but otherwise matches what I wrote above.

[AkihiroSuda]

URL can't be extended in this way.

"@" has been a portion of the URL (Appears in Twitter, Mastodon, and others)

[afbjorklund]

It made sense for things like OCI images (@sha256:123abc...), but as mentioned I don't think it is "allowed" for URI?

Other systems do other hacks like making it into anchors (#) or parameters (&) or whatever. Still confuses some tools.

But I didn't think the digest was needed for the "includes" either.

• Allow providing scripts as separate files #2442

[jandubois]

Other systems do other hacks like making it into anchors (#) or parameters (&) or whatever. Still confuses some tools.

The discussion has mostly moved to #3021 now, but the concern about "some tools" is irrelevant as limactl is the only tool that understands template locators, especially template://.

The way to download a template from any kind of template locator is (well, will be) limactl tmpl copy, which automatically supports digest suffixes once limatmpl.Read() implements them:

$ limactl tmpl copy my.yaml@sha256:60a8737145 -
basedOn:
- fedora.yaml
…
$ limactl tmpl copy my.yaml@sha256:60a8737145ff -
FATA[0000] locator "my.yaml@sha256:60a8737145ff" digest doesn't match content digest "60a87371451e"

[afbjorklund]

Sounds like yet another gateway is needed :-)

[AkihiroSuda]

Git commit ids cannot be forged

It can be sort of "forged", in the GitHub URL:

$ curl https://raw.githubusercontent.com/lima-vm/lima/79529f879b566f02eacd705e1f8746ea7cb37a5b/DEMO-3019
Demo file for <https://github.com/lima-vm/lima/discussions/3019>.

This file will be visible via <https://raw.githubusercontent.com/lima-vm/lima/.../DEMO-3019>.

However, this file does not really belong to the <https://github.com/lima-vm/lima> repo.

So, {"location": "https://raw.githubusercontent.com/ORG/REPO/TAG/FILE": "digest": "DIGEST"} is more trustworthy than https://raw.githubusercontent.com/ORG/REPO/COMMIT_HASH/FILE

[jandubois]

It can be sort of "forged", in the GitHub URL:

This did not forge the commit id. You would have to create a commit with the SHA1 of 7cea6f8 to replace the content from my link. And given that a commit with that hash already exists in the repo, I'm not sure if you could actually push it, and even if you can, if GitHub would then prefer it over the existing one.

As for "does not really belong to", that is not exactly correct, as all the forks of a repo are really only branches in the same repo on the backend. So every commit exists in the original repo and all the forks simultaneously.

That is also the reason you can't make a fork of a public repo private: if a private commit id were to leak, the data would become accessible through the public main repo.

You can of course always create a new private repo, and then push the branches from the public repo to it. That way it is not a fork, but an independent copy (and you can't do pull requests between them).

[jandubois]

I realized that the issue is more complex than just calculating digests of the template files:

provision:
- file: script.sh

Even if I have a digest for this template, it doesn't tell me if the script.sh has changed since the digest was first calculated. So hypothetically the template could include a digest for script.sh, but it doesn't, and I can't modify it.

So an obvious improvement would be to calculate the digest not on the raw template, but once it has been assembled (all dependencies have been embedded).

But that digest is still a bit brittle: if Lima is updated, and either yq or yamlfmt has some subtle change in what it outputs, the digest would now change, even though the underlying files remain the same.

Another issue would be if the template in turn depends on one of our distro templates, e.g. template://ubuntu. Every time we update the images in that template the digest of the assembled user template changes as well.

And finally, if the template creation takes --set or --param options, the digest would only match if you run an identical command.

basedOn: "template://{{.Param.distro}}"
param:
  distro: ubuntu

Now I want to run

limactl create --param distro=fedora https://templates.r.us/dynamic.yaml --digest=???

It may end up that we'll have to say: "If you want to use template digests, you have to use single-file templates, or you have to store the assembled template somewhere you control, and reference from there".

So you could do something like

limactl tmpl copy --embed --param distro=fedora https://templates.r.us/dynamic.yaml my-fedora.yaml
sha256sum my-fedora.yaml

and then share the assembled template from your own web property. That way it will have a static digest.

Anyways, this turns into a rabbit hole, and I think will be much easier to discuss once we have played with the mechanism a bit, and written a couple of multi-file templates, and seen what works well, and what doesn't.

So let's revisit this in a couple of month...

[jandubois]

Since keeping digest of many small files updated is such a chore¹, I've been wondering about alternatives. Some way how you can sign both YAML and script files, and template assembly would then be able to verify that all pieces either have a matching digest, or are signed by a trusted party (like the Lima developers).

I think I can figure out the signing part of the YAML and shell scripts (unless you tell me that the YAML files need to comply with the YAML spec to the letter and we can't create our own directive because they are all reserved).

But the problem is the the signing tool, and the list of trusted parties. This sounds like GPG is the only option, and with that we have already lost 95% of the potential users. 😞

So, I currently don't see a way to make it work, but if you have some ideas...

Footnotes

1. Look at the pain we went through updating distro images. Every time we update, users will have to update digests for using these as base templates. ↩

[afbjorklund]

The signing part and bottleneck was the biggest problem with Zero Install, as long as upstream did not get involved / on board.

The actual signature was handled quite neatly by a data document though, and I thought I would bring that from XML to YAML?

https://docs.0install.net/details/#security-note

And yes, it did use embedded GPG signatures...

---

Everyone else started talking about cosign instead, but it was a pain to use - I thought?

https://kubernetes.io/docs/tasks/administer-cluster/verify-signed-artifacts/

The end result is that no-one/nothing is checking signatures, and the signing is left in limbo.

kubernetes/release#913 - eventually "solved" by a third party, by using OBS

[AkihiroSuda]

A bike shedding for another microformat: digest://deadbeef?url=https://example.com/a.yaml
(URL portion needs to handle escape chars)

[jandubois]

A bike shedding for another microformat: digest://deadbeef?url=https://example.com/a.yaml

That would work as a format, but I continue to think that per-file digests are not really practical (see below) and hope that alternate mechanisms will work better.

[jandubois]

(URL portion needs to handle escape chars)

I glossed over this part, but if the locator has to be written as digest://deadbeef?url=https%3A%2F%2Fexample.com%2Fa.yaml then that doesn't really work for me.

In that case it will be better to add yet another custom marshaller and optionally allow:

base:
- locator: https://example.com/a.yaml
  digest: deadbeef

I guess that also means that we will never add additional info to template locator strings, and they will always be URLs, in which case we could use url instead of locator.

[jandubois]

I would like to restart this discussion from actual use cases. I think digests are generally only needed to verify that a resource under external control is not being changed without the consumer being aware of it.

Example template

The Example company provides the https://example.com/foo.yaml template for external users to consume:

basedOn: example.yaml
provision:
- script: foo.sh

All the Example company templates are based on the https://example.com/example.yaml base template:

basedOn: template://ubuntu
provision:
- file: example.sh

The shell script content doesn't matter here.

Use-case 1: Use Example's foo.yaml as the base for local use

I have my-foo-tmpl.yaml locally that uses Example's foo.yaml as the base:

basedOn: https://example.com/foo.yaml
provision:
- file: my.sh

I've reviewed and approved foo.yaml, example.yaml, foo.sh, and example.sh.

I want to be confident in using my-foo-tmpl.yaml in the future to create additional VMs and be sure the upstream dependencies have not changed in an undesirable way.

Per-file checksums are not going to work

I'm happy to include a checksum in my-foo-tmpl.yaml:

basedOn:
- locator: https://example.com/foo.yaml
  digest: sha256:deadbeef…

The problem is that this only guarantees that foo.yaml hasn't changed. Example company can update foo.sh, example.yaml, or example.sh and I would never know.

I tried to convince Example company to use checksums in their template files, but they refused: "We don't want to update 50 templates with updated checksums every time we make a change to the example.yaml base template."

So in the situation where the verification is needed the most (no existing trust relationship with the template provider), the per-file checksum mechanism is least likely to work.

Solution 1: If possible use a version controlled locator

Sometimes the templates may be hosted by an independent trusted party under a content-addresseable scheme, e.g. as a GitHub commit:

https://raw.githubusercontent.com/example/templates/0e0227063cc0416d879dcfca7cf33484064436e5/foo.yaml

In that case I can reference the base template using that URL. Due to the way relative locators work, I can be sure that all referenced files will be from the same commit as well, so the complete base template is now immutable¹.

However, this is not always possible.

Solution 2: Store the base template locally

I can always make a local copy that embeds all dependencies:

$ limactl tmpl copy --embed my-foo-tmpl.yaml my-foo.yaml

This will embed all dependencies in my local copy my-foo.yaml.

It has several advantages:

• all dependencies are now local and cannot be modified by Example company
• the template remains working even if Example is offline, deletes the template, or makes incompatible changes

For practical purposes storing my-foo.yaml instead of a digest makes no big difference; they are both relatively small.

Problem: template://ubuntu is also embedded

While I want to pin down all upstream dependencies, I would still like to be able to move to a new Ubuntu base image without having to update my-foo.yaml.

This can be achieved by adding a new limactl copy option, e.g. by moving the current --embed functionality to --embed-all, and have --embed not embed any trusted templates (the ones using template:// locators, or the $LIMA_HOME/_config/base.yaml one).

This requires that template:// locators are only used at the final level. E.g. it would work with

basedOn: [variables.yaml, my.yaml, template://docker]

but not with

basedOn: [variables.yaml, template://docker, defaults.yaml]

because we can't embed defaults.yaml without also embedding template://docker.

I can't think of a situation where this ordering would be useful/required, so this doesn't seem like a limitation.

Use-case 2: Reference the template in a README or on a blog

Solution 1 above would work too, but is not always an option.

Solution 2 is awkward; I would have to publish the copied template to e.g. a GitHub repo, to turn this back into solution 1. I can't just publish the combined template on any random URL because the end user (reader of the blog or README) would be left with the same issue of making sure the template doesn't change (now they would have to trust me instead of Example company).

Including the merged template in the blog or document is impractical.

Solution 3: Provide a checksum of all files embedded in the template

A digest of just foo.yaml is not enough, but we could compute a checksum of the fully embedded template:

$ limactl tmpl digest --all https://example.com/foo.yaml
sha256:decafbad…
$ limactl create https://example.com/foo.yaml --template-digest-all sha256:deadbeef…
ERRO[0000] Template digest does not match

The digest would be computed over all files:

$ ( curl https://example.com/foo.yaml;
>   curl https://example.com/example.yaml;
>   limactl tmpl copy --verbatim template://ubuntu -;
>   curl https://example.com/example.sh;
>   curl https://example.com/foo.sh
> ) | sha256sum | awk '{print "sha256:" $1}'
sha256:decafbad…

When using limactl tmpl digest without the --all option, the template://ubuntu template would have been omitted, and the limactl start option would have been just --template-digest.

Any $LIMA_HOME/_config/base.yaml template would have been excluded in either case².

Required changes

• rename limactl tmpl copy --embed to limactl tmpl copy --embed-all
• add limactl tmpl copy --embed option that does not embed template:// locators
• add limactl tmpl digest subcommand with --all option
• add --template-digest and --template-digest-all options to the limactl create (and limactl start) commands

The first 2 changes should be part of #3072; the latter 2 should be in a followup.

Other usecases?

Are there any other use cases that would not be covered by solutions 2 and 3?

Footnotes

1. This is true in the example here, but would not be true if there were https:// template references in foo.yaml or example.yaml. In that case solution 1 will not work. ↩

2. There is some inconsistency that limactl tmpl copy --embed-all would embed $LIMA_HOME/_config/base.yaml, but limactl temple digest --all and limactl create --template-digest-all would ignore it, that should be discussed further once there is agreement on the general proposal. ↩

[AkihiroSuda]

From now on I will refer to "per-file digests" as "solution 4".

Could you show how the yaml and the CLI will look like?

The question is if we should require some minimal length, like 7 hex digits, and what the limit should be.

7 hex digits can be easily cryptomined (16**7=268,435,456)

[jandubois]

From now on I will refer to "per-file digests" as "solution 4".

Could you show how the yaml and the CLI will look like?

For solution 4 (which I believe is what you are asking for) I would use the syntax I suggested in #3019 (reply in thread):

base:
- url: https://example.com/foo.yaml
  digest: deadbeaf
provision:
- mode: system
  file:
    url: example.sh
    digest: decafbad

If you don't want to use digests then you can still write it as

base: https://example.com/foo.yaml
provision:
- mode: system
  file: example.sh

And from the CLI you could use it directly with

limactl create --digest deadbeaf https://example.com/foo.yaml

I still think this is a feature that nobody will use, but if you feel strongly that we must have it anyways, then I will add it. Using the custom marshaller at least only the people who are going to use it will have to pay the price of the additional complexity.

Note that --digest will still function the way I explained above: it will add the content of every referenced base template or script, but will ignore those that are specified with a digest: property, or that use a template:// locator.

That way the same option automatically does the right thing for both solution 3 and solution 4.

The question is if we should require some minimal length, like 7 hex digits, and what the limit should be.

7 hex digits can be easily cryptomined (16**7=268,435,456)

If I'm concerned about malicious hash collisions, then I should of course use the full length and not abbreviate at all. But if I only want to make sure the template hasn't been updated, then the shorter hash is sufficient. Just like you can use abbreviated hashes with git.

I could see an argument for always requiring the full hash when specified in a digest: property inside the YAML file, and allowing the abbreviated form only for the CLI --digest option, because that is more likely to be typed manually.

[AkihiroSuda]

Note that --digest will still function the way I explained above: it will add the content of every referenced base template or script, but will ignore those that are specified with a digest: property, or that use a template:// locator.

This should be perhaps called something like --digest-interpolated?
I expect --digest to be the raw digest of https://example.com/foo.yaml .

[jandubois]

This should be perhaps called something like --digest-interpolated?

No, the short form must be reserved for the common use case. You could use --digest-raw for the other form, but it isn't needed. It will just work the right way automatically.

I expect --digest to be the raw digest of https://example.com/foo.yaml .

It will be the raw digest as long as the template only has template references that either include a digest, or are of the template:// form (or no references at all).

It will only include additional files if you specify them without a digest internally. In which case using a raw digest doesn't make any sense anyways. This is better than allowing an insecure digest. The alternative would be to throw an error. Which you will get here too implicitly if you specify a raw digest for a template that is "insecure" because the digest will not match.

[jandubois]

The alternative would be to throw an error. Which you will get here too implicitly if you specify a raw digest for a template that is "insecure" because the digest will not match.

We can make this more obvious by listing all the locators that went into the digest when there is a mismatch, something like

$ limactl create --digest deadbeaf https://example.com/foo.yaml
ERRO[0000] Digest `deadbeef` does not match actual digest `decafbad`, calculated from https://example.com/foo.yaml, https://example.com/example.yaml, https://example.com/example.sh, https://example.com/foo.sh

This assumes that you were unsuccessful in convincing Example company to include digest: properties inside their template files.

The error message makes it clear that the digest is calculated from multiple files, and when you look at them, you will see why.

If the templates did contain digests, but the --digest value is incorrect, then the error would be different:

$ limactl create --digest deadbeat https://example.com/foo.yaml
ERRO[0000] Digest `deadbeef` does not match actual digest `12345678`, calculated from https://example.com/foo.yaml

Or maybe don't include the URL at all if only one file was used to calculate the digest:

$ limactl create --digest deadbeat https://example.com/foo.yaml
ERRO[0000] Digest `deadbeef` does not match actual digest `12345678`

Either way, a raw digest should not be allowed to match if the template document contains further locators that are not protected by digest; it would just give a false sense of security.