Add builtin yq support to lima start command #1359

afbjorklund · 2023-02-12T09:13:42Z

Allows modifying the yaml template "inplace" for a single instance,
if you don't want to change the default or override for all instances.

This is similar to running yq on the template, from the command-line.
Except it does it on a temporary file, and for all different kinds of templates.

$ yq < examples/default.yaml > default.yaml
$

Currently it removes empty lines, some quirk with the yaml parser used.
It also seems to change the indentation on some comments (only), but OK:

$ diff -B -w examples/default.yaml default.yaml
$

This can be used as a general fallback, similar to jq, for modifying templates.
Instead of adding the usual --cpus --memory --arch, or other parameters ?

See https://github.com/mikefarah/yq for details

Support command-line specification of VM configuration #545

limactl start config.yaml --yq '.arch = "aarch64"'

It is not YAML syntax, but it is rather straightforward.

pkg/yqutil/yqutil.go

pkg/yqutil/yqutil_test.go

AkihiroSuda · 2023-02-12T11:36:04Z

cmd/limactl/start.go

@@ -56,6 +60,7 @@ $ cat template.yaml | limactl start --name=local -
 	// TODO: "survey" does not support using cygwin terminal on windows yet
 	startCommand.Flags().Bool("tty", isatty.IsTerminal(os.Stdout.Fd()), "enable TUI interactions such as opening an editor, defaults to true when stdout is a terminal")
 	startCommand.Flags().String("name", "", "override the instance name")
+	startCommand.Flags().String("yq", "", "modify the template inplace")


Maybe:

Suggested change

startCommand.Flags().String("yq", "", "modify the template inplace")

startCommand.Flags().String("set", "", "modify the template inplace, using yq syntax")

I guess we have to prohibit modifying images?

Maybe we want to treat this as an experimental?

Please update README.md too

Why is it not allowed to modified images, from the templates ?

Might be ok for new instances, but not allowable for existing instances, obviously

Right. But this new tool is not very different, from doing it yourself in the editor...

AkihiroSuda

Thanks

AkihiroSuda · 2023-02-14T02:19:04Z

Instead of adding the usual --cpus --memory --arch, or other parameters ?

We should add them too, for typechecking, shell completion, etc. (In separate PRs)

jandubois

Thanks, LGTM.

I wish the option would also be added to limactl edit, but that could be done in a followup PR.

It also makes me a little sad that this little feature increases the size of the limactl binary by 20% (20MB → 24MB), but I guess that can't be helped, and doesn't really matter when you compare it to the size of QEMU. 😄

jandubois · 2023-02-14T05:36:38Z

README.md

+To create an instance "default" with modified parameters:
+```console
+$ limactl start --set='.cpus = 2 | .memory = "2GiB"'
+```
+


I wonder if we need to include a reference to the yq documentation, so people can learn to construct more powerful expressions for e.g. scripting:

$ limactl start --set '.env.HOSTPATH=strenv(PATH)' template://alpine

I expected the same syntax to be available for the limactl edit command as well.

I guess I should have specified: I would expect limactl edit --set ... to just edit the config, but not to invoke an editor.

It would just be a mechanism to modify the config programmatically without starting the instance.

We can use something like --eval, since that is the name of the yq command being invoked ?

Available Commands: completion Generate the autocompletion script for the specified shell eval (default) Apply the expression to each document in each yaml file in sequence eval-all Loads _all_ yaml documents of _all_ yaml files and runs expression once help Help about any command shell-completion Generate completion script

I agree that --set is somewhat misleading, and that --yq might be a bit cryptic for lima users...

Adding it to edit should be straight-forward.

I would expect limactl edit --set ... to just edit the config, but not to invoke an editor.

👍

--eval sounds much more misleading

limactl edit --set can be another PR if complicated

The error handling can be a bit tricky to do, so I'd like that.

pkg/yqutil/yqutil_test.go

afbjorklund · 2023-02-14T07:10:19Z

It also makes me a little sad that this little feature increases the size of the limactl binary by 20% (20MB → 24MB), but I guess that can't be helped, and doesn't really matter when you compare it to the size of QEMU. smile

I noticed that as well.

It would be possible to make a more conservative option, either some special-case for cpus/memory/arch/etc.
Or doing the --set approach, that would create a special override.yaml just for the particular instance.

This feature (yq) as such has a lot of "overkill" in it, like using jq (or even go templates) to change the list output...

AkihiroSuda · 2023-02-14T07:28:29Z

The binary footprint seems acceptable for me

jandubois · 2023-02-14T07:41:15Z

It would be possible to make a more conservative option,

No, I think I would prefer to have the expressiveness of yq expressions. I was just commenting that I noticed the size increase, but over time bloat is inevitable, and I think most users won't care (or even notice).

Idle curiosity: yq is 30× the size of jq

$ ls -lh /usr/local/Cellar/jq/1.6/bin/jq
-r-xr-xr-x  1 jan  admin   296K  2 Jan 18:54 /usr/local/Cellar/jq/1.6/bin/jq
$ ls -lh /usr/local/Cellar/yq/4.30.8/bin/yq
-r-xr-xr-x  1 jan  admin   9.0M 14 Jan 16:37 /usr/local/Cellar/yq/4.30.8/bin/yq

afbjorklund · 2023-02-14T07:43:48Z

It also does XML and JSON, in addition to just YAML. But I don't think it had any build flags ?

jandubois · 2023-02-14T07:46:09Z

It also does XML and JSON, in addition to just YAML. But I don't think it had any build flags ?

Of course it does JSON, as JSON is just a subset of YAML; every valid JSON file is also a valid YAML file. I didn't know yq also did XML though.

afbjorklund · 2023-02-14T07:56:39Z

I meant that it links with both encoding/json and encoding/xml libraries:

1813	gopkg.in/yaml.v3
1478	syscall
1083	encoding/json
994	encoding/xml
917	time
830	os
773	fmt
677	regexp
650	github.com/magiconair/properties
552	math
524	github.com/alecthomas/participle/v2/lexer
503	unicode
497	strconv
487	strings
430	gopkg.in/op/go-logging.v1
386	bytes
368	github.com/goccy/go-json
343	bufio
315	io
302	net/url
292	sort
289	github.com/jinzhu/copier
273	io/fs
269	github.com/a8m/envsubst/parse
250	github.com/fatih/color
205	math/rand
151	github.com/goccy/go-yaml/printer
135	encoding/csv
122	encoding/base64
94	container/list
84	github.com/elliotchance/orderedmap
81	golang.org/x/net/html/charset
55	github.com/dimchansky/utfbom
44	errors
14	github.com/goccy/go-yaml/lexer

Sizes in kiB (build-size).

afbjorklund · 2023-02-14T08:28:24Z

Idle curiosity: yq is 30× the size of jq

It is written in C (not Go), so this is normal.

AkihiroSuda · 2023-02-15T03:09:45Z

Needs rebase.

Are we ready to merge this after rebasing?

jandubois · 2023-02-15T03:20:47Z

Are we ready to merge this after rebasing?

I'm fine with merging.

Allows modifying the template inplace for a single instance, if you don't want to change the default or override for all instances. There are currently some quirks with the modified yaml syntax, like removing empty lines or adding indentation on sequences... Signed-off-by: Anders F Björklund <[email protected]>

afbjorklund · 2023-02-15T07:00:13Z

I ran go mod tidy on it, so I hope those dependencies (go sum) are correct...
The logging is a direct dependency, due to the "translation" over to logrus.

For what it is worth, I made a PR to yq in order to make json and xml optional.
So that should cut the library size in half, if merged. -tags yq_nojson,yq_noxml

AkihiroSuda

Thanks

AkihiroSuda reviewed Feb 12, 2023

View reviewed changes

pkg/yqutil/yqutil.go Show resolved Hide resolved

AkihiroSuda reviewed Feb 12, 2023

View reviewed changes

pkg/yqutil/yqutil_test.go Show resolved Hide resolved

AkihiroSuda reviewed Feb 12, 2023

View reviewed changes

pkg/yqutil/yqutil_test.go Show resolved Hide resolved

AkihiroSuda reviewed Feb 12, 2023

View reviewed changes

AkihiroSuda added this to the v0.15.0 milestone Feb 12, 2023

AkihiroSuda added enhancement New feature or request area/cli limactl CLI user experience labels Feb 12, 2023

afbjorklund force-pushed the start-yq branch from 09b2a29 to c886757 Compare February 12, 2023 14:48

AkihiroSuda added the impact/changelog label Feb 12, 2023

AkihiroSuda previously approved these changes Feb 14, 2023

View reviewed changes

AkihiroSuda requested a review from jandubois February 14, 2023 02:18

jandubois previously approved these changes Feb 14, 2023

View reviewed changes

afbjorklund dismissed stale reviews from jandubois and AkihiroSuda via 511ce35 February 15, 2023 06:57

afbjorklund force-pushed the start-yq branch from c886757 to 511ce35 Compare February 15, 2023 06:57

jandubois approved these changes Feb 15, 2023

View reviewed changes

AkihiroSuda approved these changes Feb 15, 2023

View reviewed changes

AkihiroSuda merged commit eb2a4cd into lima-vm:master Feb 15, 2023

amotl mentioned this pull request Feb 28, 2023

Support jq-like filtering and transformation grafana-toolbox/grafana-wtf#1

Open

afbjorklund mentioned this pull request Mar 8, 2023

Add builtin yq support to limactl edit command #1412

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add builtin yq support to lima start command #1359

Add builtin yq support to lima start command #1359

afbjorklund commented Feb 12, 2023 •

edited

Loading

AkihiroSuda Feb 12, 2023

AkihiroSuda Feb 12, 2023

afbjorklund Feb 12, 2023

AkihiroSuda Feb 12, 2023

afbjorklund Feb 12, 2023

AkihiroSuda left a comment

AkihiroSuda commented Feb 14, 2023

jandubois left a comment

jandubois Feb 14, 2023

jandubois Feb 14, 2023

jandubois Feb 14, 2023

afbjorklund Feb 14, 2023

AkihiroSuda Feb 14, 2023

AkihiroSuda Feb 14, 2023

afbjorklund Feb 14, 2023

afbjorklund commented Feb 14, 2023 •

edited

Loading

AkihiroSuda commented Feb 14, 2023

jandubois commented Feb 14, 2023

afbjorklund commented Feb 14, 2023

jandubois commented Feb 14, 2023

afbjorklund commented Feb 14, 2023

afbjorklund commented Feb 14, 2023

AkihiroSuda commented Feb 15, 2023

jandubois commented Feb 15, 2023

afbjorklund commented Feb 15, 2023

AkihiroSuda left a comment

	startCommand.Flags().String("yq", "", "modify the template inplace")
	startCommand.Flags().String("set", "", "modify the template inplace, using yq syntax")

Add builtin yq support to lima start command #1359

Add builtin yq support to lima start command #1359

Conversation

afbjorklund commented Feb 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AkihiroSuda left a comment

Choose a reason for hiding this comment

AkihiroSuda commented Feb 14, 2023

jandubois left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afbjorklund commented Feb 14, 2023 • edited Loading

AkihiroSuda commented Feb 14, 2023

jandubois commented Feb 14, 2023

afbjorklund commented Feb 14, 2023

jandubois commented Feb 14, 2023

afbjorklund commented Feb 14, 2023

afbjorklund commented Feb 14, 2023

AkihiroSuda commented Feb 15, 2023

jandubois commented Feb 15, 2023

afbjorklund commented Feb 15, 2023

AkihiroSuda left a comment

Choose a reason for hiding this comment

afbjorklund commented Feb 12, 2023 •

edited

Loading

afbjorklund commented Feb 14, 2023 •

edited

Loading