Mutagen (dev mode) improvements

[edvald]

Combining a few related commits, around the mutagen sync mechanism:

improvement(k8s): better process mgmt and logging for dev mode sync
We now poll for more information from the mutagen daemon, and we've overall improved the process management around it.

Besides the improved runtime information, this unblocks more development around this feature, such as running a postSyncCommand (like with the old hot reloading mechanism) and also makes the use of mutagen easier for other contexts like syncing build contexts for in-cluster building.

Also closes #2519 through a few added option flags.

feat(k8s): allow owner/perm tweaks on dev mode syncs
Also updated our docs a bit.

Closes #2519

feat(k8s): add experimental mutagen-based build sync mode
You can now set GARDEN_K8S_BUILD_SYNC_MODE=mutagen to enable a new,
as yet experimental build sync mode, which replaces rsync with mutagen
for synchronizing build contexts ahead of kaniko and
cluster-buildkit builds.

Please give it a spin, since we want to battle-test this a bit before
removing the older rsync method.

[thsig]

Good stuff!

One thing I noticed: If I run garden deploy --dev vote --env remote in the vote example project, I get the following error:

Failed deploying service 'vote' (from module 'vote'). Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Command "/Users/ths/.garden/tools/mutagen/62a411c726894601/mutagen sync create /Users/ths/src/garden/examples/vote/vote/src
exec:'/Users/ths/.garden/tools/kubectl/fd63e19492c91ef3/kubectl exec -i --context=gke_garden-dev-200012_europe-west1-b_garden-dev-1
--namespace=vote-ths --container vote Deployment/vote -- /.garden/mutagen-agent synchronizer':/app/src --name Deployment--vote-ths--vote-0
 --auto-start=false --mode two-way-safe" failed with code 1:

Error: unknown flag: --mode
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[edvald]

Ah, seems the mode flag is not set right. Will fix.

[thsig]

Another thing I noticed: The Joi validation of devMode.sync[].mode for container modules isn't validating that mode is one of the allowed values.

I was able to set mode: bongo without getting a validation error when re-running the above command.

[thsig]

Looks great! Only had one (minor) additional comment there.

[edvald]

@thsig did you test this manually? Would you mind kicking it around a bit with GARDEN_K8S_BUILD_SYNC_MODE=mutagen set as well? The integ/e2e tests seem alright, but this is a little tricky to test automatically so we should probably try to break it :P

[thsig]

Yeah, I've poked at this from many different directions—looks good so far 👍