exec updater errors when using non-root user

[nicks]

Repro steps:

1. Create a NodeJS project with live_update sync
2. Follow best practices of using a non-root user, as described in https://github.com/nodejs/docker-node/blob/master/docs/BestPractices.md#non-root-user
3. tilt up with microk8s and try to modify a file

Expected result:
Tilt should sync the file successfully

Actual result:
Tilt fails with cryptic errors

Branch with repro code:
https://github.com/windmilleng/nodejs-express-k8s/tree/nicks/user

Snapshot:
https://cloud.tilt.dev/snapshot/AdyPt-YLRPKSa_XuHBg=
(see the alert)

Thanks to Marlow in the #tilt k8s channel for reporting

[nicks]

The error message is:

Live Update failed with unexpected error:
	command terminated with exit code 2
Falling back to a full image build + deploy

[nicks]

The fundamental problem is that CRI (the container runtime interface) does not support execution as a particular user, here are some threads @landism found on it:

kubernetes/kubernetes#30656
kubernetes/kubernetes#30656 (comment)

I wonder if Tilt dev images should just always set the image user to root if you're using live_update

[zkutasi]

I have just stumbled up on this issue I think. Scenario: Kubernetes cluster, each pod according to our design rules must have a securityContext, specifying a non-root user.

What I have done here is that in my Tiltfile, I have a very customizable parser and info collector logic for the YAML files, I use it to distribute non-workload objects into different groups mostly (for readablity). I added a segment where I remove the securityContext parts from the YAML file if the object is mine and I build it. This is why I love the Tiltfile :)

Just one remark, I would not add this under the hood automatically, only if it is switchable on-off, because modifying the system under test like this might cause trouble elsewhere in the user's environment/codebase. So far this WA was +5 lines of Tiltfile code. Without it, everything works OK just Tilt will always need to rely on docker building, no live update possible. With this fix, I was able to live-update properly.

[maiamcc]

@zkutasi that's awesome! I don't have a good sense, do you think this fix would be applicable for other people's use cases as well? If yes, you might consider submitting an extension so other people can make use of the code!

[markgarrigan]

@zkutasi Do you mind sharing the 5 lines? I think we might have the same issue. Thanks!

[zkutasi]

@markgarrigan , sure:

decoded = decode_yaml_stream(h)
    if decoded:
        for d in decoded:
...
            # Live update conflicts with SecurityContext, until a better solution, just remove it
            if d["kind"] == "Deployment":
                if "securityContext" in d['spec']['template']['spec']:
                    d['spec']['template']['spec'].pop('securityContext')
                for c in d['spec']['template']['spec']['containers']:
                    if "securityContext" in c:
                        c.pop('securityContext')
...
        k8s_yaml(encode_yaml_stream(decoded))

I do not feel this is extension-worthy, and whether I am able to do it, I have absolutely no time nowadays.