Support for file projection in run command

[EronWright]

Overview

This PR adds support for projecting configuration properties as temporary files, based on a new well-known block called files. For each property, a temporary file is created containing the property value, and an environment variable is set pointing to the temporary file. Note that the file contents may be dynamically generated using interpolation.

Example

This feature is designed for a key scenario of projecting a kubeconfig file based on a Pulumi environment. Here's how that would work:

Define an environment:

# import an environment that provides AWS credentials (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`).
imports:
  - dev-sandbox

# define a kubeconfig file.
values:
  files:
    KUBECONFIG: |
      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority-data: LS0tLS1...
          server: https://....gr7.us-west-2.eks.amazonaws.com
        name: arn:aws:eks:us-west-2:616138583583:cluster/mycluster
      contexts:
      - context:
          cluster: arn:aws:eks:us-west-2:616138583583:cluster/mycluster
          user: arn:aws:eks:us-west-2:616138583583:cluster/mycluster
        name: arn:aws:eks:us-west-2:616138583583:cluster/mycluster
      current-context: arn:aws:eks:us-west-2:616138583583:cluster/mycluster
      kind: Config
      preferences: {}
      users:
      - name: arn:aws:eks:us-west-2:616138583583:cluster/mycluster
        user:
          exec:
            apiVersion: client.authentication.k8s.io/v1beta1
            args:
            - --region
            - us-west-2
            - eks
            - get-token
            - --cluster-name
            - mycluster
            - --output
            - json
            command: aws

Run the environment and observe there's a KUBECONFIG environment variable that points to a temporary file containing the desired configuration:

$ esc run -i eron-kubernetes-test -- sh -c 'cat $KUBECONFIG'
apiVersion: v1
clusters:
- cluster:
...

This enables:

$ esc run -i eron-kubernetes-test -- kubectl get ns
NAME              STATUS   AGE
default           Active   46d
kube-node-lease   Active   46d
kube-public       Active   46d
kube-system       Active   46d

[lukehoban]

What are the semantics of files for consumption by tools other than esc run where there is not an "end time" for cleanup? Or is this block something that only matters for esc run?

I feel like perhaps we want to approach this in a way that doesn't require a new top level special key? Is there any option for that? Where we could somehow have the consumer (esc run in this case) be explicit about the keys it wants to map? Like esc run myenv --file kubeconfig:KUBECONFIG -- kubectl get ns?

[EronWright]

The lack of an "end time" is why I focused on run. I'm new to the project but I get the sense that various commands operate over well-known subsets. For example, esc open -f dotenv works exclusively with environmentVariables. Special keys seem foundational in Pulumi ESC and not to be feared.

[pgavlin]

Special keys seem foundational in Pulumi ESC and not to be feared.

I agree with this, FWIW.

I'll take a look at this PR today or tomorrow. I have a feeling that I'm going to need to give it some bake time.

One aside--I suspect is that it will be a best practice to define things like the contents of the kubeconfig in a "normal" value, then reference that from projections. Maybe I'm wrong about that but it smells promising to me, as the definition can then be shared around various projections as necessary. In this case that might mean defining the kubeconfig itself directly under values, then projecting it as a file by referencing it like

  files:
    KUBECONFIG: ${kubeconfig}

[pgavlin]

Overall idea looks great--just one comment on also writing these files in open when --format is shell or dotenv.

[pgavlin]

I think that we should also write these files in open for --format=shell and --format=dotenv.

[pgavlin]

(LMK if you want to have a go at that. if not we can take this as-is and add support for open in a subsequent PR)

[EronWright]

I would prefer that we tackle it in a follow-up, to keep this PR as focused as possible.

[lblackstone]

Do we want to stick with files as the section name, or should we be explicit about the fact that they're ephemeral? Something like ephemeralFiles, perhaps? We discussed that there may be other use cases for files that persist.

[pgavlin]

Do we want to stick with files as the section name, or should we be explicit about the fact that they're ephemeral? Something like ephemeralFiles, perhaps? We discussed that there may be other use cases for files that persist.

Personally I'm good with files. If we were to change the name I'd suggest tempFiles.