Move the reconciliation loop to a model where each run executes as a k8s Job

[metral]

Problem description

Currently, the reconciliation loop is shared in-process on the operator.

Ideally, a separate Job/Pod would be used instead per loop to avoid shared context issues across loops and runtimes.
See the design doc.

[EronWright]

Using a Pod to execute pulumi up may unlock some interesting capabilities. Some example:

1. mounting of stack-specific volumes to act as an alternate source for the program sources, program data, or even backend state.
2. Stack-specific service accounts. This would allow us to harden the permissions granted to the operator itself.
3. Clean separation of log output between the operator and the program.

[viveklak]

Note that if we support #122 we will have to ensure the secret injection mechanism can mount secrets into the job/pod encapsulating the stack.

[cleverguy25]

Added to epic #586

[EronWright]

Good news everyone, we just release a preview of Pulumi Kubernetes Operator v2. This new release has a whole-new architecture that uses pods as the execution environment.

Please read the announcement blog post for more information:
https://www.pulumi.com/blog/pulumi-kubernetes-operator-2-0/

Would love to hear your feedback! Feel free to engage with us on the #kubernetes channel of the Pulumi Slack workspace.
cc @metral @liamawhite @sullivtr @bgagnon @mike-chen-samsung @Alegrowin