Allow skipAwait to skip only readiness or deletion

[blampe]

Currently, the pulumi.com/skipAwait: true annotation tells the provider to
skip await logic whenever a resource is touched or deleted.

However, there are situations where you might want to skip await logic for only
deletes, or only for readiness, but not both. For example a resource might take
a long time to tear down, during which time other resources could also be
getting cleaned up.

This PR expands the acceptable values for the skipAwait annotation:

• "pulumi.com/skipAwait": "true" continues to skip await logic in all situations.
• "pulumi.com/skipAwait": "ready" will skip await logic when creating or
  updating a resource, when we would normally wait for it to become ready. We
  will still wait for deletion.
• "pulumi.com/skipAwait": "delete" will skip await logic when deleting the
  resource, but we will continue to wait for it become ready when creating or
  updating it.

Fixes #2551.
Refs #2824.
Refs #1418.

[blampe]

This change is part of the following stack:

• Unify await logic for deletes #3133
  • Add generic await logic #3143
    • Allow skipAwait to skip only readiness or deletion #3147 ◀

_{Change managed by git-spice.}

[github-actions]

Does the PR have any schema changes?

Looking good! No breaking changes found.
No new resources/functions.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 38.18%. Comparing base (9a5a997) to head (b912dd2).

Additional details and impacted files

@@                   Coverage Diff                    @@
##           blampe/2996-await-pr    #3147      +/-   ##
========================================================
+ Coverage                 38.14%   38.18%   +0.03%     
========================================================
  Files                        77       77              
  Lines                      9361     9367       +6     
========================================================
+ Hits                       3571     3577       +6     
  Misses                     5444     5444              
  Partials                    346      346              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[EronWright]

I'm pretty reluctant to skip await on deletion unless the user has explicitly opted in with skipAwait: delete. Because skipping await on deletion can lead to a race condition in the 'replacement' use case.

I think the safe move is to interpret the value of true as ready, not as ready,delete. Since that is a breaking change, another possibility would be to interpret true differently depending on the kind, to retain existing behavior. I'm still surprised that skipAwait ever applied to deletes.

In the future, maybe we should consider orphaning CRDs rather than deleting them by default, as is the behavior of Helm. This would solve one of the motivations for skipAwait applying to deletes.

[EronWright]

I would advocate for parsing the value as a comma-separated string, and then checking for the enum values of true, ready, delete. This would allow for a user to do ready,delete and for us to contemplate treating true as ready.

[blampe]

On pause #2551 (comment)