panic when interrupting deletion #3157

blampe · 2024-08-07T23:32:17Z

Spun off from #1232.

ctrl-C during deletion can lead to this panic:

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x107970948]
    goroutine 61 [running]:
    github.com/pulumi/pulumi-kubernetes/provider/v4/pkg/provider.checkpointObject(0x14001a8f550, 0x0, 0x14000ea1dd0, {0x14000e7ac68, 0x2}, {0x14000e826e0, 0x1a})
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2916 +0x38
    github.com/pulumi/pulumi-kubernetes/provider/v4/pkg/provider.(*kubeProvider).Delete(0x140003f0000, {0x109108170, 0x14000ea1da0}, 0x14000e96840)
    	/home/runner/work/pulumi-kubernetes/pulumi-kubernetes/provider/pkg/provider/provider.go:2591 +0x998
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Delete_Handler.func1({0x109108170?, 0x14000ea1da0?}, {0x108a64440?, 0x14000e96840?})
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:705 +0xd0
    github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x109108170, 0x14000ea0de0}, {0x108a64440, 0x14000e96840}, 0x140006e9c40, 0x14000e39e48)
    	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x2e4
    github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Delete_Handler({0x10901f760, 0x140003f0000}, {0x109108170, 0x14000ea0de0}, 0x14000e64f00, 0x140004881c0)
    	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:707 +0x148
    google.golang.org/grpc.(*Server).processUnaryRPC(0x1400086e000, {0x109108170, 0x14001a08180}, {0x109159c00, 0x1400076a000}, 0x14001a1a000, 0x14000b1e360, 0x10c89dac0, 0x0)
    	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1369 +0xb58
    google.golang.org/grpc.(*Server).handleStream(0x1400086e000, {0x109159c00, 0x1400076a000}, 0x14001a1a000)
    	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1780 +0xb20
    google.golang.org/grpc.(*Server).serveStreams.func2.1()
    	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1019 +0x8c
    created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 70
    	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1030 +0x13c

    error: update failed

The text was updated successfully, but these errors were encountered:

- The new `enableSecretMutable` provider configuration option treats changes to `Secrets` as updates instead of replacements (similar to the `enableConfigMapMutable` option). The default replacement behavior can be preserved for a particular `Secret` by setting its `immutable` field to `true`. (#2291) **Note:** These options (`enableSecretMutable` and `enableConfigMapMutable`) may become the default behavior in a future v5 release of the provider. Programs that depend on the replacement of `Secrets` and `ConfigMaps` (e.g. to trigger updates for downstream dependencies like `Deployments`) are recommended to explicitly specify `immutable: true`. - A warning is now emitted if an object has finalizers which might be blocking deletion. (#1418) - **EXPERIMENTAL**: Generic await logic is now available as an opt-in feature. Running a program with `PULUMI_K8S_AWAIT_ALL=true` will now cause Pulumi to await readiness for _all_ resources, including custom resources. Generic readiness is determined according to some well-known conventions (like the "Ready" condition) as determined by [cli-utils](https://github.com/kubernetes-sigs/cli-utils/tree/master/pkg/kstatus). Pulumi's current behavior, without this feature enabled, is to assume some resources are immediately available, which can cause downstream resources to fail. Existing readiness logic is unaffected by this setting. (#2996) - **EXPERIMENTAL**: The `pulumi.com/waitFor` annotation was introduced to allow for custom readiness checks. This override Pulumi's own await logic for the resource (however the `pulumi.com/skipAwait` annotation still takes precedence). The value of this annotation can take 3 forms: 1. A string prefixed with `jsonpath=` followed by a [JSONPath](https://kubernetes.io/docs/reference/kubectl/jsonpath/) expression and an optional value. The JSONPath expression accepts the same syntax as `kubectl get -o jsonpath={...}`. If a value is provided, the resource is considered ready when the JSONPath expression evaluates to the same value. For example this resource expects its "phase" field to have a value of "Running": `pulumi.com/waitFor: "jsonpath={.status.phase}=Running"` If a value is not provided, the resource will be considered ready when any value exists at the given path, similar to `kubectl wait --for jsonpath=...`. This resource will wait until it has a webhook configured with a CA bundle: `pulumi.com/waitFor: "jsonpath={.webhooks[*].clientConfig.caBundle}"` 2. A string prefixed with `condition=` followed by the type of the condition and an optional status. This matches the behavior of `kubectl wait --for=condition=...` and will wait until the resource has a matching condition. The expected status defaults to "True" if not specified. `pulumi.com/waitFor: "condition=Synced"` `pulumi.com/waitFor: "condition=Reconciling=False"` 3. A string containing a JSON array of multiple `jsonpath=` and `condition=` expressions. `pulumi.com/waitFor: '["jsonpath={.foo}", "condition=Bar"]'` - Pulumi will now emit logs for any Kubernetes "Warning" Events associated with resources being created, updated or deleted. (https://github.com/pulumi/pulumi-kubernetes/pull/3135/files) - The `immutable` field is now respected for `ConfigMaps` when the provider is configured with `enableConfigMapMutable`. (#3181) - Fixed a panic that could occur during deletion. (#3157)

>[!IMPORTANT] > The `enableSecretMutable` and `enableConfigMapMutable` options may become the default behavior in a future v5 release of the provider. Programs that depend on the replacement of `Secrets` and `ConfigMaps` (e.g. to trigger updates for downstream dependencies like `Deployments`) are recommended to explicitly specify `immutable: true`. ### Added - The new `enableSecretMutable` provider configuration option treats changes to `Secrets` as updates instead of replacements (similar to the `enableConfigMapMutable` option). The default replacement behavior can be preserved for a particular `Secret` by setting its `immutable` field to `true`. (#2291) - A warning is now emitted if an object has finalizers which might be blocking deletion. (#1418) - **EXPERIMENTAL**: Generic await logic is now available as an opt-in feature. Running a program with `PULUMI_K8S_AWAIT_ALL=true` will now cause Pulumi to await readiness for _all_ resources, including custom resources. Generic readiness is determined according to some well-known conventions (like the "Ready" condition) as determined by [cli-utils](https://github.com/kubernetes-sigs/cli-utils/tree/master/pkg/kstatus). Pulumi's current behavior without this feature enabled is to assume some resources are immediately available, which can cause downstream resources to fail. Existing readiness logic is unaffected by this setting. (#2996) - **EXPERIMENTAL**: The `pulumi.com/waitFor` annotation was introduced to allow for custom readiness checks. This override Pulumi's own await logic for the resource (however the `pulumi.com/skipAwait` annotation still takes precedence). The value of this annotation can take 3 forms: 1. A string prefixed with `jsonpath=` followed by a [JSONPath](https://kubernetes.io/docs/reference/kubectl/jsonpath/) expression and an optional value. The JSONPath expression accepts the same syntax as `kubectl get -o jsonpath={...}`. If a value is provided, the resource is considered ready when the JSONPath expression evaluates to the same value. For example this resource expects its "phase" field to have a value of "Running": `pulumi.com/waitFor: "jsonpath={.status.phase}=Running"` If a value is not provided, the resource will be considered ready when any value exists at the given path, similar to `kubectl wait --for jsonpath=...`. This resource will wait until it has a webhook configured with a CA bundle: `pulumi.com/waitFor: "jsonpath={.webhooks[*].clientConfig.caBundle}"` 2. A string prefixed with `condition=` followed by the type of the condition and an optional status. This matches the behavior of `kubectl wait --for=condition=...` and will wait until the resource has a matching condition. The expected status defaults to "True" if not specified. `pulumi.com/waitFor: "condition=Synced"` `pulumi.com/waitFor: "condition=Reconciling=False"` 3. A string containing a JSON array of multiple `jsonpath=` and `condition=` expressions. `pulumi.com/waitFor: '["jsonpath={.foo}", "condition=Bar"]'` - Pulumi will now emit logs for any Kubernetes "Warning" Events associated with resources being created, updated or deleted. (https://github.com/pulumi/pulumi-kubernetes/pull/3135/files) ### Fixed - The `immutable` field is now respected for `ConfigMaps` when the provider is configured with `enableConfigMapMutable`. (#3181) - Fixed a panic that could occur during deletion. (#3157)

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [@pulumi/kubernetes](https://pulumi.com) ([source](https://redirect.github.com/pulumi/pulumi-kubernetes)) | dependencies | minor | [`4.17.1` -> `4.18.1`](https://renovatebot.com/diffs/npm/@pulumi%2fkubernetes/4.17.1/4.18.1) | --- ### Release Notes <details> <summary>pulumi/pulumi-kubernetes (@pulumi/kubernetes)</summary> ### [`v4.18.1`](https://redirect.github.com/pulumi/pulumi-kubernetes/blob/HEAD/CHANGELOG.md#4181-September-13-2024) [Compare Source](https://redirect.github.com/pulumi/pulumi-kubernetes/compare/v4.18.0...v4.18.1) ##### Added - Schemagen is now a library that can be consumed by other packages. ([https://github.com/pulumi/pulumi-kubernetes/pull/3187](https://redirect.github.com/pulumi/pulumi-kubernetes/pull/3187)) ##### Changed - Updated beta Kubernetes client libraries to stable v1.31 release. ([https://github.com/pulumi/pulumi-kubernetes/pull/3196](https://redirect.github.com/pulumi/pulumi-kubernetes/pull/3196)) ### [`v4.18.0`](https://redirect.github.com/pulumi/pulumi-kubernetes/blob/HEAD/CHANGELOG.md#4180-September-3-2024) [Compare Source](https://redirect.github.com/pulumi/pulumi-kubernetes/compare/v4.17.1...v4.18.0) ##### Added - The new `enableSecretMutable` provider configuration option treats changes to `Secrets` as updates instead of replacements (similar to the `enableConfigMapMutable` option). The default replacement behavior can be preserved for a particular `Secret` by setting its `immutable` field to `true`. [https://github.com/pulumi/pulumi-kubernetes/issues/2291](https://redirect.github.com/pulumi/pulumi-kubernetes/issues/2291)2291) **Note:** These options (`enableSecretMutable` and `enableConfigMapMutable`) may become the default behavior in a future v5 release of the provider. Programs that depend on the replacement of `Secrets` and `ConfigMaps` (e.g. to trigger updates for downstream dependencies like `Deployments`) are recommended to explicitly specify `immutable: true`. - A warning is now emitted if an object has finalizers which might be blocking deletio[https://github.com/pulumi/pulumi-kubernetes/issues/1418](https://redirect.github.com/pulumi/pulumi-kubernetes/issues/1418)1418) - **EXPERIMENTAL**: Generic await logic is now available as an opt-in feature. Running a program with `PULUMI_K8S_AWAIT_ALL=true` will now cause Pulumi to await readiness for *all* resources, including custom resources. Generic readiness is determined according to some well-known conventions (like the "Ready" condition) as determined by [cli-utils](https://redirect.github.com/kubernetes-sigs/cli-utils/tree/master/pkg/kstatus). Pulumi's current behavior, without this feature enabled, is to assume some resources are immediately available, which can cause downstream resources to fail. Existing readiness logic is unaffected by this setting. [https://github.com/pulumi/pulumi-kubernetes/issues/2996](https://redirect.github.com/pulumi/pulumi-kubernetes/issues/2996)2996) - **EXPERIMENTAL**: The `pulumi.com/waitFor` annotation was introduced to allow for custom readiness checks. This override Pulumi's own await logic for the resource (however the `pulumi.com/skipAwait` annotation still takes precedence). The value of this annotation can take 3 forms: 1. A string prefixed with `jsonpath=` followed by a [JSONPath](https://kubernetes.io/docs/reference/kubectl/jsonpath/) expression and an optional value. The JSONPath expression accepts the same syntax as `kubectl get -o jsonpath={...}`. If a value is provided, the resource is considered ready when the JSONPath expression evaluates to the same value. For example this resource expects its "phase" field to have a value of "Running": `pulumi.com/waitFor: "jsonpath={.status.phase}=Running"` If a value is not provided, the resource will be considered ready when any value exists at the given path, similar to `kubectl wait --for jsonpath=...`. This resource will wait until it has a webhook configured with a CA bundle: `pulumi.com/waitFor: "jsonpath={.webhooks[*].clientConfig.caBundle}"` 2. A string prefixed with `condition=` followed by the type of the condition and an optional status. This matches the behavior of `kubectl wait --for=condition=...` and will wait until the resource has a matching condition. The expected status defaults to "True" if not specified. `pulumi.com/waitFor: "condition=Synced"` `pulumi.com/waitFor: "condition=Reconciling=False"` 3. A string containing a JSON array of multiple `jsonpath=` and `condition=` expressions. `pulumi.com/waitFor: '["jsonpath={.foo}", "condition=Bar"]'` - Pulumi will now emit logs for any Kubernetes "Warning" Events associated with resources being created, updated or delete[https://github.com/pulumi/pulumi-kubernetes/pull/3135](https://redirect.github.com/pulumi/pulumi-kubernetes/pull/3135)ull/3135/files) ##### Fixed - The `immutable` field is now respected for `ConfigMaps` when the provider is configured with `enableConfigMapMutable`. [https://github.com/pulumi/pulumi-kubernetes/issues/3181](https://redirect.github.com/pulumi/pulumi-kubernetes/issues/3181)3181) - Fixed a panic that could occur during deletion. ([https://github.com/pulumi/pulumi-kubernetes/issues/3157](https://redirect.github.com/pulumi/pulumi-kubernetes/issues/3157)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://redirect.github.com/renovatebot/renovate).  Co-authored-by: lumiere-bot[bot] <98047013+lumiere-bot[bot]@users.noreply.github.com>

blampe added the impact/panic This bug represents a panic or unexpected crash label Aug 7, 2024

blampe mentioned this issue Aug 7, 2024

Decide how skipAwait should interact with deletion #1232

Open

blampe added the area/await-logic label Aug 7, 2024

blampe self-assigned this Aug 7, 2024

blampe added this to the 0.108 milestone Aug 7, 2024

This was referenced Aug 7, 2024

Improve Await logic #2824

Closed

Allow skipAwait to skip only readiness or deletion #3147

Closed

mjeffryes added the kind/bug Some behavior is incorrect or out of spec label Aug 9, 2024

blampe mentioned this issue Aug 16, 2024

Unify await logic for deletes #3133

Merged

mjeffryes removed this from the 0.108 milestone Aug 19, 2024

blampe closed this as completed in #3133 Aug 21, 2024

blampe closed this as completed in adfff97 Aug 21, 2024

pulumi-bot added the resolution/fixed This issue was fixed label Aug 21, 2024

blampe mentioned this issue Sep 3, 2024

Prepare for v4.18.0 release #3191

Merged

blampe mentioned this issue Sep 16, 2024

Ctrl-c leads to panic #3209

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

panic when interrupting deletion #3157

panic when interrupting deletion #3157

blampe commented Aug 7, 2024

panic when interrupting deletion #3157

panic when interrupting deletion #3157

Comments

blampe commented Aug 7, 2024