Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GVK resolution fails if metrics-server is unavailable #335

Open
nightkr opened this issue Nov 18, 2024 · 0 comments
Open

GVK resolution fails if metrics-server is unavailable #335

nightkr opened this issue Nov 18, 2024 · 0 comments
Labels

Comments

@nightkr
Copy link
Member

nightkr commented Nov 18, 2024

Affected version

stackablectl 24.7.1

Current and expected behavior

  1. Run kubectl -n kube-system delete pods -l k8s-app=metrics-server && stackablectl release install dev
  2. Observe that stackablectl crashes with the following error message:
  ERROR  failed with status 503 Service Unavailable
    at src/client/builder.rs:199

   WARN  Unsuccessful data error parse: service unavailable

    at src/client/mod.rs:467

An unrecoverable error occured: failed to execute release (sub)command

Caused by these errors (recent errors listed first):
 1: failed to create Kubernetes client
 2: failed to run GVK discovery
 3: ApiError: "service unavailable\n": Failed to parse error data (ErrorResponse { status: "503 Service Unavailable", message: "\"service unavailable\\n\"", reason: "Failed to parse error data", code: 503 })
 4: "service unavailable\n": Failed to parse error data
  1. Observe that kubectl and k9s are able to manage the cluster just fine

I suspect that this comes down to metrics-server using K8s API aggregation, which allows it to provide a fake "resource" that is stored by itself rather than in etcd. This also means that that resource can be unavailable even if the apiserver and etcd are both doing fine.

Possible solution

We could either:

  1. Limit GVK resolution to the apigroups we care about
  2. Defer apigroup-specific resolution errors until accessing the relevant apigroup

Additional context

No response

Environment

Using k3s v1.31.0+k3s1 via k3d

Would you like to work on fixing this bug?

None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant