Knative service status to "Unknown" and "Uninitialized" #15753
Comments
|
Hi @hyde404 ,
The externalname should point to istio it is used for different purposes e.g. traffic splitting. I haven't checked all the details yet but is that external name being exposed on the AWS lb directly somehow (due to your ingresses) or Istio is not picking up changes? Could you try a more standard approach as in Knative docs as a smoke test?
This the reason you see the loadbalancer not being ready. I am wondering why https is used, what is the istio mode you use mtls? Note: Unfortunately I dont have an AWS cluster to test, so I am guessing. |
|
Hi @skonto, Thanks for your reply ! By removing proxy.istio.io/config: |
{
"gatewayTopology": {
"proxyProtocol": {}
}
}from It seems like, a probe, maybe from net-istio has issues. |
|
having the exact same issue, when testing |
|
I finally managed to make it work to use these annotations (you can also add and this envoyfilter as well So I totally got rid of "proxyProtocol": {} |
|
Observed the same behavior with knative service in a Kubeflow 1.9.1 deployment (on-prem). Restart istio-system/istio-ingressgateway deployment will make the knative service accessible and also the knative service ExternalName changes to "knative-local-gateway.istio-system.svc.cluster.local". Also observed in istio-ingressgateway log ' "GET /healthz HTTP/1.1" 404 NR route_not_found - "-" 0 0 0 - "192.168.0.238" "Knative-Ingress-Probe"' until restart the pod. Easily reproducible on my cluster by running the Kserve sklearn-iris inferenceservice example. |
What version of Knative?
1.17.0
net-istio: 1.17.0
istio: 1.24.2
Expected Behavior
The knative service status should be “Ready” instead of hanging in “Unknown” state.
Actual Behavior
When I deploy a knative service, in a EKS cluster, it remains in “Unknown” status until the istio ingress-controllers are restarted, even if the application can be reached.
It then switches to “Ready”, and the next application deployed will be in “Unknown” status, and so on.
The application is exposed with a loadbalancer and reachable
Here are the details of the knative service status:
So, for the load-balancing I use an AWS NLB, and everything seems to be ok, all the targets (15021, 443, 80) are healthy.
I also noticed a couple of logs, probably related to the issue.
{ "severity": "ERROR", "timestamp": "2025-02-06T13:25:22.20239663Z", "logger": "net-istio-controller.istio-ingress-controller", "caller": "status/status.go:421", "message": "Probing of https://test-eb7d5189.serverless-dev.xyz.crashcourse.com:443 failed, IP: 100.64.174.122:443, ready: false, error: error roundtripping https://test-eb7d5189.serverless-dev.xyz.crashcourse.com:443/healthz: read tcp 100.64.162.193:36768->100.64.174.122:443: read: connection reset by peer (depth: 0)", "commit": "4dff29e-dirty", "knative.dev/controller": "knative.dev.net-istio.pkg.reconciler.ingress.Reconciler", "knative.dev/kind": "networking.internal.knative.dev.Ingress", "knative.dev/traceid": "de3b3adb-b689-4a4c-b4d4-39c22a0911ba", "knative.dev/key": "eb7d5189/gsvc-serving-db07943b", "stacktrace": "knative.dev/networking/pkg/status.(*Prober).processWorkItem\n\tknative.dev/[email protected]/pkg/status/status.go:421\nknative.dev/networking/pkg/status.(*Prober).Start.func1\n\tknative.dev/[email protected]/pkg/status/status.go:306" }and
{ "severity": "WARNING", "timestamp": "2025-02-06T13:25:22.997317236Z", "logger": "controller", "caller": "route/reconcile_resources.go:227", "message": "Failed to update k8s service", "commit": "6265a8e", "knative.dev/pod": "controller-85c449cd99-97hgw", "knative.dev/controller": "knative.dev.serving.pkg.reconciler.route.Reconciler", "knative.dev/kind": "serving.knative.dev.Route", "knative.dev/traceid": "e6e9c300-658b-4feb-8b59-e2bf6fa95bd1", "knative.dev/key": "eb7d5189/gsvc-serving-db07943b", "error": "failed to fetch loadbalancer domain/IP from ingress status" }I'd also like to point out that I looked at the route and the ingress, the outputs of which are as follows
route
{ "address": { "url": "http://gsvc-serving-db07943b.eb7d5189.svc.cluster.local" }, "conditions": [ { "lastTransitionTime": "2025-02-06T13:43:16Z", "status": "True", "type": "AllTrafficAssigned" }, { "lastTransitionTime": "2025-02-06T15:03:04Z", "message": "Certificate route-55cecb6b-26fc-4fda-8e42-1d7d9c8fdd2b is not ready downgrade HTTP.", "reason": "HTTPDowngrade", "status": "True", "type": "CertificateProvisioned" }, { "lastTransitionTime": "2025-02-06T13:43:16Z", "message": "Waiting for load balancer to be ready", "reason": "Uninitialized", "status": "Unknown", "type": "IngressReady" }, { "lastTransitionTime": "2025-02-06T13:43:16Z", "message": "Waiting for load balancer to be ready", "reason": "Uninitialized", "status": "Unknown", "type": "Ready" } ], "observedGeneration": 1, "traffic": [ { "latestRevision": true, "percent": 100, "revisionName": "gsvc-serving-db07943b-00001" } ], "url": "http://test-eb7d5189.serverless-dev.xyz.crashcourse.com" }ingress
{ "conditions": [ { "lastTransitionTime": "2025-02-06T13:43:16Z", "message": "Waiting for load balancer to be ready", "reason": "Uninitialized", "status": "Unknown", "type": "LoadBalancerReady" }, { "lastTransitionTime": "2025-02-06T13:43:16Z", "status": "True", "type": "NetworkConfigured" }, { "lastTransitionTime": "2025-02-06T13:43:16Z", "message": "Waiting for load balancer to be ready", "reason": "Uninitialized", "status": "Unknown", "type": "Ready" } ], "observedGeneration": 1 }strange findings
Logs from
istiodServices before ingress-controller restart
Services after ingress-controller restart
The external-ip from
ExternalNameturned fromtest-eb7d5189.serverless-dev.xyz.crashcourse.comtoknative-local-gateway.istio-system.svc.cluster.local.Some tests
From a "Ready" service
From the "Unknown" service
Steps to Reproduce the Problem
Ingress controllers
My setup has some particularities. I use 3 different ingress controllers configured with the helm values as below:
In case you wonder, I use proxy config for matching source IPs and use it in AuthorizationPolicy afterwards.
Knative
I deploy knative using the knative-operator as follow :
domain-templateis linked to an operator we have, so nvm that.Knative Service
Same for the annotations/labels, is it linked to the operator
The text was updated successfully, but these errors were encountered: