Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Facing "NameResolver returned no usable address" error #14

Open
deepu-james opened this issue Jan 3, 2025 · 8 comments
Open

Facing "NameResolver returned no usable address" error #14

deepu-james opened this issue Jan 3, 2025 · 8 comments

Comments

@deepu-james
Copy link

I am facing the below error when trying to hit a server from another one. The client and server are spring boot based Java applications.

io.grpc.StatusRuntimeException: UNAVAILABLE: NameResolver returned no usable address. addrs=[], attrs={io.grpc.xds.InternalXdsAttributes.callCounterProvider=io.grpc.xds.SharedCallCounterMap@1657448d, io.grpc.xds.InternalXdsAttributes.xdsClientPool=io.grpc.xds.SharedXdsClientPoolProvider$RefCountedXdsClientObjectPool@2c5733e7, io.grpc.internal.RetryingNameResolver.RESOLUTION_RESULT_LISTENER_KEY=io.grpc.internal.RetryingNameResolver$ResolutionResultListener@18c7636}
	at io.grpc.StatusRuntimeException.fillInStackTrace(StatusRuntimeException.java:68)
	at io.grpc.StatusRuntimeException.<init>(StatusRuntimeException.java:58)
	at io.grpc.StatusRuntimeException.<init>(StatusRuntimeException.java:50)
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
	at com.verteil.air.v3.offer.OfferManagementServiceGrpc$OfferManagementServiceBlockingStub.reshopOrder(OfferManagementServiceGrpc.java:298)
	at com.verteil.offermanagement.client.connector.OfferConnectorClientImpl.reshopOrder(OfferConnectorClientImpl.java:39)
	at com.verteil.offermanagement.service.impl.OfferServiceImpl.reshopOrder(OfferServiceImpl.java:165)
	at com.verteil.offermanagement.api.OfferManagementV3Api.reshopOrder(OfferManagementV3Api.java:48)
	at com.verteil.offermanagement.api.OfferManagementV3Api$$FastClassBySpringCGLIB$$69608825.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.aspectj.AspectJAfterAdvice.invoke(AspectJAfterAdvice.java:49)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
	at com.verteil.offermanagement.api.OfferManagementV3Api$$EnhancerBySpringCGLIB$$14246d62.reshopOrder(<generated>)
	at com.verteil.air.v3.offer.OfferManagementServiceGrpc$MethodHandlers.invoke(OfferManagementServiceGrpc.java:392)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
	at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
	at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:351)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:861)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:858)

The changes I made to the app are:

  • configured ManagedChannel creation like this:
Grpc
        .newChannelBuilder("xds://<app_name>.<namespace>:<app_port>", InsecureChannelCredentials.create())
        .maxInboundMessageSize(20 * 1024 * 1024)
        .build();
  • configured client's startup as -Dio.grpc.xds.bootstrap=/opt/xds_bootstrap_config.json. The contents of the config json file being:
{
  "xds_servers": [
    {
      "server_uri": "xds-server:5000",
      "channel_creds": [
        {
          "type": "insecure"
        }
      ],
      "server_features": [
        "xds_v3"
      ]
    }
  ],
  "node": {
    "id": "anything",
    "locality": {
      "zone": "k8s"
    }
  }
}

What could be the possible reasons for this failure? How can I debug such issues?

@whs
Copy link
Member

whs commented Jan 3, 2025

Are you running into #11 ?

Also check the debug port (:9000) to see if the exact name you're connecting is listed in the response. (It has to be exact match, as this is not DNS and there's no search domains support)

@deepu-james
Copy link
Author

Are you running into #11 ?

I do not think so. I am getting the error for the first try immediately after the client has started up.
To confirm this, I updated the gRPC ManagedChannel builder to set idleTimeout to 0. I don't think gRPC-java supports setting the idleTimeout as 0 since I get the error java.lang.IllegalArgumentException: idle timeout is 0, but must be positive when I create Channel as below.

Grpc
        .newChannelBuilder("xds:///connector-flx.dev-services:8094", InsecureChannelCredentials.create())
        .maxInboundMessageSize(20 * 1024 * 1024)
        .idleTimeout(0, TimeUnit.SECONDS)
        .build();

Also check the debug port (:9000) to see if the exact name you're connecting is listed in the response. (It has to be exact match, as this is not DNS and there's no search domains support)

2025-01-03 at 15 12 37
This is under services/type.googleapis.com/envoy.config.route.v3.RouteConfiguration/items
The highlighted server is what I intend to connect to. I hope the Channel has to be build with target address as xds:///connector-flx.dev-services:8094.

@whs
Copy link
Member

whs commented Jan 3, 2025

The name should be in the listener section. The connection URI seems to be valid. Have you remapped the port in Kubernetes Service? Either way if you could find the name in listener section it should be exactly what you need to type in the xds:// URI.

@deepu-james
Copy link
Author

deepu-james commented Jan 3, 2025

Have you remapped the port in Kubernetes Service?

I'm sorry, but I couldn't get this. What should be remapped?

Either way if you could find the name in listener section it should be exactly what you need to type in the xds:// URI.

Should it be xds:// or xds:///? Double slash or triple slash?

Is there a way to manually query the xds-server to see if the service (connector-flx.dev-services:8094 in this case) is resolving to valid address?

@whs
Copy link
Member

whs commented Jan 3, 2025

Kubernetes Service resource can map a port on the ClusterIP to another port on the actual application. For example, one can make it so that http://servicename:80 points to port 8080 on the pod so that the pod doesn't need root/setcap to bind to port 80. If that feature is used, then you need to point to the mapped port, not actual port.

It also means that similar to ClusterIP, you need to list the port in the Service resource. Without that ClusterIP will drop traffic, and in xDS it should result in name not found error.

(To be extra clear, we're not using ClusterIP to implement xDS but we try to emulate ClusterIP as much as possible to make users not remembering anything extra)

The URI scheme is xds:/// with triple slash. The xDS standard actually only use the scheme and path part of the URI and not the ordinary host and port part.

The :9000 port output is how I'd manually query it. Follow the Listener > Route > Cluster > Endpoint/ClusterLoadAssignment and you should see the list of IP addresses associated with the listener. It might also be useful to turn on gRPC debug logging on the client (usually available as envar depending on the implementation), as the client itself do this following chain and not the xDS server (the server only dump parts of the :9000 output the client is interested in, and stream any changes to the client).

@deepu-james
Copy link
Author

Kubernetes Service resource can map a port on the ClusterIP to another port on the actual application. For example, one can make it so that http://servicename:80 points to port 8080 on the pod so that the pod doesn't need root/setcap to bind to port 80. If that feature is used, then you need to point to the mapped port, not actual port.

Thank you. I can confirm that I am using the right port number.

apiVersion: v1
kind: Service
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      name: grpc
      port: 8094
      targetPort: 8094
  selector:
    app: connector-flx

I am attaching the output from :9000. It looks like the Cluster configuration for the Route has no edsConfig. Could this be the reason?
xds_server_debug.json

@whs
Copy link
Member

whs commented Jan 3, 2025

edsConfig is hardcoded to be empty, so it shouldn't be the case.

I wonder if this is the case: usually we set ports on pod manifest. On service we set targetPort to the port name instead of a number. This allow running applications with different port binding under the same service name (eg. strangler pattern). Our IaC convert deployment to service automatically, so all of our services are written in this way and we never tested other setups.

Looking at our xDS source it seems that if you set a name there might have to be named all the way, or if there is no name then it has never have a name. Could you test and let me know if that is the case

@deepu-james
Copy link
Author

I hope this is what you meant.
POD definition:
2025-01-03 at 17 38 19

Service definition:
2025-01-03 at 17 40 17

I am getting the same error with this configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants