//grpclb:grpclb Bazel target missing META-INF resources

[SanjayVas]

What version of gRPC-Java are you using?

1.46.0

What is your environment?

Linux (Debian Bookworm / gLinux), JDK 11, Bazel 4.2.1

What did you expect to see?

Generated JAR from //grpclb:grpclb target includes resources under META-INF/services like how the Maven JAR does.

What did you see instead?

JAR does not include these files.

Steps to reproduce the bug

1. Depend on grpc-java as a Bazel repo using /repositories.bzl
2. Depend on google-cloud-spanner Maven artifact.
3. Attempt to connect to Google Cloud Spanner.

com.google.cloud.spanner.SpannerException: UNKNOWN: Default config is invalid: Status{code=UNKNOWN, description=None of [grpclb] specified by Service Config are available., cause=null}
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:284)
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:61)
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:181)
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:110)
        at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:517)
        at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:302)
        at com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create(SpannerOptions.java:469)
        at com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create(SpannerOptions.java:464)
        at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:560)
        at com.google.cloud.spanner.SpannerOptions.getSpannerRpcV1(SpannerOptions.java:1320)
        at com.google.cloud.spanner.SpannerImpl.<init>(SpannerImpl.java:132)
        at com.google.cloud.spanner.SpannerOptions$DefaultSpannerFactory.create(SpannerOptions.java:459)
        at com.google.cloud.spanner.SpannerOptions$DefaultSpannerFactory.create(SpannerOptions.java:454)
        at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:540)

...

Caused by: java.lang.IllegalStateException: Default config is invalid: Status{code=UNKNOWN, description=None of [grpclb] specified by Service Config are available., cause=null}
        at com.google.common.base.Preconditions.checkState(Preconditions.java:590)
        at io.grpc.internal.ManagedChannelImpl.<init>(ManagedChannelImpl.java:676)
        at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630)
        at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
        at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:388)
        at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:105)
        at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:83)
        at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:236)
        at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:230)
        at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:201)
        at com.google.cloud.spanner.v1.stub.GrpcSpannerStub.create(GrpcSpannerStub.java:231)
        at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:411)
        ... 32 more

Notes

Due to missing the above files, the default LoadBalancerRegistry is unable to find a load balancer for grpclb.

[sergiitk]

@SanjayVas Thank you for the report. To clarify, is this something that started to occur recently, or something we were missing all along?

[SanjayVas]

I think that it's something that was only surfaced by a change when we upgraded https://github.com/googleapis/java-spanner, as we had been using a very old version (3.0.3). Prior to that, I don't think we were using any client libraries that tried to use grpclb.

[sergiitk]

@SanjayVas Fixed with #9156.

[sergiitk]

@SanjayVas To clarify one thing: would you like me to backport this change to previous versions?

[SanjayVas]

For my project in particular, we have a workaround and are fine just dropping that once we upgrade to a release that has this fix.

I don't know what the backporting policy should be here. AFAICT, the bug is only surfaced when the following conditions are all true:

1. Building grpc-java grpclb using Bazel (as opposed to pulling it from Maven).
2. Using a client library that specifies the grpclb policy.

We only hit (2) after upgrading java-spanner, which I assume in turn picked up something related to grpclb policy from gax-java. I'm not sure which versions of grpc-java you'd want to backport this to as a result.

As an aside, the particularly insidious thing is that we only hit the error when attempting to connect to a real Cloud Spanner instance. All of our tests using Spanner Emulator don't hit this path.

[sergiitk]

Building grpc-java grpclb using Bazel (as opposed to pulling it from Maven).

Yes, I verified that maven jars do have this files, and bazel builds didn't. Then I fixed bazel builds in #9156.

I don't know what the backporting policy should be here.

I personally don't know how bazel-oriented users set up grpc-java dependencies. Is it pinned to a version tag, f.e. v1.46.0? Or a version branch, v1.46.x? Whatever the case, I'll be happy to backport that change if you have a practical need for it.

cc @ejona86

[ejona86]

@sergiitk, let's backport the change, but we don't need to bother yet with making a patch release for it.