Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kafka confluent schema registry error on startup on v415 #17299

Closed
metalshanked opened this issue Apr 29, 2023 · 9 comments · Fixed by #17858
Closed

Kafka confluent schema registry error on startup on v415 #17299

metalshanked opened this issue Apr 29, 2023 · 9 comments · Fixed by #17858
Assignees
@adamjshook
Labels
bug Something isn't working

Comments

@metalshanked
Copy link

Hi, Seem to be getting the below error on startup with release 415 (Docker)
No changes from <= 414 except the upgrade of docker image to 415
could it be this affecting change?

Kafka connector#
Add support for Protobuf oneof types when using the Confluent table description provider. (#16836)

2023-04-29T15:31:36.590Z	ERROR	main	io.trino.server.Server	io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema
java.lang.NoClassDefFoundError: io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema
	at ...
@hashhar
Copy link
Member

hashhar commented Apr 29, 2023

Can you share the complete stack-trace and the catalog config file for your Kafka connector?

cc: @adamjshook

@metalshanked
Copy link
Author

metalshanked commented Apr 30, 2023 •
edited
Loading

Sure, below is what i get on the 415 image. Switching back to 414 works fine.
Configs are

connector.name=kafka
kafka.default-schema=default
kafka.table-description-supplier=CONFLUENT
kafka.confluent-schema-registry-url=https://registry:8081
kafka.empty-field-strategy=IGNORE
kafka.nodes=XXXX
kafka.hide-internal-columns=false
kafka.messages-per-split=100000
kafka.timestamp-upper-bound-force-push-down-enabled=true
kafka.security-protocol=XXXX
kafka.ssl.keystore.location=XXXX
kafka.ssl.keystore.password=XXXX
kafka.ssl.keystore.type=XXXX
kafka.ssl.truststore.location=XXXX
kafka.ssl.truststore.password=XXXX
kafka.ssl.truststore.type=XXXX
kafka.ssl.key.password=XXXX
kafka.ssl.endpoint-identification-algorithm=XXXX

2023-04-30T05:33:54.706Z	INFO	main	io.airlift.bootstrap.LifeCycleManager	Life cycle starting...
2023-04-30T05:33:54.707Z	INFO	main	io.airlift.bootstrap.LifeCycleManager	Life cycle started
2023-04-30T05:33:54.710Z	ERROR	main	io.trino.server.Server	io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema
java.lang.NoClassDefFoundError: io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema
	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3402)
	at java.base/java.lang.Class.getDeclaredMethods(Class.java:2504)
	at com.google.inject.internal.DeclaredMembers.getDeclaredMethods(DeclaredMembers.java:48)
	at com.google.inject.spi.InjectionPoint.getDeclaredMethods(InjectionPoint.java:811)
	at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:730)
	at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:430)
	at com.google.inject.internal.ConstructorBindingImpl.getInternalDependencies(ConstructorBindingImpl.java:177)
	at com.google.inject.internal.InjectorImpl.getInternalDependencies(InjectorImpl.java:670)
	at com.google.inject.internal.InjectorImpl.cleanup(InjectorImpl.java:627)
	at com.google.inject.internal.InjectorImpl.initializeJitBinding(InjectorImpl.java:613)
	at com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:943)
	at com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:863)
	at com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:300)
	at com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:223)
	at com.google.inject.internal.InjectorImpl.getInternalFactory(InjectorImpl.java:949)
	at com.google.inject.internal.FactoryProxy.notify(FactoryProxy.java:48)
	at com.google.inject.internal.ProcessedBindingData.runCreationListeners(ProcessedBindingData.java:60)
	at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:137)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
	at com.google.inject.Guice.createInjector(Guice.java:87)
	at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:268)
	at io.trino.plugin.kafka.KafkaConnectorFactory.create(KafkaConnectorFactory.java:71)
	at io.trino.connector.DefaultCatalogFactory.createConnector(DefaultCatalogFactory.java:221)
	at io.trino.connector.DefaultCatalogFactory.createCatalog(DefaultCatalogFactory.java:130)
	at io.trino.connector.LazyCatalogFactory.createCatalog(LazyCatalogFactory.java:45)
	at io.trino.connector.StaticCatalogManager.lambda$loadInitialCatalogs$1(StaticCatalogManager.java:158)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
	at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
	at io.trino.util.Executors.executeUntilFailure(Executors.java:41)
	at io.trino.connector.StaticCatalogManager.loadInitialCatalogs(StaticCatalogManager.java:152)
	at io.trino.server.Server.doStart(Server.java:144)
	at io.trino.server.Server.lambda$start$0(Server.java:91)
	at io.trino.$gen.Trino_415____20230430_053321_1.run(Unknown Source)
	at io.trino.server.Server.start(Server.java:91)
	at io.trino.server.TrinoServer.main(TrinoServer.java:38)
Caused by: java.lang.ClassNotFoundException: io.confluent.kafka.schemaregistry.protobuf.ProtobufSchema
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
	at io.trino.server.PluginClassLoader.loadClass(PluginClassLoader.java:128)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	... 39 more

@adamjshook
Copy link
Member

Taking a look at what happened here.

@adamjshook adamjshook self-assigned this May 1, 2023
@adamjshook
Copy link
Member

io.confluent.kafka.schemaregistry.protobuf.ProtobufSchema is in the io.confluent:kafka-protobuf-provider package which is flagged as provided. I take it prior to #16836 this module was not needed for Confluent Schema Registry + Protobuf support? I guess not if it was working. The product tests have it in the compile scope, but it is just for testing.

Not sure how to proceed here, thoughts @hashhar?

@hashhar
Copy link
Member

hashhar commented May 2, 2023

Even in your PR I don't see any imports for io.confluent classes in non-test classes. The only pom.xml change is https://github.com/trinodb/trino/pull/16836/files#diff-5a48cbce31ddff463c604d3505f2ae42887db85a4ab5449b2f6a0df1f4ef797f which adds those to test scope.

I'll try to reproduce and see what it happening. Maybe some Confluent code is dependent on that package and calls it internally.

cc: @wendigo

@hashhar hashhar added the bug Something isn't working label May 10, 2023
@kokosing
Copy link
Member

But we have product tests for Kafka. The issue should be visible here.

@hashhar
Copy link
Member

hashhar commented May 10, 2023

product tests copy those jars into the container so we probably never noticed (or noticed and then added copying without documenting it as a requirement).

@hashhar
Copy link
Member

hashhar commented May 15, 2023

The JAR was required even in older releases but only when using actual Protobuf functionality.

I reproduced by applying following patch locally:

diff --git a/testing/trino-product-tests-launcher/src/main/resources/install-kafka-protobuf-provider.sh b/testing/trino-product-tests-launcher/src/main/resources/install-kafka-protobuf-provider.sh
index 3e81ba2c06..988cb48cc5 100644
--- a/testing/trino-product-tests-launcher/src/main/resources/install-kafka-protobuf-provider.sh
+++ b/testing/trino-product-tests-launcher/src/main/resources/install-kafka-protobuf-provider.sh
@@ -1,3 +1,3 @@
 #!/bin/bash
 set -xeuo pipefail
-cp --no-clobber --verbose /docker/kafka-protobuf-provider/* /docker/presto-server/plugin/kafka
+#cp --no-clobber --verbose /docker/kafka-protobuf-provider/* /docker/presto-server/plugin/kafka

Then follow these steps:

# First release with Protobuf support
git checkout 405
git apply patch.diff
./mvnw clean install -T 2C -nsu -DskipTests -Dskip.npm -Dskip.yarn
./testing/bin/ptl env up --environment multinode-kafka

I tested with 405 (first release with #14734), 411 (first release with #16763), 414 (first release before #16836) and 415 (first release with #16836) and it starts failing since 415 with the following stack trace:

presto-master       | 2023-05-15T18:17:53.683+0545	ERROR	main	io.trino.server.Server	io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema
presto-master       | java.lang.NoClassDefFoundError: io/confluent/kafka/schemaregistry/protobuf/ProtobufSchema
presto-master       | 	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
presto-master       | 	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3402)
presto-master       | 	at java.base/java.lang.Class.getDeclaredMethods(Class.java:2504)
presto-master       | 	at com.google.inject.internal.DeclaredMembers.getDeclaredMethods(DeclaredMembers.java:48)
presto-master       | 	at com.google.inject.spi.InjectionPoint.getDeclaredMethods(InjectionPoint.java:811)
presto-master       | 	at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:730)
presto-master       | 	at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:430)
presto-master       | 	at com.google.inject.internal.ConstructorBindingImpl.getInternalDependencies(ConstructorBindingImpl.java:177)
presto-master       | 	at com.google.inject.internal.InjectorImpl.getInternalDependencies(InjectorImpl.java:670)
presto-master       | 	at com.google.inject.internal.InjectorImpl.cleanup(InjectorImpl.java:627)
presto-master       | 	at com.google.inject.internal.InjectorImpl.initializeJitBinding(InjectorImpl.java:613)
presto-master       | 	at com.google.inject.internal.InjectorImpl.createJustInTimeBinding(InjectorImpl.java:943)
presto-master       | 	at com.google.inject.internal.InjectorImpl.createJustInTimeBindingRecursive(InjectorImpl.java:863)
presto-master       | 	at com.google.inject.internal.InjectorImpl.getJustInTimeBinding(InjectorImpl.java:300)
presto-master       | 	at com.google.inject.internal.InjectorImpl.getBindingOrThrow(InjectorImpl.java:223)
presto-master       | 	at com.google.inject.internal.InjectorImpl.getInternalFactory(InjectorImpl.java:949)
presto-master       | 	at com.google.inject.internal.FactoryProxy.notify(FactoryProxy.java:48)
presto-master       | 	at com.google.inject.internal.ProcessedBindingData.runCreationListeners(ProcessedBindingData.java:60)
presto-master       | 	at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:137)
presto-master       | 	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
presto-master       | 	at com.google.inject.Guice.createInjector(Guice.java:87)
presto-master       | 	at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:268)
presto-master       | 	at io.trino.plugin.kafka.KafkaConnectorFactory.create(KafkaConnectorFactory.java:71)
presto-master       | 	at io.trino.connector.DefaultCatalogFactory.createConnector(DefaultCatalogFactory.java:221)
presto-master       | 	at io.trino.connector.DefaultCatalogFactory.createCatalog(DefaultCatalogFactory.java:130)
presto-master       | 	at io.trino.connector.LazyCatalogFactory.createCatalog(LazyCatalogFactory.java:45)
presto-master       | 	at io.trino.connector.StaticCatalogManager.lambda$loadInitialCatalogs$1(StaticCatalogManager.java:158)
presto-master       | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
presto-master       | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
presto-master       | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
presto-master       | 	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:80)
presto-master       | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
presto-master       | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
presto-master       | 	at java.base/java.lang.Thread.run(Thread.java:833)
presto-master       | Caused by: java.lang.ClassNotFoundException: io.confluent.kafka.schemaregistry.protobuf.ProtobufSchema
presto-master       | 	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
presto-master       | 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
presto-master       | 	at io.trino.server.PluginClassLoader.loadClass(PluginClassLoader.java:128)
presto-master       | 	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
presto-master       | 	... 34 more
presto-master       |
presto-master       |
presto-master       | 2023-05-15T18:17:53.686+0545	INFO	Thread-78	io.airlift.bootstrap.LifeCycleManager	JVM is shutting down, cleaning up
presto-master       | 2023-05-15T18:17:53.687+0545	INFO	Thread-78	io.airlift.bootstrap.LifeCycleManager	Life cycle stopping...
presto-master       | 2023-05-15T18:17:53.688+0545	INFO	Thread-78	io.airlift.bootstrap.LifeCycleManager	Life cycle stopped
presto-master       | 2023-05-15T18:17:53.693+0545	INFO	Thread-77	io.airlift.bootstrap.LifeCycleManager	JVM is shutting down, cleaning up
presto-master       | 2023-05-15T18:17:53.694+0545	INFO	Thread-77	io.airlift.bootstrap.LifeCycleManager	Life cycle stopping...
presto-master       | 2023-05-15T18:17:53.699+0545	INFO	Thread-77	org.eclipse.jetty.server.Server	Stopped Server@723b8eff{STOPPING}[10.0.14,sto=0]
presto-master       | (exited)
presto-master       | (exited)

Now the hard part is finding what binding makes Guice try to load that class eagerly.

@kokosing
Copy link
Member

it starts failing since 415 with the following stack trace:

So this is a regression.

@hashhar hashhar mentioned this issue Jun 12, 2023
Merged
@tlblessing tlblessing mentioned this issue Jun 15, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adamjshook adamjshook

Labels
bug Something isn't working
Milestone
No milestone
4 participants
@adamjshook @kokosing @hashhar @metalshanked