ClientKafka messages emitted before client is ready fail silently #10449
Comments
nikolovjdmbh
changed the title
Even minimal reproduction codehttps://github.com/FelipeCavichiolliSilvestre/kafka-client-nest-bug NestJS version9.0.0 Package versionsNode.js version16.15.1 In which operating systems have you tested?
OtherThis bug does not happen when you pass the option "produceOnlyMode: true" to the Kafka client injected in app.module.ts This will work:@Module({
imports: [
ClientsModule.register([
{
name: 'KAFKA_CLIENT',
transport: Transport.KAFKA,
options: {
client: {
brokers: ['localhost:9094'],
},
producerOnlyMode: true,
},
},
]),
export class AppModule {} |
|
We faced a similar issue, we use a service that injects the Kafka client, but we explicitly called const producer = await this.client.connect();
await produer.send({ ... });After a restart of the application, it seems some messages weren't emitted correctly as awaiting the I guess in theory it could still happen, eg when the application starts up but immediately handles a request which has to produce a Kafka message (and the producer isn't connected properly yet). From what I can tell we are explicitly calling Are we supposed to explicitly handle EDIT: I guess I should have read the docu more intensely before posting 😅 https://docs.nestjs.com/microservices/basics#client. EDIT 2: Turns out the issue on our side was caused by emitting multiple messages in parallel which in turn calls Pseudocode: export class KafkaProducerService {
.
.
async emit(topic, message) {
const producer = await this.client.connect();
await producer.send({ topic, messages: [message] });
}
.
.
}
// Some service using KafkaProducerService
export class SomeService {
.
.
async doSomething {
await Promise.all([
this.kafkaProducer.emit('topic1', { ... }),
this.kafkaProducer.emit('topic2', { ... }),
]);
}
.
.
} |
|
After the journey in my last comment, I got curious whether the same or similar problem could occur when using the "nest way" of interacting with Kafka (means by using Forked repo: https://github.com/tak1n/kafka-client-nest-bug (note it uses When explicitly calling Node.js versions In which operating systems have you tested? |
|
We are experiencing the same issue and thanks @tak1n for the super clear bug report with two simple code examples. |
|
Fixed in this PR #11026 |
Is there an existing issue for this?
Current behavior
If
ClientKafka#emitis called inside a controller before the client is initialized and ready, only the first message is sent (eventually), any subsequent messages sent before the client is ready fail silently.Minimum reproduction code
https://github.com/donamo/nk
Steps to reproduce
Expected behavior
All messages sent before ClientKafka initialization should either throw an error or be sent after initialization, not just the first one.
Package
@nestjs/common@nestjs/core@nestjs/microservices@nestjs/platform-express@nestjs/platform-fastify@nestjs/platform-socket.io@nestjs/platform-ws@nestjs/testing@nestjs/websocketsOther package
No response
NestJS version
9.1.4
Packages versions
Node.js version
16.17.0
In which operating systems have you tested?
Other
Logs showing the problem: https://github.com/donamo/nk/blob/master/log
messages sent on lines 21, 26, 31 fail silently.
The text was updated successfully, but these errors were encountered: