KTable

[akshdeep30]

Hey Team

I am new to Kafka and Kafka Streams,
I want to create Ktable using my topic and want to fetch all the data present in the topic the messages can be a week old, and want to filter the messages on the basis of payload, Can you guys please help me how to achieve that. and also i want to know for long can KTable store data.

Any reply and help would be appreciated

Thanks

[akshdeep30]

Hey Team

Any Update on this??

Thanks

[krystianity]

Hi @akshdeep30 KTable (at least this implementation) stores data in memory of your application, depending on the number of partitions and instances of your application (consumer-groups spreading).

It really depends on the amount of messages you want to process and the size of the events that you want to store in KTable.

If you are not running window like one shot operations its probably a better idea to store the data that you need a database while filtering beforehand using a KStream or a simple Kafka Consumer.

[akshdeep30]

So you mean so say that Ktable does not stores data permanently? So we have to save data in database either using kafka-connect, consumer or streams, right?

[akshdeep30]

Hey Team
Any update would be helpful from you guys :)
Thanks