[Merged by Bors] - feat: added topic-level deduplication mechanism #3385
Conversation
| } else { | ||
| error!("follower replica {} didn't exists!", old_replica.id); | ||
| } | ||
| outputs | ||
| } | ||
|
|
||
| pub async fn apply_replica_update( |
There was a problem hiding this comment.
Is these formatting changes only? If so, would prefer to move out of this PR since it's not related to this feature
There was a problem hiding this comment.
This is not formatting. This is propagating errors up from replica initialization.
| use std::{ | ||
| ops::{Deref}, | ||
| }; | ||
| use std::ops::Deref; |
There was a problem hiding this comment.
Same comment as previously. let's move formatting changes out of this PR
There was a problem hiding this comment.
If I modify a file where there are warnings from the rust-analyzer, I fix them. In that way, such warnings will be gradually fixed. Nobody ever will fix those warnings separately unless all tasks on the project are gone.
|
This is a rather large PR. Can this break it into smaller chunks? Suggest
|
Ok, this is big, indeed. I am going to split it into parts. |
|
First part #3392 |
This is the first part of [Dedup Impl PR](#3385) split for a more convenient review process.
|
This PR only contains SPU/SC changes |
|
bors r+ |
Upd.: Split into two parts: [first](#3392) Added topic-level deduplication mechanism. Key impacted components: - Topic Config - Fluvio CLI (topic creation, topic list, topic describe) - SPU (replica and producer handler) - SC - K8s CRDs (topic and partition) The feature requires for Dedup SmartModule filter to be provided. It is coming, but the one from the examples can be used to try it out: ```bash cd smartmodule/examples/filter_hashset smdk build smdk load ``` then create a file `topic.yaml`: ```yaml version: 0.1.0 meta: name: dedup-topic deduplication: bounds: count: 10 filter: transform: uses: infinyon/[email protected] ``` and ```bash fluvio topic create -c topic.yaml ``` with such configuration the window will be last 10 records. Closes infinyon/roadmap#114
|
Canceled. |
|
bors r+ |
Upd.: Split into two parts: [first](#3392) Added topic-level deduplication mechanism. Key impacted components: - Topic Config - Fluvio CLI (topic creation, topic list, topic describe) - SPU (replica and producer handler) - SC - K8s CRDs (topic and partition) The feature requires for Dedup SmartModule filter to be provided. It is coming, but the one from the examples can be used to try it out: ```bash cd smartmodule/examples/filter_hashset smdk build smdk load ``` then create a file `topic.yaml`: ```yaml version: 0.1.0 meta: name: dedup-topic deduplication: bounds: count: 10 filter: transform: uses: infinyon/[email protected] ``` and ```bash fluvio topic create -c topic.yaml ``` with such configuration the window will be last 10 records. Closes infinyon/roadmap#114
|
Pull request successfully merged into master. Build succeeded! The publicly hosted instance of bors-ng is deprecated and will go away soon. If you want to self-host your own instance, instructions are here. If you want to switch to GitHub's built-in merge queue, visit their help page. |
bors
bot
changed the title
Upd.: Split into two parts: first
Added topic-level deduplication mechanism.
Key impacted components:
The feature requires for Dedup SmartModule filter to be provided. It is coming, but the one from the examples can be used to try it out:
cd smartmodule/examples/filter_hashset smdk build smdk loadthen create a file
topic.yaml:and
with such configuration the window will be last 10 records.
Closes infinyon/roadmap#114