-
Notifications
You must be signed in to change notification settings - Fork 174
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Behavior of View Sync #440
Comments
Hey @aditya-msd, very sorry for the long response time - somehow I missed the issues here. What you're describing is a series of processing steps, which require transactional semantics.
You're right. When a processor writes a value to is table using Right now, I would see three ways how to solve this: first, like you described, resending the input-message to a "retry"-topic. That could be achieved by sending it to the The second approach means avoiding the use of views. For this we should consider the operations done by processors 1 and 2 as part of a single transaction. One of them starts, performs operations and then sends all information necessary (i.e. the state of the transaction) to processor 2, which does its operations, maybe sends the transaction back etc. Third option that works sometimes: we decouple the conflicting operations in a timely manner. For example:
Hope that helps, let me know if you have more problems or how you solved it :) Thanks! |
Recently I have encountered the below scenario .
I have 2 processors .
Processor 1 does some processing on a list of items and then stored the metadata like length of list , no of items and other info in its group table .
Processor 2 reads some message from a topic and does some processing. Additionally it also has a view of the group table of Processor 1 .
If Processor has processed 10 items , then in the group table , we would meta of sth like ListLength = 10 and so on. Additionally its also expected that Processor 2 would receive 10 items in its topic .
What I have seen is the following :
Effectively , what happening is that the batch of 10 items is kind of stuck , since the first 2 items did not get processed cleanly due to the View not having the metadata .
How do I solve this problem , basically I want to keep track of the unprocessed 2 items , to effectively conclude the batch of 10 is fully processed . Is there a way to retry or some kind of push back to self topic or sth .
To me this looks like there is some time taken for the View to get updated .
The text was updated successfully, but these errors were encountered: