Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Change Feed Processor code migration sample #782

Merged
merged 4 commits into from
Sep 16, 2019

Conversation

ealsur
Copy link
Member

@ealsur ealsur commented Sep 9, 2019

Pull Request Template

Description

Adding a sample to cover code migration from the Change Feed Processor library (SDK V2) to V3 SDK.

@ealsur ealsur added ChangeFeed Samples Adding sample code for user's reference labels Sep 9, 2019
@ealsur ealsur self-assigned this Sep 9, 2019
@ealsur ealsur closed this Sep 9, 2019
@ealsur ealsur reopened this Sep 9, 2019
@ealsur
Copy link
Member Author

ealsur commented Sep 9, 2019

/azp run

@azure-pipelines
Copy link

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

@bartelink
Copy link
Contributor

bartelink commented Sep 9, 2019

Can the 'new CFP' emit a message when a range is assigned / unassigned? (Yes, I'm attempting to once again draw attention to #400 :P)

(As a workaround, does the v3 lib still use Liblog so I can at least echo such status changes to the console) ?

@ealsur
Copy link
Member Author

ealsur commented Sep 9, 2019

We are actually discussing that this week. I can add Traces you can hook to as a Diagnostics Source here: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos/src/ChangeFeedProcessor/FeedProcessing/ChangeFeedObserverBase.cs#L23

@bartelink
Copy link
Contributor

I believe that would be a very important feature (also the assignment side is equally interesting to be able to log timings of etc)

A useful thing in the context of this PR might also be to log when batches are obtained in the new API (or am I missing it?)


This does bring me to a second set of requirements in this space, which are covered in #616:

  • being able to let the processor read multiple batches ahead [and then do manual checkpointing under control of a processor]. (But being able to provide backpressure as was possible with the V2 API by sleeping until such time as sufficient progress has been made in order to prevent avoid excess memory consumption)
  • be able to distinguish from which rangeId a given batch was obtained (in order to be able to ensure that each assigned range's reading only gets a certain amount ahead)
  • (for avoidance of doubt, AIUI, the logic is still identical?) being able to handle messages from all assigned ranges concurrently (i.e. if one observer sleeps, the others should continue to loop)
  • (being able to signal that a range assignment has been revoked, which would be achieved by the CloseAsync hook)

(I know this is quite the tangent for a PR discussion; I believe the two cited issues sum up the concerns and needs, but am happy to expand on them and/or consolidate them if desired)

@bartelink
Copy link
Contributor

I added a comment to #400 (comment) summarizing the bulk of these tangential concerns, which might represent a better place to discuss such things.

As noted earlier, the basic logging hooks probably make sense to consider in the context of this particular PR (though manual checkpointing support #616 might inform the API design)

@ealsur ealsur changed the title Adding Change Feed Processor migration sample Adding Change Feed Processor code migration sample Sep 11, 2019
@kirankumarkolli kirankumarkolli merged commit 6459cec into master Sep 16, 2019
@kirankumarkolli kirankumarkolli deleted the users/ealsur/cfpmigration branch September 16, 2019 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ChangeFeed Samples Adding sample code for user's reference
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants