Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KPL sent records out of order #23

Closed
loneknightpy opened this issue Sep 2, 2015 · 2 comments
Closed

KPL sent records out of order #23

loneknightpy opened this issue Sep 2, 2015 · 2 comments

Comments

@loneknightpy
Copy link

KPL sent records of out order even when maxConnections was set to 1 and no error/retry occurred. I cannot find detail of KPL order guarantee in the documents. Is there a way to ensure ordering of records sent from KPL?

@gharikum
Copy link

gharikum commented Sep 8, 2015

Kinesis in general does not guarantee "write ordering". It does guarantee "read order", in that records successfully written to Kinesis are guaranteed to be always read out in the same order.

Even if you are not using the KPL and used the SDK directly, records are not guaranteed to be lodged in Kinesis in the same order they are written to the SDK, for a variety of reasons related to the way distributed systems work. Because the KPL uses the putRecords web API, it has the same side effect.

The only way to guarantee "write order" is if you have only one producer, and you use the "putRecord" (not putRecords) API of the SDK one record at a time, and use the "sequenceNumberForOrdering" feature. Basically you will put record n only after getting a successful response to the put for record n-1.

Hope this helped.

@loneknightpy
Copy link
Author

@gharikum Got it, thanks for the clarification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants