Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add note about fan-out for increased consumer throughput #222

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions content/articles/kinesis-in-production.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,8 @@ Now onto the part that may be the most important for the prospective Kinesis use

### You get five (reads) (#five-reads)

__Update (2019/01/10) —__ Amazon has introduced [enhanced fan-out capability](https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/) which gives each consumer a dedicated 2MB/second of read throughput per shard. Enhanced fan-out uses a streaming protocol, making it unnecessary for consumers to poll continually and use up the read limit.

Scalability is right there on the Kinesis front page as one of the core features of the product, and indeed it is scalable: by default a stream in US East can have up to 50 shards (this limit can be increased by opening a support ticket with AWS), each of which can handle 1 MB/s in and 2 MB/s out for a theoretically maximum of 50 MB/s in and 100 MB/s out. That's an incredible amount of data! However, despite being very scalable along this one dimension, it scales very poorly along another: the number of consumers that a stream can have.

Each shard on a stream supports a maximum limit of 5 reads per second. That number is [right on the limits page](http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-and-limits.html), but at no point does the documentation really spell out its repercussions (or at least nowhere that I could find). If each shard only supports 5 read operations, and each application consuming the stream must consume every shard, then you can only have a _maximum of 5 applications if you want to read the stream once per second_. If you want to have ten applications consuming the stream, then you will have to limit each so that on average it only consumes the stream once every two seconds. Each read can pull down 10 MB of data, so keeping up with the stream isn't a problem, but you can see how you'll have to sacrifice latency in order to scale up the number of consumers reading the stream.
Expand Down