Add dynamic throughput configuration of writes on DynamoDB tables

[bwplotka]

Right now user needs to deduct how much throughput user needs for Index storage using these static flags in table-manager:

f.Int64Var(&cfg.ProvisionedWriteThroughput, "dynamodb.periodic-table.write-throughput", 3000, "DynamoDB periodic tables write throughput")
f.Int64Var(&cfg.ProvisionedReadThroughput, "dynamodb.periodic-table.read-throughput", 300, "DynamoDB periodic tables read throughput")
f.Int64Var(&cfg.InactiveWriteThroughput, "dynamodb.periodic-table.inactive-write-throughput", 1, "DynamoDB periodic tables write throughput for inactive tables.")
f.Int64Var(&cfg.InactiveReadThroughput, "dynamodb.periodic-table.inactive-read-throughput", 300, "DynamoDB periodic tables read throughput for inactive tables")

Would be awesome to keep these flags to configure maximum read/write throughput and just update the active table dynamically with correct throughput based on some logic e.g:

• for the first iteration, it could be just naive dynamic table updates based on number of ingested samples for writes.
• for read I don't have an easy solution in mind, despite machine learning. But writes are the most important for now, we don't care so much about query latencies, so we can keep read throughput low, right?

That would save some costs spent on Dynamo, especially for non deterministic traffic on our clusters.

[tomwilkie]

Yes! I've been wanting to do this for a few months now - thanks for filling a ticket.

We can observe consumed write capacity and queue length as inputs. For reads I agree, probably don't need it just yet.

Table Manager is the obvious place to put this all.

[bwplotka]

Cool, so actually every needed info we can have directly from CloudWatch, no need to dive into ingesters (:

[tomwilkie]

Yes, we could you cloud watch - but I'd probably use Prometheus instead...

…

On Wed, Mar 8, 2017 at 2:09 PM, Bplotka ***@***.***> wrote: Cool, so actually every needed info we can have directly from CloudWatch, no need to dive into ingesters (: — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#318 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhdfBOTJ789IlRKoan8ffnQkIAS9Iks5rjratgaJpZM4MWybk> .

[bboreham]

Addressed in #507