Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Consider techniques to reduce s3 writes #141

Closed
tomwilkie opened this issue Nov 16, 2016 · 3 comments
Closed

Consider techniques to reduce s3 writes #141

tomwilkie opened this issue Nov 16, 2016 · 3 comments
Assignees
@tomwilkie

Comments

@tomwilkie
Copy link
Contributor

We could batch together chunks (from different timeseries) into "megachunks", and write the key of the megachunk + offset into dynamo. This would reduce S3 writes but increase cost of reads.
@tomwilkie tomwilkie mentioned this issue Feb 7, 2017
Closed
@tomwilkie
Copy link
Contributor Author

S3 writes are 70% of the costs of running cortex. Batching chunks into groups of 10 would make cortex >50% cheaper.

We should also consider writing the chunks to dynamodb. Might be cheaper/faster, but need to do the sums.

@tomwilkie
Copy link
Contributor Author

We should also consider writing the chunks to dynamodb. Might be cheaper/faster, but need to do the sums.

Back-of-the-envelope calculations show this would reduce costs by 50%. So not as much as "superchunks" in S3, but it is a large cost reduction.

@tomwilkie tomwilkie self-assigned this May 10, 2017
tomwilkie added a commit that referenced this issue May 10, 2017
@tomwilkie
Various chunk store / DynamoDB config refactorings.
570fd37 
(Motivation: tidying a few things up before handling #141)

- Move /dep.go into /pkg/ingester/client, where is belongs.
- Rename DynamoTableManager to TableManager, as there is nothing DynamoDB-specific about it - it would be useful to BigTable too.
- Rename DynamoTableClient interface to TableClient, as there is nothing DynamoDB-specici about it - it would be useful to BigTable too.
- Move dynamoTableClient (implementation of TableClient) into /pkg/chunk/table_manager.go, where it belongs.
- Delete chunk.DynamoTableClientConfig, as the table-manager-specific in-memory construction wasn't used.
- Move warning about unused DynamoDB table name in path to dynamoClientFromURL.
- Move duplicated OriginalTableName into PeriodicTableConfig, as per comment.
- Fix usage of flag.StringVar - should be f.StringVar.
@tomwilkie tomwilkie mentioned this issue May 10, 2017
Merged
tomwilkie added a commit that referenced this issue May 10, 2017
@tomwilkie
Various chunk store / DynamoDB config refactorings. (#417)
87c4678 
* Various chunk store / DynamoDB config refactorings.

(Motivation: tidying a few things up before handling #141)

- Move /dep.go into /pkg/ingester/client, where is belongs.
- Rename DynamoTableManager to TableManager, as there is nothing DynamoDB-specific about it - it would be useful to BigTable too.
- Rename DynamoTableClient interface to TableClient, as there is nothing DynamoDB-specici about it - it would be useful to BigTable too.
- Move dynamoTableClient (implementation of TableClient) into /pkg/chunk/table_manager.go, where it belongs.
- Delete chunk.DynamoTableClientConfig, as the table-manager-specific in-memory construction wasn't used.
- Move warning about unused DynamoDB table name in path to dynamoClientFromURL.
- Move duplicated OriginalTableName into PeriodicTableConfig, as per comment.
- Fix usage of flag.StringVar - should be f.StringVar.

* Review feedback
@tomwilkie tomwilkie mentioned this issue May 18, 2017
Merged
@tomwilkie
Copy link
Contributor Author

This is in prod now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomwilkie tomwilkie

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tomwilkie