Cherry-pick TimeSeriesDatabase from megacarbon branch

[iain-buclaw-sociomantic]

It is becoming quite a pet peeve of mine to be stuck on the abandoned megacarbon branch and having to manually merge in upstream fixes. :-)

While the megacarbon branch does a lot more, it is really only this part that is really relevant (to me), where all database actions are abstracted away, allowing any other database implementation to be plugged in.

I've stopped short of duplicating parseRetentionDefs from whisper and adding a ceres backend. If there is no object against such things, I will push forward with those too. However what's been changed here is enough to allow a pause for review.

[mleinart]

Thanks, I'll try and take a look at this today
On Dec 3, 2015 4:18 AM, "Iain Buclaw" [email protected] wrote:

It is becoming quite a pet peeve of mine to be stuck on the abandoned
megacarbon branch and having to manually merge in upstream fixes. :-)

While the megacarbon branch does a lot more, it is really only this part
that is really relevant (to me), where all database actions are abstracted
away, allowing any other database implementation to be plugged in.

I've stopped short of duplicating parseRetentionDefs from whisper and
adding a ceres backend. If there is no object against such things, I will
push forward with those too. However what's been changed here is enough to

allow a pause for review.

You can view, comment on, or merge this pull request online at:

#484
Commit Summary

• Add abstract TimeSeriesDatabase class with Whisper override
• Move whisper configuration to WhisperDatabase
• Move write action into WhisperDatabase
• Move exists action into WhisperDatabase
• Move create action into WhisperDatabase
• Move get/set metadata action into WhisperDatabase
• Remove dependency on getFilesystemPath

File Changes

• M conf/carbon.conf.example
  https://github.com/graphite-project/carbon/pull/484/files#diff-0 (4)
• M lib/carbon/conf.py
  https://github.com/graphite-project/carbon/pull/484/files#diff-1
  (27)
• A lib/carbon/database.py
  https://github.com/graphite-project/carbon/pull/484/files#diff-2
  (111)
• M lib/carbon/management.py
  https://github.com/graphite-project/carbon/pull/484/files#diff-3
  (17)
• M lib/carbon/state.py
  https://github.com/graphite-project/carbon/pull/484/files#diff-4 (1)
• M lib/carbon/storage.py
  https://github.com/graphite-project/carbon/pull/484/files#diff-5 (5)
• M lib/carbon/writer.py
  https://github.com/graphite-project/carbon/pull/484/files#diff-6
  (39)

Patch Links:

• https://github.com/graphite-project/carbon/pull/484.patch
• https://github.com/graphite-project/carbon/pull/484.diff

—
Reply to this email directly or view it on GitHub
#484.

[iain-buclaw-sociomantic]

OK, thanks.

FYI, I've finished the conversion here: master...iain-buclaw-sociomantic:tsdatabase2

Doing some small tests and it all works fine with both whisper and ceres.

[iain-buclaw-sociomantic]

Rebased, and fiddled with the getFilesystemPath test until it passed.

[mleinart]

Note: when I say "I'll look at this today" it often means "I'll start feeling guilty about not looking at this starting today" ;)

I promise I haven't forgotten about this. This is something that has needed to be done for some time, thanks for the work!

[mleinart]

I can kindof go either way with the change to TimeSeriesDatabase.create()'s method signature from (self, metric, **options) to specifying the options explicitly, but I'm just going to ship as is.

FYI, I'd like to hold off merging Ceres from the other branch until graphite-web supports reading past retentions which is something I've been working on getting put in

[mleinart]

Merged in eeba611

[iain-buclaw-sociomantic]

Note: when I say "I'll look at this today" it often means "I'll start feeling guilty about not looking at this starting today" ;)

I am on holidays so there is no rush on my side, yet. :-)

I can kindof go either way with the change to TimeSeriesDatabase.create()'s method signature from (self, metric, **options) to specifying the options explicitly, but I'm just going to ship as is.

In megacarbon, there is a determine_metadata function which returns a value to pass as this param. I'm not really fond of the signature, but when considering any of the following.

• The absence of any function/method to return 'determined' metadata that is relevant for the storage database.
• Whisper and Ceres use the same config options.
• I'm not aware of any other storage backend for carbon-cache (other tsdb's implement the carbon protocol natively).

I chose the quick path for now, with intent to properly address it later.

FYI, I'd like to hold off merging Ceres from the other branch until graphite-web supports reading past retentions which is something I've been working on getting put in

This is something I implemented in Ceres itself for now (I think you know this), I should rebase that PR for FYI purposes. Good to know that someone is working on pushing it out to the client though, that is definitely the right way to go about it.

I'll raise a supplementary PR then which completes the TimeSeriesDatabase API (but not the create signature), stopping short of actual Ceres support.