Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[coordinator] Add support for prometheus remote write compatible backend #3742

Merged
merged 33 commits into from
Sep 22, 2021

Conversation

Antanukas
Copy link
Collaborator

@Antanukas Antanukas commented Sep 13, 2021

What this PR does / why we need it:

Adds support for new backend type - prometheus remote write. This enables using M3 Coordinator and M3 Aggregator with Prometheus Remote Write compatible API.

Special notes for your reviewer:

Read part is not implemented mainly because using remote read API is not efficient since it requires to load raw timeseries.

In this PR integration with aggregator is not fully implemented. For now one can specify endpoints with no storage policy only so they will receive raw writes.

I plan to fix this in subsequent PR.

Example of configuration when data aggregation will be available:

backend: prom-remote

prometheusRemoteBackend:
  endpoints:
    - name: raw
      address: "http://raw-metrics/api/v1/prom/remote/write"
    - name: aggregated_30d
      address: "http://aggregated_30d/api/v1/prom/remote/write"
      storagePolicy:
        retention: 720h
        resolution: 1m
    - name: aggregated_60d
      address: "http://aggregated_60d/api/v1/prom/remote/write"
      storagePolicy:
        retention: 1440h
        resolution: 1m
		
clusterManagement:
  etcd:
    env: test-env
    zone: embedded
    service: m3db
    cacheDir: /var/lib/m3kv
    etcdClusters:
    - zone: embedded
      endpoints:
      - http://etcd-0:2379
      - http://etcd-1:2379
      - http://etcd-2:2379

Does this PR introduce a user-facing and/or backwards incompatible change?:

This feature will be available and can be enabled using configuration switch. 

Does this PR require updating code package or user-facing documentation?:

Documentation will follow after this PR is merged.

@Antanukas Antanukas force-pushed the antanas/prom-remote-write-backend branch from d783d29 to 7c19144 Compare September 13, 2021 14:10
@codecov
Copy link

codecov bot commented Sep 14, 2021

Codecov Report

Merging #3742 (4a18a72) into master (9fea52c) will decrease coverage by 0.0%.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #3742     +/-   ##
========================================
- Coverage    57.0%   57.0%   -0.1%     
========================================
  Files         552     552             
  Lines       62950   62950             
========================================
- Hits        35919   35913      -6     
- Misses      23840   23841      +1     
- Partials     3191    3196      +5     
Flag Coverage Δ
aggregator 63.3% <ø> (ø)
cluster ∅ <ø> (∅)
collector 58.4% <ø> (ø)
dbnode 60.7% <ø> (-0.1%) ⬇️
m3em 46.4% <ø> (ø)
metrics 19.7% <ø> (ø)
msg 74.3% <ø> (-0.2%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9fea52c...4a18a72. Read the comment docs.

@Antanukas Antanukas marked this pull request as ready for review September 14, 2021 08:52
Copy link
Collaborator

@vpranckaitis vpranckaitis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few initial comments, will continue reviewing tomorrow.

src/query/storage/promremote/storage.go Outdated Show resolved Hide resolved
src/query/storage/promremote/storage_unsupported.go Outdated Show resolved Hide resolved
src/cmd/services/m3query/config/config.go Outdated Show resolved Hide resolved
src/cmd/services/m3query/config/config.go Outdated Show resolved Hide resolved
src/query/server/query_test.go Outdated Show resolved Hide resolved
src/query/storage/promremote/storage.go Outdated Show resolved Hide resolved
src/query/server/query.go Show resolved Hide resolved
@vpranckaitis vpranckaitis self-requested a review September 14, 2021 15:22
@robskillington robskillington changed the title [coordiantor] add support for prometheus remote write compatible backend [coordinator] Add support for prometheus remote write compatible backend Sep 15, 2021
@Antanukas Antanukas force-pushed the antanas/prom-remote-write-backend branch 2 times, most recently from 6b64ad3 to b37164e Compare September 16, 2021 08:37
Copy link
Collaborator

@vpranckaitis vpranckaitis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, though it would be best if someone else would take a look, too.

src/query/storage/promremote/storage_unsupported.go Outdated Show resolved Hide resolved
src/query/storage/promremote/promremotetest/test_server.go Outdated Show resolved Hide resolved
src/query/storage/promremote/storage_unsupported.go Outdated Show resolved Hide resolved
src/cmd/services/m3query/config/config.go Outdated Show resolved Hide resolved
src/query/server/query.go Show resolved Hide resolved
src/query/server/query.go Show resolved Hide resolved
src/query/server/query_test.go Outdated Show resolved Hide resolved
src/query/server/query_test.go Outdated Show resolved Hide resolved
src/query/storage/promremote/storage_test.go Outdated Show resolved Hide resolved
src/query/storage/promremote/storage_test.go Outdated Show resolved Hide resolved
src/query/storage/promremote/storage_test.go Outdated Show resolved Hide resolved
src/query/storage/promremote/storage.go Show resolved Hide resolved
src/query/storage/promremote/storage_test.go Show resolved Hide resolved
src/query/storage/promremote/doc.go Outdated Show resolved Hide resolved
src/query/storage/promremote/options.go Outdated Show resolved Hide resolved
src/query/storage/promremote/query_coverter.go Outdated Show resolved Hide resolved
metrics.ReportError(methodDuration)
response, err := ioutil.ReadAll(resp.Body)
if err != nil {
response = []byte(fmt.Sprintf("error reading body: %v", err))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you can return a generic error now, and TODO just logging.

src/query/storage/promremote/storage_unsupported.go Outdated Show resolved Hide resolved
src/query/storage/promremote/query_converter_test.go Outdated Show resolved Hide resolved
src/query/storage/promremote/query_coverter.go Outdated Show resolved Hide resolved
@Antanukas Antanukas force-pushed the antanas/prom-remote-write-backend branch from b8a3265 to 8157671 Compare September 17, 2021 11:57
Copy link
Collaborator

@linasm linasm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a few comments.

src/query/storage/promremote/storage.go Outdated Show resolved Hide resolved
src/query/storage/promremote/query_converter_test.go Outdated Show resolved Hide resolved
src/query/storage/promremote/query_converter_test.go Outdated Show resolved Hide resolved
src/query/server/query.go Outdated Show resolved Hide resolved
src/query/server/query_test.go Show resolved Hide resolved
src/query/storage/promremote/options.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@robskillington robskillington left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except (outside of nits/minor comments) I do feel relatively strongly that we return an error when retention and resolution doesn't match any of the endpoints listed before we merge this (referencing this comment here https://github.com/m3db/m3/pull/3742/files#r713168023)

@Antanukas Antanukas merged commit f071279 into master Sep 22, 2021
@Antanukas Antanukas deleted the antanas/prom-remote-write-backend branch September 22, 2021 17:00
soundvibe pushed a commit that referenced this pull request Sep 23, 2021
* master:
  [dbnode,query] Allow blocking operations on main thread to be interrupted (#3764)
  [coordinator] Add support for prometheus remote write compatible backend (#3742)
  [tests] Add dbnode capable of being started in-process. (#3759)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:coordinator All issues pertaining to coordinator
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants