Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query] influxdb write improvements #3373

Merged
merged 3 commits into from
Oct 4, 2021

Conversation

tsharju
Copy link
Contributor

@tsharju tsharju commented Mar 19, 2021

What this PR does / why we need it:

For improved performance InfluxDB protocol suggest using Content-Encoding: gzip for the line protocol messages. At the moment, for example Telegraf sends InfluxDB output using gzip by default, influxdata/telegraf#8269. The current implementation of the M3 InfluxDB write handler did not support this. This PR adds this capability.

Another fix included is specifying the timestamp precision for the timestamp included in the InfluxDB line protocol. InfluxDB client implementations send the precision along with the write requests and if this is not taken into account, the timestamp value is parsed with incorrect value due to the missing precision information.

Also, added fix for handling the case where the HTTP request body is empty.

Special notes for your reviewer:

Does this PR introduce a user-facing and/or backwards incompatible change?:

NONE

Does this PR require updating code package or user-facing documentation?:

NONE

@tsharju tsharju marked this pull request as draft March 19, 2021 12:46
@tsharju tsharju force-pushed the tsharju-influxdb-write-improvements branch 2 times, most recently from 074d0f8 to 5c534ad Compare March 19, 2021 13:34
@tsharju tsharju marked this pull request as ready for review March 19, 2021 13:41
@codecov
Copy link

codecov bot commented Mar 19, 2021

Codecov Report

Merging #3373 (d9fed3d) into master (6b1a24d) will increase coverage by 15.5%.
The diff coverage is n/a.

❗ Current head d9fed3d differs from pull request most recent head 3bc2f6c. Consider uploading reports for the commit 3bc2f6c to get more accurate results

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3373      +/-   ##
==========================================
+ Coverage    56.8%    72.4%   +15.5%     
==========================================
  Files         552     1101     +549     
  Lines       63077   102840   +39763     
==========================================
+ Hits        35857    74498   +38641     
+ Misses      24016    23229     -787     
- Partials     3204     5113    +1909     
Flag Coverage Δ
aggregator 76.7% <0.0%> (+13.3%) ⬆️
cluster 84.9% <0.0%> (∅)
collector 84.3% <0.0%> (+25.8%) ⬆️
dbnode 79.0% <0.0%> (+18.5%) ⬆️
m3em 74.4% <0.0%> (+27.9%) ⬆️
m3ninx 73.6% <0.0%> (?)
metrics 19.8% <0.0%> (+<0.1%) ⬆️
msg 76.0% <0.0%> (+1.6%) ⬆️
query 67.1% <0.0%> (?)
x 79.7% <0.0%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b1a24d...3bc2f6c. Read the comment docs.

Copy link
Collaborator

@wesleyk wesleyk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tsharju
Copy link
Contributor Author

tsharju commented Jul 2, 2021

This PR has been approved for quite a while already. Would there be something required from me to get it merged?

@tsharju tsharju force-pushed the tsharju-influxdb-write-improvements branch 2 times, most recently from 50dfd4b to 155cc80 Compare September 20, 2021 10:40
@mhoffm-aiven mhoffm-aiven force-pushed the tsharju-influxdb-write-improvements branch from 155cc80 to 3bc2f6c Compare October 4, 2021 14:18
@wesleyk wesleyk merged commit 7672c09 into m3db:master Oct 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants