Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fields.idx frequent writes cause lock contention #20500

Closed
davidby-influx opened this issue Jan 12, 2021 · 0 comments · Fixed by #20504
Closed

fields.idx frequent writes cause lock contention #20500

davidby-influx opened this issue Jan 12, 2021 · 0 comments · Fixed by #20504
Assignees
@davidby-influx
Labels
1.x area/performance area/writes kind/perf needs-backport/1.8 team/edge

Comments

@davidby-influx
Copy link
Contributor

The fields.idx file is completely rewritten every time a new measurement or field is added. The locking around this rewrite is causing slow writes at a customer who is creating 5M new fields.

Steps to reproduce:

  1. Add a new field or a new measurement to a database
  2. Note that fields.idx.tmp is written, and renamed to fields.idx each time
  3. During the creation of fields.idx.tmp, a lock is help preventing any new fields or measurements from being added.

Expected behavior:
The locking for updating fields.idx should be the minimum necessary.

Actual behavior:
Writes that create new fields and measurements are blocked while fields.idx is replaced.
Environment info:

  • InfluxDB version: v1.8.3 and earlier

Performance:
block.txt
mutex.txt
@davidby-influx davidby-influx self-assigned this Jan 12, 2021
This was referenced Jan 12, 2021
Closed
Closed
davidby-influx added a commit that referenced this issue Jan 12, 2021
@davidby-influx
fix(tsdb): minimize lock contention when adding new fields or measure…
0247ec9 
…ments

fields.idx frequent writes cause lock contention and fields.idx is recreated
when a field or measurement is added in a WritePointsWithContext()
This eliminates locking during the actual file rewrite, and limits it to
the times when the MeasurementFieldSet is actually being read or written

Fixes #20500
@davidby-influx davidby-influx mentioned this issue Jan 12, 2021
Merged
3 tasks
davidby-influx added a commit that referenced this issue Jan 13, 2021
@davidby-influx
chore(tsdb): code review refactor and test improvement for #20500
a82a426 
Addresses code review comments for better single-responsibility per method
Adds to test verification of correct behavior by checking the fields.idx
file matches the in-memory copy.

Fixes #20500
davidby-influx added a commit that referenced this issue Jan 13, 2021
@davidby-influx
chore: refactor for clarity on fix for #20500
70bfd92
@davidby-influx davidby-influx linked a pull request Jan 14, 2021 that will close this issue
fix(tsdb): minimize lock contention when adding new fields or measurements #20504
Merged
3 tasks
davidby-influx added a commit that referenced this issue Jan 15, 2021
@davidby-influx
fix(tsdb): minimize lock contention when adding new fields or measure…
fe3af66 
…ments (#20504)

fields.idx frequent writes cause lock contention and fields.idx is recreated
when a field or measurement is added in a WritePointsWithContext()
This eliminates locking during the actual file rewrite, and limits it to
the times when the MeasurementFieldSet is actually being read or written 
in memory and when the new file is being renamed.

Test verification of correct behavior by checking the fields.idx
file matches the in-memory copy after heavily parallel measurement addition.

Fixes #20500
davidby-influx added a commit that referenced this issue Mar 9, 2021
@davidby-influx
fix(tsdb): minimize lock contention when adding new fields or measure…
3d4938c 
…ments (#20504)

fields.idx frequent writes cause lock contention and fields.idx is recreated
when a field or measurement is added in a WritePointsWithContext()
This eliminates locking during the actual file rewrite, and limits it to
the times when the MeasurementFieldSet is actually being read or written
in memory and when the new file is being renamed.

Test verification of correct behavior by checking the fields.idx
file matches the in-memory copy after heavily parallel measurement addition.

Fixes #20500

(cherry picked from commit fe3af66)
@davidby-influx davidby-influx mentioned this issue Mar 9, 2021
Merged
4 tasks
davidby-influx added a commit that referenced this issue Mar 9, 2021
@davidby-influx
fix(tsdb): minimize lock contention when adding new fields or measure…
1d8fb76 
…ments (#20504)

fields.idx frequent writes cause lock contention and fields.idx is recreated
when a field or measurement is added in a WritePointsWithContext()
This eliminates locking during the actual file rewrite, and limits it to
the times when the MeasurementFieldSet is actually being read or written
in memory and when the new file is being renamed.

Test verification of correct behavior by checking the fields.idx
file matches the in-memory copy after heavily parallel measurement addition.

Fixes #20500

(cherry picked from commit fe3af66)
davidby-influx added a commit that referenced this issue Mar 10, 2021
@davidby-influx
fix(tsdb): minimize lock contention when adding new fields or measure (…
283ea0e 
…#20912)

fields.idx frequent writes cause lock contention and fields.idx is recreated
when a field or measurement is added in a WritePointsWithContext()
This eliminates locking during the actual file rewrite, and limits it to
the times when the MeasurementFieldSet is actually being read or written
in memory and when the new file is being renamed.

Test verification of correct behavior by checking the fields.idx
file matches the in-memory copy after heavily parallel measurement addition.

Fixes #20500

(cherry picked from commit fe3af66)
@davidby-influx davidby-influx linked a pull request Mar 15, 2021 that will close this issue
fix(tsdb): minimize lock contention when adding new fields or measure #20912
Merged
4 tasks
@davidby-influx davidby-influx removed a link to a pull request Mar 15, 2021
fix(tsdb): minimize lock contention when adding new fields or measure #20912
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@davidby-influx davidby-influx

Labels
1.x area/performance area/writes kind/perf needs-backport/1.8 team/edge
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(tsdb): minimize lock contention when adding new fields or measurements influxdata/influxdb
1 participant
@davidby-influx