Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename fast field precision parameter to fast_precision #3940

Closed
wants to merge 1 commit into from
Closed

Rename fast field precision parameter to fast_precision #3940

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion quickwit/quickwit-opentelemetry/src/otlp/traces.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,13 +99,14 @@ doc_mapping:
output_format: unix_timestamp_nanos
indexed: false
fast: true
precision: milliseconds
precision: nanoseconds
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the fast field precision. The stored value remains in nanoseconds. The fast field won't compress as well with a precision in nanoseconds, that's why we opted for milliseconds since we were not sure it made sense for filtering and aggregations.

Do you have a use case for fast fields with nanoseconds precision?

@fmassot, @fulmicoton, do we want to revisit this decision?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'am exporing the tantivy index from quickwit's opentelementry traces index. I found the datetime field's precision is wrong:

  1. the span_start_timestamp_nanos field in quickwit_opentelemetry::otlp::traces::Span use nanoseconds
  2. the index's field use milliseconds
  3. the field's data stored in tantivy index use microseconds

I think the precision of span_start_timestamp_nanos should be nanoseconds according to the field's name.

Or microseconds? (the data stored in index is in microseconds)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or rename precision to fast_precision indicating that is for fastfield?

Copy link
Member

@guilload guilload Oct 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. the field's data stored in tantivy index use microseconds

We should be on a tantivy version that uses nanoseconds for storing datetimes (quickwit-oss/tantivy#2016)

Or rename precision to fast_precision indicating that is for fastfield?

I totally agree.

- name: span_end_timestamp_nanos
type: datetime
input_formats: [unix_timestamp]
output_format: unix_timestamp_nanos
indexed: false
fast: false
precision: nanoseconds
- name: span_duration_millis
type: u64
indexed: false
Expand Down