TSDB: document timestamp is outside of ranges of currently writable indices

[aspacca]

While working on #8429 I setup 20k events with a timestamp of 5m far from each other.

When I run the rally track the result was 100% failure ingestion rate.
Looking at the debug response ES cluster was returning the errors like the followin:

HTTP status: 400, message: the document timestamp [2023-12-04T07:27:45.000Z] is outside of ranges of currently writable indices [[2023-11-08T05:13:28.000Z,2023-11-08T09:13:28.000Z]

[ruflin]

@martijnvg @gizas The above is a TSDB rally track. How do we make this work?

[martijnvg]

I don't know which tsdb rally track this is. For tsdb k8s queries rally track we hard code that data streams to accept data from 2000-01-01T00:00:00Z to 2099-12-31T23:59:59Z.

[ruflin]

With elastic/elastic-package#1522 landed we have now the capability to directly generate rally tracks from integration packages as long as the corpus templates exist. Andrea linked to the example above in this PR: #8429 It generates the metrics for aws.sqs.

I think there are a few challenges here:

• The templates and everything is loaded through the package manager in Fleet. We only call the API.
• We only generate the data for the rally track and the config on where to send the data

That means, we are not in control of the template (and I think shouldn't be) as it is the template that will be used by all users and is what we actually want to test. This works except for TSDB.

How do we overcome this? Should elastic-package for the package template load an @custom template to overwrite start / end time? Should maybe rally to this by default?

[martijnvg]

How do we overcome this? Should elastic-package for the package template load an @Custom template to overwrite start / end time? Should maybe rally to this by default?

Does the data set that gets generated have fixed timestamps? If so, then I think adding a custom template that sets a fixed start and end time is an acceptable workaround.

The core of the problem is that tsdb data streams don't handle metrics that are older than 2 hours before the creation time of the tsdb data stream. If we want to address this we need to go over the ideas we have for indexing late or historic metrics and see whether those ideas make sense.

[aspacca]

Does the data set that gets generated have fixed timestamps? If so, then I think adding a custom template that sets a fixed start and end time is an acceptable workaround.

the data are dynamic and stars always on now

The core of the problem is that tsdb data streams don't handle metrics that are older than 2 hours before the creation time of the tsdb data stream

there's no data older than 2 hours before the creation time of the tsdb, but there might be for a month later

[ruflin]

Seems also for future data the same limitation applies. My understanding is, the TSDB team is working on getting this fixed but this will take some time. Long term I expect this to be fixed by rally or TSDB itself but because of timing I proposed the following hack.

elastic-package loads for each dataset it runs the benchmark on the @custom template with the following content:

https://github.com/elastic/rally-tracks/blob/dff7aa1a4f2e0133509d59a14fa1180cda666da5/tsdb_k8s_queries/pod-template.json#L94-L97

This means all data is indexed into a single index but assuming data is ingested between 2000 and 2099 we are fine. Unfortunately it means extra logic in the elastic-package command but I assume it is the quickest one to do.

If we have a dataset logs-foo the template that would have to be loaded is logs-foo@custom. We could go more specific and load logs-foo-ep@custom but I think that is overkill. The template will work without any further reference as it is already referenced in the template installed by the package. The loading of the custom template should happen directly before or after the installation of the package, both should work. I have not tested the above but in theory should get the job done. @martijnvg Thoughts?

@aspacca Would this possible to be added (and tested) to unblock this? For the rally tracks, I was also wondering if we should take the defined time range and do now()-timerange as the starting point. This makes debugging during / after the run easier as Kibana is tuned to show historical data.

[aspacca]

Would this possible to be added (and tested) to unblock this?

sure, I'll start working on it.

For the rally tracks, I was also wondering if we should take the defined time range and do now()-timerange as the starting point. This makes debugging during / after the run easier as Kibana is tuned to show historical data.

currently the timestamp is now + (period / tot events) * nth event)
meaning that if we have a period of 10 seconds and 10 events, each event is one second from the other starting from now

[martijnvg]

If we have a dataset logs-foo the template that would have to be loaded is logs-foo@custom. We could go more specific and load logs-foo-ep@custom but I think that is overkill. The template will work without any further reference as it is already referenced in the template installed by the package. The loading of the custom template should happen directly before or after the installation of the package, both should work. I have not tested the above but in theory should get the job done. @martijnvg Thoughts?

I think the custom component template approach should for Rally tracks that use tsdb. The index.time_series.start_time and index.time_series.end_time index settings can only be configured if the the index.mode setting has been set to time_series. So for tracks that don't use tsdb, these start and end time index settings shouldn't be configured. The custom component template can be configured (in case this makes that this makes this workaround easier).

[ruflin]

My proposal for the timestamp would be to change it to (now - period) + ((period / tot events) * nth event) so the last events that are shipped are now. When you open Kibana during the benchmark and you select "period" as the time range to now, you see all the data.

[aspacca]

If we have a dataset logs-foo the template that would have to be loaded is logs-foo@custom. We could go more specific and load logs-foo-ep@custom but I think that is overkill. The template will work without any further reference as it is already referenced in the template installed by the package. The loading of the custom template should happen directly before or after the installation of the package, both should work. I have not tested the above but in theory should get the job done. @martijnvg Thoughts?

I think the custom component template approach should for Rally tracks that use tsdb. The index.time_series.start_time and index.time_series.end_time index settings can only be configured if the the index.mode setting has been set to time_series. So for tracks that don't use tsdb, these start and end time index settings shouldn't be configured. The custom component template can be configured (in case this makes that this makes this workaround easier).

1. The template will be created as a create-component-template operation in the rally track: that's because the command has the option to persist the track with the corpus in order to replay it directly with rally. if we create the template with the command it will be lost then we replay the track. side note: I'm not 100% sure that I should tear down the template with delete-component-template operation afterwards.
2. this is the template body:

{
                    "template": {
                        "settings": {
                            "index": {
                                "mode": "time_series",
                                "time_series": {
                                    "start_time": "2000-01-01T00:00:00Z",
                                    "end_time": "2099-12-31T23:59:59Z"
                                }
                            }
                        }
                    }
                }
            }

it is added only for time series data stream. there is one problem: [index.mode=time_series] requires a non-empty [index.routing_path]

since the template body is generic we don't have a fixed set of dimensions for index.routing_path: I've tried with "routing_path": "*" with no luck, and I'm not sure of the implications as well. Ideally the templates for the datastream loaded at the package installation should have already all the proper settings, and I should be able to override just the index.time_series.start_time and index.time_series.end_time

[martijnvg]

I forgot about this detail. The index.routing_path normally doesn't need to be defined as it gets generated by composable index template that creates a tsdb data stream. In this case the index.mode field is defined in the component template. Each component template on its needs to be valid. In this case the index.mode index setting requires defining index.routing_path. The routing path does support wildcards.

Is it possible to to use a wildcard pattern that matches with a keyword dimension field that always exists?

[aspacca]

@martijnvg I ended up simulating the template installed by the package, manipulating it with the new end and start time and adding an higher priority for the index pattern of the datastream we run the rally benchmark against

see: elastic/elastic-package#1555

[martijnvg]

I was thinking about this is another possible solution as well. Great that this works out in that PR.

[ruflin]

I'm glad we find a workaround but I'm not too happy about the solution as it means we overwrite the full template: elastic/elastic-package#1555 (comment) Before we move forward with it, @martijnvg can you think of other potential solutions?

[ruflin]

As the routing path is generated out of the dimension fields, I wonder if having one field defined in the custom template would solve this, for example having host.name as dimension? It would again be a bit a hack but the assumption is, that host.name as dimension is always correct.

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!