Speed up date_histogram by precomputing ranges #61467

nik9000 · 2020-08-24T13:02:40Z

A few of us were talking about ways to speed up the date_histogram
using the index for the timestamp rather than the doc values. To do that
we'd have to pre-compute all of the "round down" points in the index. It
turns out that just precomputing those values speeds up rounding
fairly significantly:

Benchmark  (count)      (interval)                   (range)            (zone)  Mode  Cnt          Score         Error  Units
before    10000000  calendar month  2000-10-28 to 2000-10-31               UTC  avgt   10   96461080.982 ±  616373.011  ns/op
before    10000000  calendar month  2000-10-28 to 2000-10-31  America/New_York  avgt   10  130598950.850 ± 1249189.867  ns/op
after     10000000  calendar month  2000-10-28 to 2000-10-31               UTC  avgt   10   52311775.080 ±  107171.092  ns/op
after     10000000  calendar month  2000-10-28 to 2000-10-31  America/New_York  avgt   10   54800134.968 ±  373844.796  ns/op

That's a 46% speed up when there isn't a time zone and a 58% speed up
when there is.

This doesn't work for every time zone, specifically those that have two
midnights in a single day due to daylight savings time will produce wonky
results. So they don't get the optimization.

Second, this requires a few expensive computation up front to make the
transition array. And if the transition array is too large then we give
up and use the original mechanism, throwing away all of the work we did
to build the array. This seems appropriate for most usages of round,
but this change uses it for all usages of round. That seems ok for
now, but it might be worth investigating in a follow up.

I ran a macrobenchmark as well which showed an 11% preformance
improvement. BUT the benchmark wasn't tuned for my desktop so it
overwhelmed it and might have produced "funny" results. I think it is
pretty clear that this is an improvement, but know the measurement is
weird:

Before:
|               Min Throughput | hourly_agg |        0.11 |  ops/s |
|            Median Throughput | hourly_agg |        0.11 |  ops/s |
|               Max Throughput | hourly_agg |        0.11 |  ops/s |
|      50th percentile latency | hourly_agg |      650623 |     ms |
|      90th percentile latency | hourly_agg |      821478 |     ms |
|      99th percentile latency | hourly_agg |      859780 |     ms |
|     100th percentile latency | hourly_agg |      864030 |     ms |
| 50th percentile service time | hourly_agg |     9268.71 |     ms |
| 90th percentile service time | hourly_agg |        9380 |     ms |
| 99th percentile service time | hourly_agg |     9626.88 |     ms |
|100th percentile service time | hourly_agg |     9884.27 |     ms |
|                   error rate | hourly_agg |           0 |      % |

After:
|               Min Throughput | hourly_agg |        0.12 |  ops/s |
|            Median Throughput | hourly_agg |        0.12 |  ops/s |
|               Max Throughput | hourly_agg |        0.12 |  ops/s |
|      50th percentile latency | hourly_agg |      519254 |     ms |
|      90th percentile latency | hourly_agg |      653099 |     ms |
|      99th percentile latency | hourly_agg |      683276 |     ms |
|     100th percentile latency | hourly_agg |      686611 |     ms |
| 50th percentile service time | hourly_agg |     8371.41 |     ms |
| 90th percentile service time | hourly_agg |     8407.02 |     ms |
| 99th percentile service time | hourly_agg |     8536.64 |     ms |
|100th percentile service time | hourly_agg |     8538.54 |     ms |
|                   error rate | hourly_agg |           0 |      % |

elasticmachine · 2020-08-24T13:02:42Z

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

A few of us were talking about ways to speed up the `date_histogram` using the index for the timestamp rather than the doc values. To do that we'd have to pre-compute all of the "round down" points in the index. It turns out that *just* precomputing those values speeds up rounding fairly significantly: ``` Benchmark (count) (interval) (range) (zone) Mode Cnt Score Error Units before 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 96461080.982 ± 616373.011 ns/op before 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 130598950.850 ± 1249189.867 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 52311775.080 ± 107171.092 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 54800134.968 ± 373844.796 ns/op ``` That's a 46% speed up when there isn't a time zone and a 58% speed up when there is. This doesn't work for every time zone, specifically those that have two midnights in a single day due to daylight savings time will produce wonky results. So they don't get the optimization. Second, this requires a few expensive computation up front to make the transition array. And if the transition array is too large then we give up and use the original mechanism, throwing away all of the work we did to build the array. This seems appropriate for most usages of `round`, but this change uses it for *all* usages of `round`. That seems ok for now, but it might be worth investigating in a follow up. I ran a macrobenchmark as well which showed an 11% preformance improvement. *BUT* the benchmark wasn't tuned for my desktop so it overwhelmed it and might have produced "funny" results. I think it is pretty clear that this is an improvement, but know the measurement is weird: ``` Benchmark (count) (interval) (range) (zone) Mode Cnt Score Error Units before 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 96461080.982 ± 616373.011 ns/op before 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 g± 1249189.867 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 52311775.080 ± 107171.092 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 54800134.968 ± 373844.796 ns/op Before: | Min Throughput | hourly_agg | 0.11 | ops/s | | Median Throughput | hourly_agg | 0.11 | ops/s | | Max Throughput | hourly_agg | 0.11 | ops/s | | 50th percentile latency | hourly_agg | 650623 | ms | | 90th percentile latency | hourly_agg | 821478 | ms | | 99th percentile latency | hourly_agg | 859780 | ms | | 100th percentile latency | hourly_agg | 864030 | ms | | 50th percentile service time | hourly_agg | 9268.71 | ms | | 90th percentile service time | hourly_agg | 9380 | ms | | 99th percentile service time | hourly_agg | 9626.88 | ms | |100th percentile service time | hourly_agg | 9884.27 | ms | | error rate | hourly_agg | 0 | % | After: | Min Throughput | hourly_agg | 0.12 | ops/s | | Median Throughput | hourly_agg | 0.12 | ops/s | | Max Throughput | hourly_agg | 0.12 | ops/s | | 50th percentile latency | hourly_agg | 519254 | ms | | 90th percentile latency | hourly_agg | 653099 | ms | | 99th percentile latency | hourly_agg | 683276 | ms | | 100th percentile latency | hourly_agg | 686611 | ms | | 50th percentile service time | hourly_agg | 8371.41 | ms | | 90th percentile service time | hourly_agg | 8407.02 | ms | | 99th percentile service time | hourly_agg | 8536.64 | ms | |100th percentile service time | hourly_agg | 8538.54 | ms | | error rate | hourly_agg | 0 | % | ```

nik9000 · 2020-08-24T13:04:59Z

Instead of building a shed or playing video games this was my weekend project......

I think it is a start towards the kind of thing that we'd want to use the lucene index to speed up date ranges. And I think it is a nice incremental improvement. And that it'll still apply in some cases when we don't want to use the search index. Also, it manages to re-use a ton of tests to make sure that the pre-computing is correct.

jpountz

I like the idea.

jpountz · 2020-08-24T13:26:51Z

server/src/main/java/org/elasticsearch/common/Rounding.java

+         * Time zones with two midnights get "funny" non-continuous rounding
+         * that isn't compatible with the pre-computed array rounding.
+         */
+        private static final Set<String> HAS_TWO_MIDNIGHTS = Set.of("America/Moncton", "America/St_Johns", "Canada/Newfoundland");


would it be doable to detect timezones that don't work with this optimization at runtime instead of maintaining an allowlist?

I can't think of a way to do right now. At least, not a good way.

I thought Canada was changing time at 2 am, on another side, some other known time zones such as Asia/Gaza for example seems to indeed have 2 mightnights. I did some quick and dirty test and I have got a slightly different list for timezones in my JVM:

America/Asuncion 2020-03-22T00:00 -> 2020-03-21T23:00 America/Havana 2020-11-01T01:00 -> 2020-11-01T00:00 America/Santiago 2020-04-05T00:00 -> 2020-04-04T23:00 America/Scoresbysund 2020-10-25T01:00 -> 2020-10-25T00:00 Asia/Amman 2020-10-30T01:00 -> 2020-10-30T00:00 Asia/Beirut 2020-10-25T00:00 -> 2020-10-24T23:00 Asia/Damascus 2020-10-30T00:00 -> 2020-10-29T23:00 Asia/Gaza 2020-10-31T01:00 -> 2020-10-31T00:00 Asia/Hebron 2020-10-31T01:00 -> 2020-10-31T00:00 Asia/Tehran 2020-09-21T00:00 -> 2020-09-20T23:00 Atlantic/Azores 2020-10-25T01:00 -> 2020-10-25T00:00 Chile/Continental 2020-04-05T00:00 -> 2020-04-04T23:00 Cuba 2020-11-01T01:00 -> 2020-11-01T00:00 Iran 2020-09-21T00:00 -> 2020-09-20T23:00

Is this really only a problem when there are two midnights, or is it a more general problem when the time goes back due to daylight savings, and you just need special values of offset to make the bug occur if the transition is not around midnight?

I've played with offset to double check and can't cause issues with it. However do you think we could detect timezones that don't work dynamically instead of relying on a static list? E.g. could we iterate transitions in the considered interval and check whether there's one that brings us to a different day?

We could use something like the test case uses to find candidates, but it'd require loading all of the time zone rules on startup. I'm hoping that the test can prevent us from having to do that by being very careful about this list.

Actually I wasn't thinking about testing all timezones up-front on startup, I was more thinking of doing the test when building the Rounding object by only looking at transitions of the considered timezone.

Hmmmm - the assert that I have below sort of asserts that. But it isn't nearly as strong. I'll think about it!

@jpountz I've pushed a change that does this.

jpountz · 2020-08-24T13:28:17Z

server/src/main/java/org/elasticsearch/common/Rounding.java

@@ -1015,7 +1031,7 @@ public byte id() {

        @Override
        public Prepared prepare(long minUtcMillis, long maxUtcMillis) {
-            return wrapPreparedRounding(delegate.prepare(minUtcMillis, maxUtcMillis));
+            return wrapPreparedRounding(delegate.prepare(minUtcMillis - offset, maxUtcMillis - offset));


is this change fixing an existing bug?

Yes, but I think the bug is worse with this change.

The new assertions in the ArrayRounding fail without this.

jpountz · 2020-08-24T13:29:55Z

server/src/main/java/org/elasticsearch/common/Rounding.java

+     */
+    private static class ArrayRounding implements Prepared {
+        private final long[] values;
+        private int max;


make it final?

👍. I have no idea why I didn't do that the first time around.

jpountz · 2020-08-24T13:30:11Z

server/src/main/java/org/elasticsearch/common/Rounding.java

+        @Override
+        public long round(long utcMillis) {
+            int idx = Arrays.binarySearch(values, 0, max, utcMillis);
+            if (idx < 0) {


maybe assert that idx is neither -1 nor -1 - max?

imotov

Looks good to me in general, but timezone list is also a concern of mine, I played a bit with ZoneRules, the information that is needed is definitely there, we just need to figure out how to correctly interpret it.

imotov · 2020-08-26T20:47:35Z

server/src/main/java/org/elasticsearch/common/Rounding.java

+         * Time zones with two midnights get "funny" non-continuous rounding
+         * that isn't compatible with the pre-computed array rounding.
+         */
+        private static final Set<String> HAS_TWO_MIDNIGHTS = Set.of("America/Moncton", "America/St_Johns", "Canada/Newfoundland");


I thought Canada was changing time at 2 am, on another side, some other known time zones such as Asia/Gaza for example seems to indeed have 2 mightnights. I did some quick and dirty test and I have got a slightly different list for timezones in my JVM:

America/Asuncion 2020-03-22T00:00 -> 2020-03-21T23:00 America/Havana 2020-11-01T01:00 -> 2020-11-01T00:00 America/Santiago 2020-04-05T00:00 -> 2020-04-04T23:00 America/Scoresbysund 2020-10-25T01:00 -> 2020-10-25T00:00 Asia/Amman 2020-10-30T01:00 -> 2020-10-30T00:00 Asia/Beirut 2020-10-25T00:00 -> 2020-10-24T23:00 Asia/Damascus 2020-10-30T00:00 -> 2020-10-29T23:00 Asia/Gaza 2020-10-31T01:00 -> 2020-10-31T00:00 Asia/Hebron 2020-10-31T01:00 -> 2020-10-31T00:00 Asia/Tehran 2020-09-21T00:00 -> 2020-09-20T23:00 Atlantic/Azores 2020-10-25T01:00 -> 2020-10-25T00:00 Chile/Continental 2020-04-05T00:00 -> 2020-04-04T23:00 Cuba 2020-11-01T01:00 -> 2020-11-01T00:00 Iran 2020-09-21T00:00 -> 2020-09-20T23:00

imotov · 2020-08-26T20:50:03Z

server/src/main/java/org/elasticsearch/common/Rounding.java

+    /**
+     * Implementation of {@link Prepared} using pre-calculated "round down" points.
+     */
+    private static class ArrayRounding implements Prepared {


This will need to implement new methods added in #61369.

nik9000 · 2020-08-27T22:25:18Z

I believe it'd be safe so long as no dst transition causes the round function to go backwards. I think I can get a list of those in a test. It's only those that transitioned from a time after midnight to a time before it. Mostly the "two midnights" case doesn't hit this because they transition right on midnight. Those always round down which is correct. Beyond that, I'm hoping we can rely on the randomized testing that compares this against the known good java until time implementation to catch things

…

On Thu, Aug 27, 2020, 16:54 Adrien Grand ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In server/src/main/java/org/elasticsearch/common/Rounding.java <#61467 (comment)> : > @@ -401,8 +404,22 @@ private LocalDateTime truncateLocalDateTime(LocalDateTime localDateTime) { } } + /** + * Time zones with two midnights get "funny" non-continuous rounding + * that isn't compatible with the pre-computed array rounding. + */ + private static final Set<String> HAS_TWO_MIDNIGHTS = Set.of("America/Moncton", "America/St_Johns", "Canada/Newfoundland"); Even if offsets are not working as advertised with DST, I worry that they would behave differently again with this optimization. It would be safer to disable this optimization whenever there are transitions that make the time go backwards in the local time in the time range of the index? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61467 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUXIX3HKTYGD53PE7K6RTSC3BXRANCNFSM4QJPHWDQ> .

not-napoleon

Realized I started a review and never submitted it. Sorry about that. LGTM, couple nits if you feel like addressing them.

not-napoleon · 2020-09-02T16:55:27Z

server/src/main/java/org/elasticsearch/common/Rounding.java

+         * rounding if the transition is <strong>at</strong> midnight. Just not
+         * if it is <strong>after</strong> it.
+         */
+        private static final Map<String, Long> FORWARDS_BACKWRADS_ZONES = Map.ofEntries(


Nit - It looks like this maps timezone name to milliseconds since epoch for when the zone changed to use a different transition time. Can you just add a comment clarifying that's how the map is structured? It seems unlikely we'd need to add to it, but just in case we do, it'd help to have a reference for how to do so handy.

not-napoleon · 2020-09-02T17:26:20Z

server/src/main/java/org/elasticsearch/common/Rounding.java

+            if (i >= max) {
+                return orig;
+            }
+            assert values[i - 1] == orig.round(rounded - 1);


It's not super clear to me what this assert is guarding against. Can you add a comment please?

nik9000 · 2020-09-15T17:05:08Z

server/src/test/java/org/elasticsearch/common/LocalTimeOffsetTests.java

+        assertKnownMovesBacktoPreviousDay("America/Moncton", "2005-10-29T03:01:00");
+        assertKnownMovesBacktoPreviousDay("America/St_Johns", "2010-11-07T02:31:00");
+        assertKnownMovesBacktoPreviousDay("Canada/Newfoundland", "2010-11-07T02:31:00");
+        assertKnownMovesBacktoPreviousDay("Pacific/Guam", "1969-01-25T13:01:00");


This test fails in CI but not for me locally! I'm looking into it. Might be the specific JVM or something.

As far as I understand this list constantly changes, so it might be different between versions of JVM and OSes.

I had sort of thought the past was relatively well established, but it looks like it is a Java 12+ thing....

nik9000 · 2020-09-23T12:53:09Z

@jpountz would you like to have another look at this? I think I managed to remove the list of bad time zones.

jpountz · 2020-09-23T19:55:14Z

server/src/main/java/org/elasticsearch/common/LocalTimeOffset.java

+            if (transition.getDateTimeBefore().getDayOfMonth() == transition.getDateTimeAfter().getDayOfMonth()) {
+                return false;
+            }
+            if (transition.getDateTimeBefore().getMinute() == 0) {


should we check getHour() too for safety? Eg. if such a thing existed as moving from 1AM to 11PM the day before it looks like we'd still consider that the transition doesn't move back to the previous day with this code?

I pushed a change that makes sure the "NANO_OF_DAY" is 0. That seems fairly good at covering it

nik9000 · 2020-09-24T12:52:51Z

Thanks @jpountz !

nik9000 · 2020-09-24T13:39:46Z

run elasticsearch-ci/packaging-sample-windows

A few of us were talking about ways to speed up the `date_histogram` using the index for the timestamp rather than the doc values. To do that we'd have to pre-compute all of the "round down" points in the index. It turns out that *just* precomputing those values speeds up rounding fairly significantly: ``` Benchmark (count) (interval) (range) (zone) Mode Cnt Score Error Units before 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 96461080.982 ± 616373.011 ns/op before 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 130598950.850 ± 1249189.867 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 52311775.080 ± 107171.092 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 54800134.968 ± 373844.796 ns/op ``` That's a 46% speed up when there isn't a time zone and a 58% speed up when there is. This doesn't work for every time zone, specifically those that have two midnights in a single day due to daylight savings time will produce wonky results. So they don't get the optimization. Second, this requires a few expensive computation up front to make the transition array. And if the transition array is too large then we give up and use the original mechanism, throwing away all of the work we did to build the array. This seems appropriate for most usages of `round`, but this change uses it for *all* usages of `round`. That seems ok for now, but it might be worth investigating in a follow up. I ran a macrobenchmark as well which showed an 11% preformance improvement. *BUT* the benchmark wasn't tuned for my desktop so it overwhelmed it and might have produced "funny" results. I think it is pretty clear that this is an improvement, but know the measurement is weird: ``` Benchmark (count) (interval) (range) (zone) Mode Cnt Score Error Units before 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 96461080.982 ± 616373.011 ns/op before 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 g± 1249189.867 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 52311775.080 ± 107171.092 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 54800134.968 ± 373844.796 ns/op Before: | Min Throughput | hourly_agg | 0.11 | ops/s | | Median Throughput | hourly_agg | 0.11 | ops/s | | Max Throughput | hourly_agg | 0.11 | ops/s | | 50th percentile latency | hourly_agg | 650623 | ms | | 90th percentile latency | hourly_agg | 821478 | ms | | 99th percentile latency | hourly_agg | 859780 | ms | | 100th percentile latency | hourly_agg | 864030 | ms | | 50th percentile service time | hourly_agg | 9268.71 | ms | | 90th percentile service time | hourly_agg | 9380 | ms | | 99th percentile service time | hourly_agg | 9626.88 | ms | |100th percentile service time | hourly_agg | 9884.27 | ms | | error rate | hourly_agg | 0 | % | After: | Min Throughput | hourly_agg | 0.12 | ops/s | | Median Throughput | hourly_agg | 0.12 | ops/s | | Max Throughput | hourly_agg | 0.12 | ops/s | | 50th percentile latency | hourly_agg | 519254 | ms | | 90th percentile latency | hourly_agg | 653099 | ms | | 99th percentile latency | hourly_agg | 683276 | ms | | 100th percentile latency | hourly_agg | 686611 | ms | | 50th percentile service time | hourly_agg | 8371.41 | ms | | 90th percentile service time | hourly_agg | 8407.02 | ms | | 99th percentile service time | hourly_agg | 8536.64 | ms | |100th percentile service time | hourly_agg | 8538.54 | ms | | error rate | hourly_agg | 0 | % | ```

…62880) A few of us were talking about ways to speed up the `date_histogram` using the index for the timestamp rather than the doc values. To do that we'd have to pre-compute all of the "round down" points in the index. It turns out that *just* precomputing those values speeds up rounding fairly significantly: ``` Benchmark (count) (interval) (range) (zone) Mode Cnt Score Error Units before 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 96461080.982 ± 616373.011 ns/op before 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 130598950.850 ± 1249189.867 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 52311775.080 ± 107171.092 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 54800134.968 ± 373844.796 ns/op ``` That's a 46% speed up when there isn't a time zone and a 58% speed up when there is. This doesn't work for every time zone, specifically those that have two midnights in a single day due to daylight savings time will produce wonky results. So they don't get the optimization. Second, this requires a few expensive computation up front to make the transition array. And if the transition array is too large then we give up and use the original mechanism, throwing away all of the work we did to build the array. This seems appropriate for most usages of `round`, but this change uses it for *all* usages of `round`. That seems ok for now, but it might be worth investigating in a follow up. I ran a macrobenchmark as well which showed an 11% preformance improvement. *BUT* the benchmark wasn't tuned for my desktop so it overwhelmed it and might have produced "funny" results. I think it is pretty clear that this is an improvement, but know the measurement is weird: ``` Benchmark (count) (interval) (range) (zone) Mode Cnt Score Error Units before 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 96461080.982 ± 616373.011 ns/op before 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 g± 1249189.867 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 52311775.080 ± 107171.092 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 54800134.968 ± 373844.796 ns/op Before: | Min Throughput | hourly_agg | 0.11 | ops/s | | Median Throughput | hourly_agg | 0.11 | ops/s | | Max Throughput | hourly_agg | 0.11 | ops/s | | 50th percentile latency | hourly_agg | 650623 | ms | | 90th percentile latency | hourly_agg | 821478 | ms | | 99th percentile latency | hourly_agg | 859780 | ms | | 100th percentile latency | hourly_agg | 864030 | ms | | 50th percentile service time | hourly_agg | 9268.71 | ms | | 90th percentile service time | hourly_agg | 9380 | ms | | 99th percentile service time | hourly_agg | 9626.88 | ms | |100th percentile service time | hourly_agg | 9884.27 | ms | | error rate | hourly_agg | 0 | % | After: | Min Throughput | hourly_agg | 0.12 | ops/s | | Median Throughput | hourly_agg | 0.12 | ops/s | | Max Throughput | hourly_agg | 0.12 | ops/s | | 50th percentile latency | hourly_agg | 519254 | ms | | 90th percentile latency | hourly_agg | 653099 | ms | | 99th percentile latency | hourly_agg | 683276 | ms | | 100th percentile latency | hourly_agg | 686611 | ms | | 50th percentile service time | hourly_agg | 8371.41 | ms | | 90th percentile service time | hourly_agg | 8407.02 | ms | | 99th percentile service time | hourly_agg | 8536.64 | ms | |100th percentile service time | hourly_agg | 8538.54 | ms | | error rate | hourly_agg | 0 | % | ```

nik9000 · 2020-09-27T21:29:37Z

I looked at implementing this for interval roundings (7d instead of month) and I can't do it easily! It turns out that a critical method nextRoundingValue is kind of "weird" for those roundings. It doesn't really do what it was documented to do. I think we have to fix that first, then we can likely use the this technique to speed up those roundings.

We could also start investigating speeding up date_histogram by not rounding at all - now that we have the bounds we can use a search instead of loading the values. Something like that!

Also, we now have a much better estimate for the upper bound of the number of buckets a particular date_histogram could make. We don't do anything with that information now, but I expect we could save quite a lot of memory with it.

…62881) A few of us were talking about ways to speed up the `date_histogram` using the index for the timestamp rather than the doc values. To do that we'd have to pre-compute all of the "round down" points in the index. It turns out that *just* precomputing those values speeds up rounding fairly significantly: ``` Benchmark (count) (interval) (range) (zone) Mode Cnt Score Error Units before 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 96461080.982 ± 616373.011 ns/op before 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 130598950.850 ± 1249189.867 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 52311775.080 ± 107171.092 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 54800134.968 ± 373844.796 ns/op ``` That's a 46% speed up when there isn't a time zone and a 58% speed up when there is. This doesn't work for every time zone, specifically those that have two midnights in a single day due to daylight savings time will produce wonky results. So they don't get the optimization. Second, this requires a few expensive computation up front to make the transition array. And if the transition array is too large then we give up and use the original mechanism, throwing away all of the work we did to build the array. This seems appropriate for most usages of `round`, but this change uses it for *all* usages of `round`. That seems ok for now, but it might be worth investigating in a follow up. I ran a macrobenchmark as well which showed an 11% preformance improvement. *BUT* the benchmark wasn't tuned for my desktop so it overwhelmed it and might have produced "funny" results. I think it is pretty clear that this is an improvement, but know the measurement is weird: ``` Benchmark (count) (interval) (range) (zone) Mode Cnt Score Error Units before 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 96461080.982 ± 616373.011 ns/op before 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 g± 1249189.867 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 UTC avgt 10 52311775.080 ± 107171.092 ns/op after 10000000 calendar month 2000-10-28 to 2000-10-31 America/New_York avgt 10 54800134.968 ± 373844.796 ns/op Before: | Min Throughput | hourly_agg | 0.11 | ops/s | | Median Throughput | hourly_agg | 0.11 | ops/s | | Max Throughput | hourly_agg | 0.11 | ops/s | | 50th percentile latency | hourly_agg | 650623 | ms | | 90th percentile latency | hourly_agg | 821478 | ms | | 99th percentile latency | hourly_agg | 859780 | ms | | 100th percentile latency | hourly_agg | 864030 | ms | | 50th percentile service time | hourly_agg | 9268.71 | ms | | 90th percentile service time | hourly_agg | 9380 | ms | | 99th percentile service time | hourly_agg | 9626.88 | ms | |100th percentile service time | hourly_agg | 9884.27 | ms | | error rate | hourly_agg | 0 | % | After: | Min Throughput | hourly_agg | 0.12 | ops/s | | Median Throughput | hourly_agg | 0.12 | ops/s | | Max Throughput | hourly_agg | 0.12 | ops/s | | 50th percentile latency | hourly_agg | 519254 | ms | | 90th percentile latency | hourly_agg | 653099 | ms | | 99th percentile latency | hourly_agg | 683276 | ms | | 100th percentile latency | hourly_agg | 686611 | ms | | 50th percentile service time | hourly_agg | 8371.41 | ms | | 90th percentile service time | hourly_agg | 8407.02 | ms | | 99th percentile service time | hourly_agg | 8536.64 | ms | |100th percentile service time | hourly_agg | 8538.54 | ms | | error rate | hourly_agg | 0 | % | ```

This speeds up `date_histogram` aggregations without a parent or children. This is quite common - it's the aggregation that Kibana's Discover uses all over the place. Also, we hope to be able to use the same mechanism to speed aggs with children one day, but that day isn't today. The kind of speedup we're seeing is fairly substantial in many cases: ``` | | | before | after | | | 90th percentile service time | date_histogram_calendar_interval | 9266.07 | 1376.13 | ms | | 90th percentile service time | date_histogram_calendar_interval_with_tz | 9217.21 | 1372.67 | ms | | 90th percentile service time | date_histogram_fixed_interval | 8817.36 | 1312.67 | ms | | 90th percentile service time | date_histogram_fixed_interval_with_tz | 8801.71 | 1311.69 | ms | <-- discover's agg | 90th percentile service time | date_histogram_fixed_interval_with_metrics | 44660.2 | 43789.5 | ms | ``` This uses the work we did in #61467 to precompute the rounding points for a `date_histogram`. Now, when we know the rounding points we execute the `date_histogram` as a `range` aggregation. This is nice for two reasons: 1. We can further rewrite the `range` aggregation (see below) 2. We don't need to allocate a hash to convert rounding points to ordinals. 3. We can send precise cardinality estimates to sub-aggs. Points 2 and 3 above are nice, but most of the speed difference comes from point 1. Specifically, we now look into executing `range` aggregations as a `filters` aggregation. Normally the `filters` aggregation is quite slow but when it doesn't have a parent or any children then we can execute it "filter by filter" which is significantly faster. So fast, in fact, that it is faster than the original `date_histogram`. The `range` aggregation is *fairly* careful in how it rewrites, giving up on the `filters` aggregation if it won't collect "filter by filter" and falling back to its original execution mechanism. So an aggregation like this: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "date_histogram": { "field": "dropoff_datetime", "fixed_interval": "60d", "time_zone": "America/New_York" } } } } ``` is executed like: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "range": { "field": "dropoff_datetime", "ranges": [ {"from": 1415250000000, "to": 1420434000000}, {"from": 1420434000000, "to": 1425618000000}, {"from": 1425618000000, "to": 1430798400000}, {"from": 1430798400000, "to": 1435982400000}, {"from": 1435982400000, "to": 1441166400000}, {"from": 1441166400000, "to": 1446350400000}, {"from": 1446350400000, "to": 1451538000000}, {"from": 1451538000000} ] } } } } ``` Which in turn is executed like this: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "filters": { "filters": { "1": {"range": {"dropoff_datetime": {"gte": "2014-12-30 00:00:00", "lt": "2015-01-05 05:00:00"}}}, "2": {"range": {"dropoff_datetime": {"gte": "2015-01-05 05:00:00", "lt": "2015-03-06 05:00:00"}}}, "3": {"range": {"dropoff_datetime": {"gte": "2015-03-06 00:00:00", "lt": "2015-05-05 00:00:00"}}}, "4": {"range": {"dropoff_datetime": {"gte": "2015-05-05 00:00:00", "lt": "2015-07-04 00:00:00"}}}, "5": {"range": {"dropoff_datetime": {"gte": "2015-07-04 00:00:00", "lt": "2015-09-02 00:00:00"}}}, "6": {"range": {"dropoff_datetime": {"gte": "2015-09-02 00:00:00", "lt": "2015-11-01 00:00:00"}}}, "7": {"range": {"dropoff_datetime": {"gte": "2015-11-01 00:00:00", "lt": "2015-12-31 00:00:00"}}}, "8": {"range": {"dropoff_datetime": {"gte": "2015-12-31 00:00:00"}}} } } } } } ``` And *that* is faster because we can execute it "filter by filter". Finally, notice the `range` query filtering the data. That is required for the data set that I'm using for testing. The "filter by filter" collection mechanism for the `filters` agg needs special case handling when the query is a `range` query and the filter is a `range` query and they are both on the same field. That special case handling "merges" the range query. Without it "filter by filter" collection is substantially slower. Its still quite a bit quicker than the standard `filter` collection, but not nearly as fast as it could be.

This speeds up `date_histogram` aggregations without a parent or children. This is quite common - it's the aggregation that Kibana's Discover uses all over the place. Also, we hope to be able to use the same mechanism to speed aggs with children one day, but that day isn't today. The kind of speedup we're seeing is fairly substantial in many cases: ``` | | | before | after | | | 90th percentile service time | date_histogram_calendar_interval | 9266.07 | 1376.13 | ms | | 90th percentile service time | date_histogram_calendar_interval_with_tz | 9217.21 | 1372.67 | ms | | 90th percentile service time | date_histogram_fixed_interval | 8817.36 | 1312.67 | ms | | 90th percentile service time | date_histogram_fixed_interval_with_tz | 8801.71 | 1311.69 | ms | <-- discover's agg | 90th percentile service time | date_histogram_fixed_interval_with_metrics | 44660.2 | 43789.5 | ms | ``` This uses the work we did in elastic#61467 to precompute the rounding points for a `date_histogram`. Now, when we know the rounding points we execute the `date_histogram` as a `range` aggregation. This is nice for two reasons: 1. We can further rewrite the `range` aggregation (see below) 2. We don't need to allocate a hash to convert rounding points to ordinals. 3. We can send precise cardinality estimates to sub-aggs. Points 2 and 3 above are nice, but most of the speed difference comes from point 1. Specifically, we now look into executing `range` aggregations as a `filters` aggregation. Normally the `filters` aggregation is quite slow but when it doesn't have a parent or any children then we can execute it "filter by filter" which is significantly faster. So fast, in fact, that it is faster than the original `date_histogram`. The `range` aggregation is *fairly* careful in how it rewrites, giving up on the `filters` aggregation if it won't collect "filter by filter" and falling back to its original execution mechanism. So an aggregation like this: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "date_histogram": { "field": "dropoff_datetime", "fixed_interval": "60d", "time_zone": "America/New_York" } } } } ``` is executed like: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "range": { "field": "dropoff_datetime", "ranges": [ {"from": 1415250000000, "to": 1420434000000}, {"from": 1420434000000, "to": 1425618000000}, {"from": 1425618000000, "to": 1430798400000}, {"from": 1430798400000, "to": 1435982400000}, {"from": 1435982400000, "to": 1441166400000}, {"from": 1441166400000, "to": 1446350400000}, {"from": 1446350400000, "to": 1451538000000}, {"from": 1451538000000} ] } } } } ``` Which in turn is executed like this: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "filters": { "filters": { "1": {"range": {"dropoff_datetime": {"gte": "2014-12-30 00:00:00", "lt": "2015-01-05 05:00:00"}}}, "2": {"range": {"dropoff_datetime": {"gte": "2015-01-05 05:00:00", "lt": "2015-03-06 05:00:00"}}}, "3": {"range": {"dropoff_datetime": {"gte": "2015-03-06 00:00:00", "lt": "2015-05-05 00:00:00"}}}, "4": {"range": {"dropoff_datetime": {"gte": "2015-05-05 00:00:00", "lt": "2015-07-04 00:00:00"}}}, "5": {"range": {"dropoff_datetime": {"gte": "2015-07-04 00:00:00", "lt": "2015-09-02 00:00:00"}}}, "6": {"range": {"dropoff_datetime": {"gte": "2015-09-02 00:00:00", "lt": "2015-11-01 00:00:00"}}}, "7": {"range": {"dropoff_datetime": {"gte": "2015-11-01 00:00:00", "lt": "2015-12-31 00:00:00"}}}, "8": {"range": {"dropoff_datetime": {"gte": "2015-12-31 00:00:00"}}} } } } } } ``` And *that* is faster because we can execute it "filter by filter". Finally, notice the `range` query filtering the data. That is required for the data set that I'm using for testing. The "filter by filter" collection mechanism for the `filters` agg needs special case handling when the query is a `range` query and the filter is a `range` query and they are both on the same field. That special case handling "merges" the range query. Without it "filter by filter" collection is substantially slower. Its still quite a bit quicker than the standard `filter` collection, but not nearly as fast as it could be.

This speeds up `date_histogram` aggregations without a parent or children. This is quite common - it's the aggregation that Kibana's Discover uses all over the place. Also, we hope to be able to use the same mechanism to speed aggs with children one day, but that day isn't today. The kind of speedup we're seeing is fairly substantial in many cases: ``` | | | before | after | | | 90th percentile service time | date_histogram_calendar_interval | 9266.07 | 1376.13 | ms | | 90th percentile service time | date_histogram_calendar_interval_with_tz | 9217.21 | 1372.67 | ms | | 90th percentile service time | date_histogram_fixed_interval | 8817.36 | 1312.67 | ms | | 90th percentile service time | date_histogram_fixed_interval_with_tz | 8801.71 | 1311.69 | ms | <-- discover's agg | 90th percentile service time | date_histogram_fixed_interval_with_metrics | 44660.2 | 43789.5 | ms | ``` This uses the work we did in #61467 to precompute the rounding points for a `date_histogram`. Now, when we know the rounding points we execute the `date_histogram` as a `range` aggregation. This is nice for two reasons: 1. We can further rewrite the `range` aggregation (see below) 2. We don't need to allocate a hash to convert rounding points to ordinals. 3. We can send precise cardinality estimates to sub-aggs. Points 2 and 3 above are nice, but most of the speed difference comes from point 1. Specifically, we now look into executing `range` aggregations as a `filters` aggregation. Normally the `filters` aggregation is quite slow but when it doesn't have a parent or any children then we can execute it "filter by filter" which is significantly faster. So fast, in fact, that it is faster than the original `date_histogram`. The `range` aggregation is *fairly* careful in how it rewrites, giving up on the `filters` aggregation if it won't collect "filter by filter" and falling back to its original execution mechanism. So an aggregation like this: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "date_histogram": { "field": "dropoff_datetime", "fixed_interval": "60d", "time_zone": "America/New_York" } } } } ``` is executed like: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "range": { "field": "dropoff_datetime", "ranges": [ {"from": 1415250000000, "to": 1420434000000}, {"from": 1420434000000, "to": 1425618000000}, {"from": 1425618000000, "to": 1430798400000}, {"from": 1430798400000, "to": 1435982400000}, {"from": 1435982400000, "to": 1441166400000}, {"from": 1441166400000, "to": 1446350400000}, {"from": 1446350400000, "to": 1451538000000}, {"from": 1451538000000} ] } } } } ``` Which in turn is executed like this: ``` POST _search { "size": 0, "query": { "range": { "dropoff_datetime": { "gte": "2015-01-01 00:00:00", "lt": "2016-01-01 00:00:00" } } }, "aggs": { "dropoffs_over_time": { "filters": { "filters": { "1": {"range": {"dropoff_datetime": {"gte": "2014-12-30 00:00:00", "lt": "2015-01-05 05:00:00"}}}, "2": {"range": {"dropoff_datetime": {"gte": "2015-01-05 05:00:00", "lt": "2015-03-06 05:00:00"}}}, "3": {"range": {"dropoff_datetime": {"gte": "2015-03-06 00:00:00", "lt": "2015-05-05 00:00:00"}}}, "4": {"range": {"dropoff_datetime": {"gte": "2015-05-05 00:00:00", "lt": "2015-07-04 00:00:00"}}}, "5": {"range": {"dropoff_datetime": {"gte": "2015-07-04 00:00:00", "lt": "2015-09-02 00:00:00"}}}, "6": {"range": {"dropoff_datetime": {"gte": "2015-09-02 00:00:00", "lt": "2015-11-01 00:00:00"}}}, "7": {"range": {"dropoff_datetime": {"gte": "2015-11-01 00:00:00", "lt": "2015-12-31 00:00:00"}}}, "8": {"range": {"dropoff_datetime": {"gte": "2015-12-31 00:00:00"}}} } } } } } ``` And *that* is faster because we can execute it "filter by filter". Finally, notice the `range` query filtering the data. That is required for the data set that I'm using for testing. The "filter by filter" collection mechanism for the `filters` agg needs special case handling when the query is a `range` query and the filter is a `range` query and they are both on the same field. That special case handling "merges" the range query. Without it "filter by filter" collection is substantially slower. Its still quite a bit quicker than the standard `filter` collection, but not nearly as fast as it could be.

nik9000 added >enhancement :Analytics/Aggregations Aggregations v8.0.0 v7.10.0 labels Aug 24, 2020

nik9000 requested review from jpountz, not-napoleon and polyfractal August 24, 2020 13:02

elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Aug 24, 2020

nik9000 force-pushed the binary_search_for_rounding branch from a817685 to 7687e30 Compare August 24, 2020 13:03

nik9000 requested a review from imotov August 24, 2020 13:29

jpountz reviewed Aug 24, 2020

View reviewed changes

nik9000 added 2 commits August 24, 2020 09:44

iter

e68463d

Another test

c13740b

imotov reviewed Aug 26, 2020

View reviewed changes

nik9000 added 2 commits August 27, 2020 09:04

Merge branch 'master' into binary_search_for_rounding

4162ef1

Implement new method

02f0fe3

nik9000 added 2 commits August 31, 2020 10:47

Skip method for bad times

4fbcfbf

Merge branch 'master' into binary_search_for_rounding

7a7ef49

not-napoleon approved these changes Sep 3, 2020

View reviewed changes

nik9000 requested a review from jpountz September 14, 2020 13:22

nik9000 added 6 commits September 14, 2020 09:25

Merge branch 'master' into binary_search_for_rounding

4f23de0

comments

178a158

Detect!

f201a25

Test update

0e93729

Comment on number

35ef1c4

Only 12+

f1e286e

nik9000 commented Sep 15, 2020

View reviewed changes

jpountz reviewed Sep 23, 2020

View reviewed changes

nik9000 added 2 commits September 24, 2020 08:35

Merge branch 'master' into binary_search_for_rounding

d2f1d23

Nano of day!

2281d35

jpountz approved these changes Sep 24, 2020

View reviewed changes

nik9000 merged commit d9acac2 into elastic:master Sep 24, 2020

nik9000 added the backport pending label Sep 24, 2020

nik9000 removed the backport pending label Sep 24, 2020

nik9000 mentioned this pull request Oct 13, 2020

Speed up date_histogram without children #63643

Merged

nik9000 mentioned this pull request Nov 9, 2020

Speed up date_histogram without children (backport of #63643) #64823

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Speed up date_histogram by precomputing ranges #61467

Speed up date_histogram by precomputing ranges #61467

Conversation

nik9000 commented Aug 24, 2020 • edited Loading

elasticmachine commented Aug 24, 2020

nik9000 commented Aug 24, 2020

jpountz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 Aug 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imotov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented Aug 27, 2020 via email

not-napoleon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented Sep 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nik9000 commented Sep 24, 2020

nik9000 commented Sep 24, 2020

nik9000 commented Sep 27, 2020

nik9000 commented Aug 24, 2020 •

edited

Loading

nik9000 Aug 24, 2020 •

edited

Loading