Total data set size in stats #70625

henningandersen · 2021-03-22T10:54:09Z

With shared cache searchable snapshots we have shards that have a size
in S3 that differs from the locally occupied disk space. This commit
introduces store.total_data_set_size to node and indices stats, allowing to
differ between the two.

Relates #69820

With shared cache searchable snapshots we have shards that have a size in S3 that differs from the locally occupied disk space. This commit introduces `store.local_size` to node and indices stats, allowing to differ between the two. Relates elastic#69820

elasticmachine · 2021-03-22T10:54:12Z

Pinging @elastic/es-distributed (Team:Distributed)

elasticmachine · 2021-03-22T10:54:12Z

Pinging @elastic/es-core-features (Team:Core/Features)

…ize_to_stats

henningandersen · 2021-03-22T16:44:32Z

...ava/org/elasticsearch/xpack/monitoring/collector/cluster/ClusterStatsMonitoringDocTests.java

@@ -422,6 +422,7 @@ public void testToXContent() throws IOException {
                + "      },"
                + "      \"store\": {"
                + "        \"size_in_bytes\": 0,"
+                + "        \"local_size_in_bytes\": 0,"


Since this is mapped as a dynamic=false object, I chose the easy path here, but it is slightly inconsistent with index and indices stats monitoring docs.

I don't follow this - what is the mapping you mean?

The mapping is in monitoring-es.json here:

elasticsearch/x-pack/plugin/core/src/main/resources/monitoring-es.json

Line 482 in 0b567d6

"type": "object"

DaveCTurner

Looks good, with one comment on the tests.

Should we change the InternalClusterInfoService to use the local size of shards in this change too?

DaveCTurner · 2021-03-23T10:03:10Z

...sterTest/java/org/elasticsearch/xpack/searchablesnapshots/SearchableSnapshotsIntegTests.java

+            final long expectedSize = snapshotShards.get(shardStats.getShardRouting().getId()).getStats().getTotalSize();
+            assertThat(shardStats.getShardRouting().toString(), store.getSize().getBytes(), greaterThanOrEqualTo(expectedSize));
+            // we expect the new segments file to make it significantly less than 2x.
+            assertThat(shardStats.getShardRouting().toString(), store.getSize().getBytes(), lessThan(expectedSize * 2));


This seems like it might be flaky - why *2? The index could in theory have one tiny doc per segment, are we sure that each segment will always be larger than the per-segment entry in the segments_N file?

If you're sure this is ok, can we explain the reasoning in a more detailed comment?

The 2x is there because there is one segments_N file already and the bootstrap/translog association adds another segments_N file with only changes being the two UUIDs that are now different. AFAICS, that will always lead to a new segments_N file of identical size. Thus a worst case is 2x. I added a comment to clarify this in the code too.

Ah good point I had forgotten about the older segments_N file. Yes this looks solid now.

…ize_to_stats

henningandersen · 2021-03-24T11:13:28Z

After receiving feedback on this outside this PR, I have left size as the "local, on-disk size" metric and added a new total_data_set_size metric instead, representing the full size of shared cache searchable snapshots (as well as the size of all regular shards).

This is ready for another review round.

DaveCTurner

LGTM

…ize_to_stats

henningandersen · 2021-03-30T12:22:06Z

Build failure reported in #66495.

@elasticmachine run elasticsearch-ci/2

With shared cache searchable snapshots we have shards that have a size in S3 that differs from the locally occupied disk space. This commit introduces `store.total_data_set_size` to node and indices stats, allowing to differ between the two. Relates elastic#69820

With shared cache searchable snapshots we have shards that have a size in S3 that differs from the locally occupied disk space. This commit introduces `store.total_data_set_size` to node and indices stats, allowing to differ between the two. Relates #69820

StoreStats serialization was changed in #70625.

Reenable bwc and serialize new field `totalDataSetSizeInBytes` to 7.13+ now that the backport of #70625 is done.

…shotsIntegTests (#73243) In #70625 we added the total data set size of shards to the Indices Stats API and we enhanced the test testCreateAndRestorePartialSearchableSnapshot to also verify the correctness of this data set size. Because restoring a searchable snapshot shard creates a new in-memory segment size, the verification of the data set size was implemented in an approximative fashion: between the expected size and twice the expected size. This approximation sometimes fails for shards that have no documents indexed (see #73194). This commit changes the test so that it now verifies the exact data set size returned by the Indices Stats API, which should be the sum of the original expected size of the snapshotted size + the length of the extra segment file in memory. Closes #73194

…shotsIntegTests (elastic#73243) In elastic#70625 we added the total data set size of shards to the Indices Stats API and we enhanced the test testCreateAndRestorePartialSearchableSnapshot to also verify the correctness of this data set size. Because restoring a searchable snapshot shard creates a new in-memory segment size, the verification of the data set size was implemented in an approximative fashion: between the expected size and twice the expected size. This approximation sometimes fails for shards that have no documents indexed (see elastic#73194). This commit changes the test so that it now verifies the exact data set size returned by the Indices Stats API, which should be the sum of the original expected size of the snapshotted size + the length of the extra segment file in memory. Closes elastic#73194

…shotsIntegTests (#73243) (#73455) In #70625 we added the total data set size of shards to the Indices Stats API and we enhanced the test testCreateAndRestorePartialSearchableSnapshot to also verify the correctness of this data set size. Because restoring a searchable snapshot shard creates a new in-memory segment size, the verification of the data set size was implemented in an approximative fashion: between the expected size and twice the expected size. This approximation sometimes fails for shards that have no documents indexed (see #73194). This commit changes the test so that it now verifies the exact data set size returned by the Indices Stats API, which should be the sum of the original expected size of the snapshotted size + the length of the extra segment file in memory. Closes #73194

…shotsIntegTests (#73243) (#73454) In #70625 we added the total data set size of shards to the Indices Stats API and we enhanced the test testCreateAndRestorePartialSearchableSnapshot to also verify the correctness of this data set size. Because restoring a searchable snapshot shard creates a new in-memory segment size, the verification of the data set size was implemented in an approximative fashion: between the expected size and twice the expected size. This approximation sometimes fails for shards that have no documents indexed (see #73194). This commit changes the test so that it now verifies the exact data set size returned by the Indices Stats API, which should be the sum of the original expected size of the snapshotted size + the length of the extra segment file in memory. Closes #73194

The telemetry for data tiers was using the size in bytes, however, for the frozen tier using searchable snapshots, this was the disk usage rather than the size of the actual data. This commit changes the telemetry to use `total_data_set_size` as introduced in elastic#70625 so that the telemetry is correct. Resolves elastic#86055

The telemetry for data tiers was using the size in bytes, however, for the frozen tier using searchable snapshots, this was the disk usage rather than the size of the actual data. This commit changes the telemetry to use `total_data_set_size` as introduced in #70625 so that the telemetry is correct. Resolves #86055

…c#86580) The telemetry for data tiers was using the size in bytes, however, for the frozen tier using searchable snapshots, this was the disk usage rather than the size of the actual data. This commit changes the telemetry to use `total_data_set_size` as introduced in elastic#70625 so that the telemetry is correct. Resolves elastic#86055

#86749) The telemetry for data tiers was using the size in bytes, however, for the frozen tier using searchable snapshots, this was the disk usage rather than the size of the actual data. This commit changes the telemetry to use `total_data_set_size` as introduced in #70625 so that the telemetry is correct. Resolves #86055

#86748) The telemetry for data tiers was using the size in bytes, however, for the frozen tier using searchable snapshots, this was the disk usage rather than the size of the actual data. This commit changes the telemetry to use `total_data_set_size` as introduced in #70625 so that the telemetry is correct. Resolves #86055

Local size in stats

37bad9a

With shared cache searchable snapshots we have shards that have a size in S3 that differs from the locally occupied disk space. This commit introduces `store.local_size` to node and indices stats, allowing to differ between the two. Relates elastic#69820

henningandersen added >enhancement :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs :Data Management/Stats Statistics tracking and retrieval APIs v8.0.0 v7.13.0 labels Mar 22, 2021

elasticmachine added Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Data Management Meta label for data/management team labels Mar 22, 2021

henningandersen added 4 commits March 22, 2021 12:34

Fix monitoring cluster stats test.

8872133

Merge remote-tracking branch 'origin/master' into enhance_add_local_s…

39c1ede

…ize_to_stats

Fix bwc of test

b8137a8

Merge remote-tracking branch 'origin/master' into enhance_add_local_s…

a1c1503

…ize_to_stats

henningandersen mentioned this pull request Mar 22, 2021

[CI] SqlSearchIT.testAllTypesWithRequestToOldNodes #70630

Closed

henningandersen requested a review from DaveCTurner March 22, 2021 14:04

henningandersen commented Mar 22, 2021

View reviewed changes

DaveCTurner reviewed Mar 23, 2021

View reviewed changes

henningandersen added 2 commits March 24, 2021 10:38

total_data_set_size rather than local_size

cba2965

Merge remote-tracking branch 'origin/master' into enhance_add_local_s…

c7552f0

…ize_to_stats

henningandersen changed the title ~~Local size in stats~~ Total data set size in stats Mar 24, 2021

henningandersen requested a review from DaveCTurner March 24, 2021 11:14

DaveCTurner approved these changes Mar 30, 2021

View reviewed changes

Merge remote-tracking branch 'origin/master' into enhance_add_local_s…

775144a

…ize_to_stats

henningandersen merged commit 0f28e97 into elastic:master Mar 30, 2021

henningandersen added the backport pending label Mar 30, 2021

henningandersen mentioned this pull request Mar 30, 2021

Total data set size in stats (#70625) #71057

Merged

henningandersen removed the backport pending label Mar 31, 2021

henningandersen added a commit that referenced this pull request Mar 31, 2021

Disable bwc due to StoreStats

06526a4

StoreStats serialization was changed in #70625.

henningandersen added a commit that referenced this pull request Mar 31, 2021

StoreStats update serialization version (#71107)

632d23d

Reenable bwc and serialize new field `totalDataSetSizeInBytes` to 7.13+ now that the backport of #70625 is done.

ywelsch mentioned this pull request Apr 1, 2021

[CI] SearchableSnapshotsIntegTests testCreateAndRestorePartialSearchableSnapshot failing #71132

Closed

stevejgordon mentioned this pull request Apr 21, 2021

7.13.0 Meta Ticket elastic/elasticsearch-net#5584

Closed

62 tasks

tlrx mentioned this pull request May 19, 2021

More precise total data set size verification in FrozenSearchableSnapshotsIntegTests #73243

Merged

tlrx mentioned this pull request May 27, 2021

[7.x] More precise total data set size verification in FrozenSearchableSnapshotsIntegTests #73454

Merged

tlrx mentioned this pull request May 27, 2021

[7.13] More precise total data set size verification in FrozenSearchableSnapshotsIntegTests #73455

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

dakrone mentioned this pull request May 9, 2022

Correctly calculate disk usage for frozen data tier telemetry #86580

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Total data set size in stats #70625

Total data set size in stats #70625

henningandersen commented Mar 22, 2021 •

edited

Loading

elasticmachine commented Mar 22, 2021

elasticmachine commented Mar 22, 2021

henningandersen Mar 22, 2021

DaveCTurner Mar 23, 2021

henningandersen Mar 24, 2021

DaveCTurner left a comment

DaveCTurner Mar 23, 2021

henningandersen Mar 24, 2021

DaveCTurner Mar 30, 2021

henningandersen commented Mar 24, 2021

DaveCTurner left a comment

henningandersen commented Mar 30, 2021

Total data set size in stats #70625

Total data set size in stats #70625

Conversation

henningandersen commented Mar 22, 2021 • edited Loading

elasticmachine commented Mar 22, 2021

elasticmachine commented Mar 22, 2021

henningandersen Mar 22, 2021

Choose a reason for hiding this comment

DaveCTurner Mar 23, 2021

Choose a reason for hiding this comment

henningandersen Mar 24, 2021

Choose a reason for hiding this comment

DaveCTurner left a comment

Choose a reason for hiding this comment

DaveCTurner Mar 23, 2021

Choose a reason for hiding this comment

henningandersen Mar 24, 2021

Choose a reason for hiding this comment

DaveCTurner Mar 30, 2021

Choose a reason for hiding this comment

henningandersen commented Mar 24, 2021

DaveCTurner left a comment

Choose a reason for hiding this comment

henningandersen commented Mar 30, 2021

henningandersen commented Mar 22, 2021 •

edited

Loading