[Profiling] Support index migrations

[danielmitterdorfer]

With this commit we add the required infrastructure to detect whether indices need to be migrated and define migrations to update mappings or dynamic index settings.

[elasticsearchmachine]

Pinging @elastic/profiling (Team:Universal Profiling)

[elasticsearchmachine]

Hi @danielmitterdorfer, I've created a changelog YAML for you.

[danielmitterdorfer]

Testing this in action is a bit tricky because index and index template versions are defined in code. I've tested this as follows:

1. Start Elasticsearch locally (note how we preserve data): ./gradlew :run -Drun.license_type=trial -Dtests.heap.size=4G -Dtests.es.xpack.profiling.templates.enabled=true --https --preserve-data.
2. Enable debug or trace logging (depending on what you'd like to see, TRACE is a bit noisy):

curl -k -u elastic-admin:elastic-password -X PUT "https://localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "logger.org.elasticsearch.xpack.profiler": "DEBUG"
  }
}
'

3. Stop the cluster and apply the following modifications:

diff --git a/x-pack/plugin/profiler/src/main/java/org/elasticsearch/xpack/profiler/ProfilingIndexManager.java b/x-pack/plugin/profiler/src/main/java/org/elasticsearch/xpack/profiler/ProfilingIndexManager.java
index 6c649836877..bfa79847cba 100644
--- a/x-pack/plugin/profiler/src/main/java/org/elasticsearch/xpack/profiler/ProfilingIndexManager.java
+++ b/x-pack/plugin/profiler/src/main/java/org/elasticsearch/xpack/profiler/ProfilingIndexManager.java
@@ -19,8 +19,10 @@ import org.elasticsearch.client.internal.Client;
 import org.elasticsearch.cluster.ClusterState;
 import org.elasticsearch.cluster.metadata.IndexMetadata;
 import org.elasticsearch.cluster.service.ClusterService;
+import org.elasticsearch.common.settings.Settings;
 import org.elasticsearch.common.xcontent.LoggingDeprecationHandler;
 import org.elasticsearch.core.TimeValue;
+import org.elasticsearch.index.IndexSettings;
 import org.elasticsearch.threadpool.ThreadPool;
 import org.elasticsearch.xpack.core.ClientHelper;
 
@@ -45,7 +47,13 @@ public class ProfilingIndexManager extends AbstractProfilingPersistenceManager<P
         ProfilingIndex.regular(
             "profiling-returnpads-private",
             ProfilingIndexTemplateRegistry.PROFILING_RETURNPADS_PRIVATE_VERSION,
-            OnVersionBump.KEEP_OLD
+            OnVersionBump.KEEP_OLD,
+            new Migration.Builder().migrateToIndexTemplateVersion(2)
+                .addProperty("new-property", "keyword")
+                .migrateToIndexTemplateVersion(3)
+                .dynamicSettings(
+                    Settings.builder().put(IndexSettings.INDEX_REFRESH_INTERVAL_SETTING.getKey(), TimeValue.timeValueSeconds(30)).build()
+                )
         ),
         ProfilingIndex.regular(
             "profiling-sq-executables",
diff --git a/x-pack/plugin/profiler/src/main/java/org/elasticsearch/xpack/profiler/ProfilingIndexTemplateRegistry.java b/x-pack/plugin/profiler/src/main/java/org/elasticsearch/xpack/profiler/ProfilingIndexTemplateRegistry.java
index f2b9d0cedd9..69490f347ef 100644
--- a/x-pack/plugin/profiler/src/main/java/org/elasticsearch/xpack/profiler/ProfilingIndexTemplateRegistry.java
+++ b/x-pack/plugin/profiler/src/main/java/org/elasticsearch/xpack/profiler/ProfilingIndexTemplateRegistry.java
@@ -40,7 +40,7 @@ public class ProfilingIndexTemplateRegistry extends IndexTemplateRegistry {
     private static final Logger logger = LogManager.getLogger(ProfilingIndexTemplateRegistry.class);
     // history (please add a comment why you increased the version here)
     // version 1: initial
-    public static final int INDEX_TEMPLATE_VERSION = 1;
+    public static final int INDEX_TEMPLATE_VERSION = 3;
 
     // history for individual indices / index templates. Only bump these for breaking changes that require to create a new index
     public static final int PROFILING_EVENTS_VERSION = 1;

4. Start the cluster again: ./gradlew :run -Drun.license_type=trial -Dtests.heap.size=4G -Dtests.es.xpack.profiling.templates.enabled=true --https --preserve-data
5. Observe in the logs that the migrations happen, e.g.:

[2023-07-19T08:23:07,760][DEBUG][o.e.x.p.ProfilingIndexManager] [runTask-0] Applying migration [put mapping to target version [2]] for [.profiling-returnpads-private-v001].
[2023-07-19T08:23:07,762][DEBUG][o.e.x.p.ProfilingIndexManager] [runTask-0] Applying migration [update settings to target version [3]] for [.profiling-returnpads-private-v001].
[2023-07-19T08:23:07,762][DEBUG][o.e.x.p.ProfilingIndexManager] [runTask-0] Applying migration [put mapping to target version [3]] for [.profiling-returnpads-private-v001].
[2023-07-19T08:23:07,850][INFO ][o.e.c.m.MetadataMappingService] [runTask-0] [.profiling-returnpads-private-v001/P5g1p981TY60rLRSpeeIOw] update_mapping [_doc]
[2023-07-19T08:23:07,852][INFO ][o.e.c.m.MetadataMappingService] [runTask-0] [.profiling-returnpads-private-v001/P5g1p981TY60rLRSpeeIOw] update_mapping [_doc]

Note: The second PUT mapping call is injected by the migration infrastructure to ensure that _meta is updated to the most recent index template version.

To check mappings and settings, use:

curl -k -u elastic-admin:elastic-password "https://localhost:9200/.profiling-returnpads-private-v001/_mapping?pretty"
curl -k -u elastic-admin:elastic-password "https://localhost:9200/.profiling-returnpads-private-v001/_settings?pretty"

[rockdaboot]

I can see the index-template-version set to 3 in the mappings.
But I am not sure about the output of the settings:

{
  ".profiling-returnpads-private-v001" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "refresh_interval" : "30s",
        "hidden" : "true",
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".profiling-returnpads-private-v001",
        "creation_date" : "1688550338149",
        "number_of_replicas" : "0",
        "uuid" : "6ms8nJaPSxaq4ydWz70CdA",
        "version" : {
          "created" : "8100099"
        }
      }
    }
  }
}

Shouldn't there be a .profiling-returnpads-private-v003 as well ? It isn't here.

[danielmitterdorfer]

Shouldn't there be a .profiling-returnpads-private-v003 as well ? It isn't here.

No, this is exactly working as intended. Only the index template version is bumped but not the index version. This helps us upgrade indices gradually without too much disruption in the cluster unless it is truly required (e.g. changing a static index setting requires a version bump, but not adding a property)

[rockdaboot]

Just for my curiosity, why not have a constructor taking the version to avoid the checkVersionSet() function (and also migrateToIndexTemplateVersion()?

[danielmitterdorfer]

We could probably do that for the initial version but we allow for migrations across multiple versions and thus we need to be able to call it multiple times. But there's definitely multiple ways to approach this. :)

[rockdaboot]

LGTM