Adding reindex data stream rest action #118109

masseyke · 2024-12-05T21:11:43Z

This adds the rest endpoint that will be used to reindex a data stream for upgrade. An example request will look like:

POST /_migration/reindex
{
  "mode": "upgrade",
  "source": {
    "index": "my-data-stream"
  }
}

with the response immediately returning the persistent task id, like:

{
  "task" : "reindex-data-stream-my-data-stream"
}

This builds on the transport action and persistent task from #117927. The APIs to query the status of this task and to cancel this task are coming in future PRs.

dakrone

LGTM, I left some minor comments, but they're all nothing major.

dakrone · 2024-12-05T23:13:39Z

...in/migrate/src/main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamAction.java

@@ -70,22 +81,55 @@ public boolean equals(Object other) {

    }

-    public static class ReindexDataStreamRequest extends ActionRequest {
+    public static class ReindexDataStreamRequest extends ActionRequest implements IndicesRequest, ToXContent {
+        private final String mode;


Can we make the mode an enum for this?

I did that and then undid it because I was worried about deserialization failures when we add modes in the future. But maybe failing fast and/or dealing with that with a transport version check is the way to go.

dakrone · 2024-12-05T23:15:18Z

...in/migrate/src/main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamAction.java

+            });
+
+        private static final ConstructingObjectParser<String, Void> SOURCE_PARSER = new ConstructingObjectParser<>(
+            "source",


We should pull this (and the ParseField("index") and "mode" below) into public static variables so that they are accessible from other classes. It's minor, but it will make it easier to refactor if we decide to change the fields later.

Thanks for pulling it into fields, this one can change to

Suggested change

"source",

SOURCE_FIELD.getPreferredName(),

I think

I actually think that's just a label for the parser and it doesn't matter what it is. But it can't hurt to set it to SOURCE_FIELD.getPreferredName(), so I'll do that.

dakrone · 2024-12-05T23:15:41Z

...in/migrate/src/main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamAction.java

+            builder.field("mode", mode);
+            builder.startObject("source");
+            builder.field("index", sourceDataStream);


Same here for using the ParseFields

dakrone · 2024-12-05T23:16:28Z

...e/src/main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamTransportAction.java

+        if (request.getMode().equals("upgrade") == false) {
+            listener.onFailure(new IllegalArgumentException("Only mode 'upgrade' is supported"));
+            return;
+        }


If we make it an enum, we can move this into the request parsing, so that it doesn't go over the transport before being rejected

dakrone

Still LGTM, I left one more comment

dakrone · 2024-12-06T20:18:29Z

...in/migrate/src/main/java/org/elasticsearch/xpack/migrate/action/ReindexDataStreamAction.java

+            });
+
+        private static final ConstructingObjectParser<String, Void> SOURCE_PARSER = new ConstructingObjectParser<>(
+            "source",


Thanks for pulling it into fields, this one can change to

Suggested change

"source",

SOURCE_FIELD.getPreferredName(),

I think

elasticsearchmachine · 2024-12-08T23:57:24Z

💚 Backport successful

Status	Branch	Result
✅	8.x

* Adding a _migration/reindex endpoint * Adding rest api spec and test * Adding a feature flag for reindex data streams * updating json spec * fixing a typo * Changing mode to an enum * Moving ParseFields into public static finals * Commenting out test that leaves task running, until we add a cancel API * Removing persistent task id from output * replacing a string with a variable

This adds a new transport action to get the status of a migration reindex (started via the API at #118109), and a new rest action to use it. The rest action accepts the data stream or index name, and returns the status. For example if the reindex task exists for data stream `my-data-stream`: ``` GET /_migration/reindex/my-data-stream/_status?pretty ``` returns ``` { "start_time" : 1733519098570, "complete" : true, "total_indices" : 1, "total_indices_requiring_upgrade" : 0, "successes" : 0, "in_progress" : 0, "pending" : 0, "errors" : [ ] } ``` If a reindex task does not exist: ``` GET _migration/reindex/my-data-stream/_status?pretty ``` Then a 404 is returned: ``` { "error" : { "root_cause" : [ { "type" : "resource_not_found_exception", "reason" : "No migration reindex status found for [my-data-stream]" } ], "type" : "resource_not_found_exception", "reason" : "No migration reindex status found for [my-data-stream]" }, "status" : 404 } ```

This adds a new transport action to get the status of a migration reindex (started via the API at elastic#118109), and a new rest action to use it. The rest action accepts the data stream or index name, and returns the status. For example if the reindex task exists for data stream `my-data-stream`: ``` GET /_migration/reindex/my-data-stream/_status?pretty ``` returns ``` { "start_time" : 1733519098570, "complete" : true, "total_indices" : 1, "total_indices_requiring_upgrade" : 0, "successes" : 0, "in_progress" : 0, "pending" : 0, "errors" : [ ] } ``` If a reindex task does not exist: ``` GET _migration/reindex/my-data-stream/_status?pretty ``` Then a 404 is returned: ``` { "error" : { "root_cause" : [ { "type" : "resource_not_found_exception", "reason" : "No migration reindex status found for [my-data-stream]" } ], "type" : "resource_not_found_exception", "reason" : "No migration reindex status found for [my-data-stream]" }, "status" : 404 } ```

* Adding get migration reindex status (#118267) This adds a new transport action to get the status of a migration reindex (started via the API at #118109), and a new rest action to use it. The rest action accepts the data stream or index name, and returns the status. For example if the reindex task exists for data stream `my-data-stream`: ``` GET /_migration/reindex/my-data-stream/_status?pretty ``` returns ``` { "start_time" : 1733519098570, "complete" : true, "total_indices" : 1, "total_indices_requiring_upgrade" : 0, "successes" : 0, "in_progress" : 0, "pending" : 0, "errors" : [ ] } ``` If a reindex task does not exist: ``` GET _migration/reindex/my-data-stream/_status?pretty ``` Then a 404 is returned: ``` { "error" : { "root_cause" : [ { "type" : "resource_not_found_exception", "reason" : "No migration reindex status found for [my-data-stream]" } ], "type" : "resource_not_found_exception", "reason" : "No migration reindex status found for [my-data-stream]" }, "status" : 404 } ``` * adding migration reindex actions to OperatorPrivilegesIT

This introduces the migration reindex cancel API, which cancels a migration reindex task for a given data stream name that was started with #118109. For example: ``` POST localhost:9200/_migration/reindex/my-data-stream/_cancel?pretty ``` returns ``` { "acknowledged" : true } ``` This cancels the task, and cancels any ongoing reindexing of backing indices, but does not do any cleanup.

This introduces the migration reindex cancel API, which cancels a migration reindex task for a given data stream name that was started with elastic#118109. For example: ``` POST localhost:9200/_migration/reindex/my-data-stream/_cancel?pretty ``` returns ``` { "acknowledged" : true } ``` This cancels the task, and cancels any ongoing reindexing of backing indices, but does not do any cleanup.

This introduces the migration reindex cancel API, which cancels a migration reindex task for a given data stream name that was started with #118109. For example: ``` POST localhost:9200/_migration/reindex/my-data-stream/_cancel?pretty ``` returns ``` { "acknowledged" : true } ``` This cancels the task, and cancels any ongoing reindexing of backing indices, but does not do any cleanup.

This introduces the migration reindex cancel API, which cancels a migration reindex task for a given data stream name that was started with elastic#118109. For example: ``` POST localhost:9200/_migration/reindex/my-data-stream/_cancel?pretty ``` returns ``` { "acknowledged" : true } ``` This cancels the task, and cancels any ongoing reindexing of backing indices, but does not do any cleanup.

masseyke added 4 commits December 4, 2024 16:20

Adding a _migration/reindex endpoint

e727a43

Merge branch 'main' into adding-reindex-data-stream-rest-action

91dc605

Adding rest api spec and test

e3cb722

Adding a feature flag for reindex data streams

ddecd24

masseyke added >enhancement backport :Data Management/Data streams Data streams and their lifecycles v9.0.0 v8.18.0 labels Dec 5, 2024

masseyke added 3 commits December 5, 2024 15:18

updating json spec

7e2124c

fixing a typo

fdba8fb

Merge branch 'main' into adding-reindex-data-stream-rest-action

151bac5

masseyke marked this pull request as ready for review December 5, 2024 21:44

masseyke requested a review from dakrone December 5, 2024 21:44

dakrone approved these changes Dec 5, 2024

View reviewed changes

masseyke added 8 commits December 5, 2024 17:54

Changing mode to an enum

9af08c9

Moving ParseFields into public static finals

21a3be4

Merge branch 'main' into adding-reindex-data-stream-rest-action

fd0bef9

Merge branch 'main' into adding-reindex-data-stream-rest-action

5404783

Merge branch 'main' into adding-reindex-data-stream-rest-action

d32096b

Merge branch 'main' into adding-reindex-data-stream-rest-action

0164919

Commenting out test that leaves task running, until we add a cancel API

245a564

Removing persistent task id from output

09b6e4f

dakrone approved these changes Dec 6, 2024

View reviewed changes

masseyke added 2 commits December 6, 2024 16:25

replacing a string with a variable

b3524ad

Merge branch 'main' into adding-reindex-data-stream-rest-action

873fdba

masseyke added auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) auto-backport Automatically create backport pull requests when merged labels Dec 6, 2024

Merge branch 'main' into adding-reindex-data-stream-rest-action

8aeb5c8

elasticsearchmachine merged commit 8107cc9 into elastic:main Dec 8, 2024
16 checks passed

masseyke deleted the adding-reindex-data-stream-rest-action branch December 8, 2024 23:56

masseyke mentioned this pull request Dec 8, 2024

[8.x] Adding reindex data stream rest action (#118109) #118228

Merged

This was referenced Dec 9, 2024

Adding a migration reindex cancel API #118291

Merged

Adding get migration reindex status #118267

Merged

masseyke mentioned this pull request Dec 13, 2024

Adding a migration reindex cancel API (#118291) #118679

Merged

masseyke removed the backport label Dec 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding reindex data stream rest action #118109

Adding reindex data stream rest action #118109

masseyke commented Dec 5, 2024 •

edited

Loading

dakrone left a comment

dakrone Dec 5, 2024

masseyke Dec 5, 2024

dakrone Dec 5, 2024

dakrone Dec 6, 2024

masseyke Dec 6, 2024

dakrone Dec 5, 2024

dakrone Dec 5, 2024

dakrone left a comment

dakrone Dec 6, 2024

elasticsearchmachine commented Dec 8, 2024

Adding reindex data stream rest action #118109

Adding reindex data stream rest action #118109

Conversation

masseyke commented Dec 5, 2024 • edited Loading

dakrone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakrone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticsearchmachine commented Dec 8, 2024

💚 Backport successful

masseyke commented Dec 5, 2024 •

edited

Loading