-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce cross-cluster replication API docs #34726
Conversation
This commit is our first introduction to cross-cluster replication docs. In this commit, we introduce the cross-cluster replication API docs. We also add skelton docs for additional content that will be added in a series of follow-up commits.
Pinging @elastic/es-distributed |
This PR is the first in a series of PRs that will introduce docs for CCR. We breakdown the introduction of these docs into a series of PRs to ease reviewing. I am aware that there is work outstanding that will render some of these docs broken (that is, we are actively making breaking API changes). I want that work to go in first, and then I will keep this PR up-to-date with those changes. However, I want reviewing to get started, and I want to start opening up the follow-up PRs. |
For reviewers not use to the workflow of working with the docs, here is the simplest way to read these docs:
Then you can open the resulting index HTML page, navigate to X-Pack APIs, and then navigate to cross-cluster replication (e.g., on macOS you could do If you already have the |
@@ -149,7 +149,6 @@ buildRestTests.setups['host'] = ''' | |||
- do: | |||
nodes.info: | |||
metric: [ http, transport ] | |||
- is_true: nodes.$master.http.publish_address |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is removed because it is now used in the setup section of some of the docs tests, and setup
sections do not support is_true
sections. While we could add support for is_true
sections to setup
section, this assertion is adding little value to begin with.
* master: (24 commits) ingest: better support for conditionals with simulate?verbose (elastic#34155) [Rollup] Job deletion should be invoked on the allocated task (elastic#34574) [DOCS] .Security index is never auto created (elastic#34589) CCR: Requires soft-deletes on the follower (elastic#34725) re-enable bwc tests (elastic#34743) Empty GetAliases authorization fix (elastic#34444) INGEST: Document Processor Conditional (elastic#33388) [CCR] Add total fetch time leader stat (elastic#34577) SQL: Support pattern against compatible indices (elastic#34718) [CCR] Auto follow pattern APIs adjustments (elastic#34518) [Test] Remove dead code from ExceptionSerializationTests (elastic#34713) A small typo in migration-assistance doc (elastic#34704) ingest: processor stats (elastic#34724) SQL: Implement IN(value1, value2, ...) expression. (elastic#34581) Tests: Add checks to GeoDistanceQueryBuilderTests (elastic#34273) INGEST: Rename Pipeline Processor Param. (elastic#34733) Core: Move IndexNameExpressionResolver to java time (elastic#34507) [DOCS] Force Merge: clarify execution and storage requirements (elastic#33882) TESTING.asciidoc fix examples using forbidden annotation (elastic#34515) SQL: Implement `CONVERT`, an alternative to `CAST` (elastic#34660) ...
@jasontedor I took a quick look and one thing I don't see in the APIs is an "Authorization" section, which lists what roles or privileges are required to run the API when security is enabled. Is this something you plan to add? I noticed a few other sections missing, so I'll submit suggestions for your perusal. |
// CONSOLE | ||
// TEST[s/<follower_index>/follower_index/] | ||
|
||
Here, `<follower_index>` is the name of the follower index to pause. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here, `<follower_index>` is the name of the follower index to pause. | |
==== Path Parameters | |
`follower_id` (required):: | |
(string) Identifier for the follower index. | |
Here, the auto-follow patterns associated with the | ||
<<modules-remote-clusters,remote cluster>> specified by `<leader_cluster>` | ||
will be deleted. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
==== Path Parameters | |
`auto_follow_pattern_name` (required):: | |
(string) Specifies auto-follow patterns that you want to delete. | |
|
||
Here, the named auto-follow pattern collection specified by | ||
`<auto_follow_pattern_name>` will be returned. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
==== Path Parameters | |
`auto_follow_pattern_name` :: | |
(string) Specifies the auto-follow patterns that you want to retrieve. If you do not specify any patterns, the API returns information for all patterns. | |
pattern to use to name the resulting follower indices. The | ||
`<follow_index_pattern>` can use the template `{{leader_index}}` to refer to the | ||
name of the leader index being followed. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
==== Path Parameters | |
`auto_follow_pattern_name` (required):: | |
(string) Identifier for the auto-follow patterns. | |
////////////////////////// | ||
|
||
Here, `<auto_follow_pattern_name>` is a user-chosen name for the collection of | ||
auto-follow patterns being created, `<leader_cluster>` is the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are there rules for what this name can contain? For example, for job names, we say "This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters."
name of the leader index being followed. | ||
|
||
==== Request Body | ||
`leader_cluster`:: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
`leader_cluster`:: | |
`leader_cluster` (required) :: |
|
||
==== Request Body | ||
`leader_cluster`:: | ||
(required string) the <<modules-remote-clusters,remote cluster>> containing the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(required string) the <<modules-remote-clusters,remote cluster>> containing the | |
(string) The <<modules-remote-clusters,remote cluster>> containing the |
leader indices to match against | ||
|
||
`leader_index_patterns`:: | ||
(array) an array of simple index patterns to match against indices in the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(array) an array of simple index patterns to match against indices in the | |
(array) An array of simple index patterns to match against indices in the |
remote cluster specified by the `leader_cluster` field | ||
|
||
`follow_index_pattern`:: | ||
(string) the name of follower index; the template `{{leader_index}}` can be |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(string) the name of follower index; the template `{{leader_index}}` can be | |
(string) The name of follower index; the template `{{leader_index}}` can be |
@@ -0,0 +1,30 @@ | |||
`max_batch_operation_count`:: | |||
(integer) the maximum number of operations to pull per fetch from the remote |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(integer) the maximum number of operations to pull per fetch from the remote | |
(integer) The maximum number of operations to pull per fetch from the remote |
cluster | ||
|
||
`max_concurrent_read_batches`:: | ||
(long) the maximum number of concurrent fetches from the remote cluster |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(long) the maximum number of concurrent fetches from the remote cluster | |
(long) The maximum number of concurrent fetches from the remote cluster |
(long) the maximum number of concurrent fetches from the remote cluster | ||
|
||
`max_batch_size`:: | ||
(<<byte-units,byte value>>) the maximum size in bytes of batches pulled per |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(<<byte-units,byte value>>) the maximum size in bytes of batches pulled per | |
(<<byte-units,byte value>>) The maximum size in bytes of batches pulled per |
|
||
==== Example | ||
|
||
The following is an example of using the pause follower API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following is an example of using the pause follower API. | |
This example pauses a follower index named `follower_index`: |
|
||
==== Example | ||
|
||
The following is an example of using the resume follower API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following is an example of using the resume follower API. | |
This example resumes a follower index named `follower_index`: |
|
||
==== Example | ||
|
||
The following is an example of using the unfollow API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following is an example of using the unfollow API. | |
This example converts `follower_index` from a follower index to a regular index: |
|
||
==== Example | ||
|
||
The following is an example of using the create follower API. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The following is an example of using the create follower API. | |
This example creates a follower index named `follower_index`: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I looked at the snippets and have two comments:
- You have a bunch of empty
// TEST
comments.// CONSOLE
implies// TEST
so you shouldn't need them but they don't hurt anything.// TEST
exists primarily to send extra stuff to the test framework rather than the doc generation framework. - In a few places you clean up after a test using
// TEST[continued]
. I think the// TEARDOWN
annotation that you built is better for that. But, I wonder if these are things that should be cleaned up higher up, like we do with rollups and such. I'm don't think the answer to that question should block merging, but it is a thing to think about.
* master: (74 commits) XContent: Check for bad parsers (elastic#34561) Docs: Align prose with snippet (elastic#34839) document the search context is freed if the scroll is not extended (elastic#34739) Test: Lookup node versions on rest test start (elastic#34657) SQL: Return error with ORDER BY on non-grouped. (elastic#34855) Reduce channels in AbstractSimpleTransportTestCase (elastic#34863) [DOCS] Updates Elasticsearch monitoring tasks (elastic#34339) Check self references in metric agg after last doc collection (elastic#33593) (elastic#34001) [Docs] Add `indices.query.bool.max_clause_count` setting (elastic#34779) Add 6.6.0 version to master (elastic#34847) Test: ensure char[] doesn't being with prefix (elastic#34816) Remove static import from HLRC doc snippet (elastic#34834) Logging: server: clean up logging (elastic#34593) Logging: tests: clean up logging (elastic#34606) SQL: Fix edge case: `<field> IN (null)` (elastic#34802) [Test] Mute FullClusterRestartIT.testShrink() until test is fixed SQL: Introduce ODBC mode, similar to JDBC (elastic#34825) SQL: handle X-Pack or X-Pack SQL not being available in a more graceful way (elastic#34736) [Docs] Add explanation for code snippets line width (elastic#34796) CCR: Rename follow-task parameters and stats (elastic#34836) ...
This commit adds some fields that were missing from put follow, and fixes a bug in resume follow.
* 'master' of github.com:elastic/elasticsearch: Fix line length for org.elasticsearch.common.* files (elastic#34888) [ML] Extract common native process base class (elastic#34856) Refactor children aggregator into a generic ParentJoinAggregator (elastic#34845) [Style] Fix line lengths in action.admin.indices (elastic#34890) HLRC - add support for source exists API (elastic#34519) [CCR] Retry when no index shard stats can be found (elastic#34852) [Docs] audit logfile structured format (elastic#34584) [Test] Fix FullClusterRestartIT.testShrink() with copy_settings param (elastic#34853) Fix LineLength Check Suppressions: index.fielddata (elastic#34891) TEST: Stablize Minio Free Port Search (elastic#34894) Delete flaky SettingsBasedHostProviderIT test (elastic#34813) [ML] Include message in field_stats for text log files (elastic#34861) [TEST] HLRC: Expand failure messages in API checks (elastic#34838) Lowercase static final DeprecationLogger instance names (elastic#34887)
* fix-put-resume-follow: Fix put/resume follow request parsing
* master: Fix put/resume follow request parsing (elastic#34913)
Thanks @lcawl, I've incorporated your feedback. |
This commit is our first introduction to cross-cluster replication docs. In this commit, we introduce the cross-cluster replication API docs. We also add skelton docs for additional content that will be added in a series of follow-up commits.
This commit is our first introduction to cross-cluster replication docs. In this commit, we introduce the cross-cluster replication API docs. We also add skelton docs for additional content that will be added in a series of follow-up commits.
* master: Introduce cross-cluster replication API docs (elastic#34726) Responses can use Writeable.Reader interface (elastic#34655) SQL: Provide null-safe scripts for Not and Neg (elastic#34877) Fix put/resume follow request parsing (elastic#34913) Fix line length for org.elasticsearch.common.* files (elastic#34888) [ML] Extract common native process base class (elastic#34856) Refactor children aggregator into a generic ParentJoinAggregator (elastic#34845) [Style] Fix line lengths in action.admin.indices (elastic#34890) HLRC - add support for source exists API (elastic#34519)
This commit is our first introduction to cross-cluster replication docs. In this commit, we introduce the cross-cluster replication API docs. We also add skelton docs for additional content that will be added in a series of follow-up commits.
This commit is our first introduction to cross-cluster replication docs. In this commit, we introduce the cross-cluster replication API docs. We also add skelton docs for additional content that will be added in a series of follow-up commits.