Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enrich indices can clash with user allocation settings #54151

Closed
markharwood opened this issue Mar 25, 2020 · 9 comments · Fixed by #69334
Closed

Enrich indices can clash with user allocation settings #54151

markharwood opened this issue Mar 25, 2020 · 9 comments · Fixed by #69334
Assignees
@DaveCTurner
Labels
:Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@markharwood
Copy link
Contributor

A user reported that enrich indices were not being allocated because of the following incompatible objectives:

  1. The system requirement to have an enrich shard replica on every node
  2. The user choice to have >1 node per host
  3. The user choice to not allocate same replica shards to the same host via cluster.routing.allocation.same_shard.host

While the enrich indices were not fulfilling their objective to have a replica on every node the system is functional and there is good locality of data (each host has a copy) but the cluster state is permanently yellow. This is an annoyance from a monitoring perspective.
@markharwood markharwood added the :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) label Mar 25, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Allocation)

@martijnvg martijnvg added the :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP label Mar 25, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Ingest)

@martijnvg martijnvg added team-discuss and removed :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) labels Mar 25, 2020
@ywelsch ywelsch added the :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) label Apr 1, 2020
@jasontedor
Copy link
Member

There is a general problem here, not specific to the enrich indices, relating to auto-expand replicas and how it interplays with other allocation settings (e.g., #2869). Therefore, we are re-adding the distributed label, and treating this as a distributed problem.

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Apr 1, 2020 •
edited
Loading

We discussed this today and think it would be a good idea to ignore the same-host allocation decider and the allocation awareness decider if using auto-expand replicas with an upper bound of all. Both of these deciders aim to achieve resilience by avoiding a concentration of shards on a single failure domain (host or zone) but you can't really get more resilient than the case where there's a copy of every shard on every node, which is what 0-all does, so these deciders do not have any real benefits.

A question to the Enrich folks: does this do what you want? We were working on the assumption that you really want a copy of every shard on each node. An alternative, if cluster.routing.allocation.same_shard.host were set, would be to require a copy of every shard on every host and to route Enrich's searches to other nodes on the same host. That sounds harder, and although it avoids a proper network hop it still entails some more de/serialization than you might like.

This also may relate to system indices so I'm adding the :Core/Infra/Core label. System indices folks: will there be system indices that should have a copy on every node and, if so, does this proposal work for you?

All other feedback also welcomed.

@DaveCTurner DaveCTurner added :Core/Infra/Core Core issues without another label feedback_needed labels Apr 1, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (:Core/Infra/Core)

@gwbrown
Copy link
Contributor

gwbrown commented Apr 2, 2020 •
edited
Loading

Enrich indices will be system indices, so if Enrich requires a shard copy on every node, by definition that will be a requirement for some system indices. I believe there are some others that would benefit from that treatment as well.

How allocation is configured for system indices is something we’ve been thinking about but have not yet reached a conclusion on, in part because of radically varying requirements for different system indices. Now that we’ve gotten hidden indices fully ready to ship, we’re looking to solidify things very soon. In any event, I don’t foresee either of the solutions proposed here colliding with our plans (per our current thinking at least).

@martijnvg
Copy link
Member

We discussed this today and think it would be a good idea to ignore the same-host allocation decider and the allocation awareness decider if using auto-expand replicas with an upper bound of all. Both of these deciders aim to achieve resilience by avoiding a concentration of shards on a single failure domain (host or zone) but you can't really get more resilient than the case where there's a copy of every shard on every node, which is what 0-all does, so these deciders do not have any real benefits.

Thanks. I was leaning in the same direction too. I think the allocation deciders that you mentioned should implement the shouldAutoExpandToNode(...) method? (similar to what a happened to filter allocation decider in #48974)

A question to the Enrich folks: does this do what you want?

Yes. Enrich was built with the assumption that not all enrichments/lookups are going to be local. Especially in the case with clusters that only have dedicated ingest nodes. We built a buffering mechanism to bundle as search requests as much as possible under a certain load.

@DaveCTurner
Copy link
Contributor

Thanks. I was leaning in the same direction too. I think the allocation deciders that you mentioned should implement the shouldAutoExpandToNode(...) method? (similar to what a happened to filter allocation decider in #48974)

No, it's kind of the opposite of that. The filter allocation decider overrules auto-expand replicas, whereas I'm proposing having auto-expand replicas overrule allocation awareness/same host.

@rjernst rjernst added Team:Data Management Meta label for data/management team Team:Core/Infra Meta label for core/infra team Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. labels May 4, 2020
@rjernst rjernst added the needs:triage Requires assignment of a team area label label Dec 3, 2020
@pgomulka pgomulka added help wanted adoptme and removed needs:triage Requires assignment of a team area label labels Dec 15, 2020
@DaveCTurner DaveCTurner removed :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP :Core/Infra/Core Core issues without another label Team:Data Management Meta label for data/management team Team:Core/Infra Meta label for core/infra team feedback_needed labels Feb 22, 2021
@DaveCTurner
Copy link
Contributor

DaveCTurner commented Feb 22, 2021 •
edited
Loading

Hearing no objections, we will proceed with the plan to skip the allocation-awareness and same-host deciders on shards that have auto-expand replicas with an upper bound of all. I've removed the labels that were bringing this to the attention of other teams.

@DaveCTurner DaveCTurner added the good first issue low hanging fruit label Feb 22, 2021
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Feb 22, 2021
@DaveCTurner
Skip zone/host awareness with auto-expand replicas
4f601e9 
Today if an index is set to `auto_expand_replicas: N-all` then we will
try and create a shard copy on every node that matches the applicable
allocation filters. This conflits with shard allocation awareness and
the same-host allocation decider if there is an uneven distribution of
nodes across zones or hosts, since these deciders prevent shard copies
from being allocated unevenly and may therefore leave some unassigned
shards.

The point of these two deciders is to improve resilience given a limited
number of shard copies but there is no need for this behaviour when the
number of shard copies is not limited, so this commit supresses them in
that case.

Closes elastic#54151
Closes elastic#2869
@DaveCTurner DaveCTurner mentioned this issue Feb 22, 2021
Merged
@DaveCTurner DaveCTurner removed good first issue low hanging fruit help wanted adoptme labels Feb 22, 2021
@DaveCTurner DaveCTurner self-assigned this Feb 22, 2021
DaveCTurner added a commit that referenced this issue Feb 22, 2021
@DaveCTurner
Skip zone/host awareness with auto-expand replicas (#69334)
bb3ea99 
Today if an index is set to `auto_expand_replicas: N-all` then we will
try and create a shard copy on every node that matches the applicable
allocation filters. This conflits with shard allocation awareness and
the same-host allocation decider if there is an uneven distribution of
nodes across zones or hosts, since these deciders prevent shard copies
from being allocated unevenly and may therefore leave some unassigned
shards.

The point of these two deciders is to improve resilience given a limited
number of shard copies but there is no need for this behaviour when the
number of shard copies is not limited, so this commit supresses them in
that case.

Closes #54151
Closes #2869
DaveCTurner added a commit that referenced this issue Feb 22, 2021
@DaveCTurner
Skip zone/host awareness with auto-expand replicas (#69334)
1c1c3bb 
Today if an index is set to `auto_expand_replicas: N-all` then we will
try and create a shard copy on every node that matches the applicable
allocation filters. This conflits with shard allocation awareness and
the same-host allocation decider if there is an uneven distribution of
nodes across zones or hosts, since these deciders prevent shard copies
from being allocated unevenly and may therefore leave some unassigned
shards.

The point of these two deciders is to improve resilience given a limited
number of shard copies but there is no need for this behaviour when the
number of shard copies is not limited, so this commit supresses them in
that case.

Closes #54151
Closes #2869
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DaveCTurner DaveCTurner

Labels
:Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Skip zone/host awareness with auto-expand replicas DaveCTurner/elasticsearch
9 participants
@markharwood @rjernst @martijnvg @gwbrown @ywelsch @jasontedor @DaveCTurner @pgomulka @elasticmachine