Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

restart indexing pipeline on index update #5265

Merged
merged 4 commits into from
Sep 3, 2024

Conversation

trinity-1686a
Copy link
Contributor

Description

restart indexing pipelines on index update
part of #5084

How was this PR tested?

todo

@trinity-1686a trinity-1686a force-pushed the trinity/update-restart-ingest branch from 40ae187 to 9c5d6f2 Compare July 28, 2024 18:14
Copy link

github-actions bot commented Jul 29, 2024

On SSD:

Average search latency is 0.999x that of the reference (lower is better).
Ref run id: 3206, ref commit: 9d8fab0
Link

On GCS:

Average search latency is 1.06x that of the reference (lower is better).
Ref run id: 3207, ref commit: 9d8fab0
Link

@trinity-1686a
Copy link
Contributor Author

this still needs automated tests, but should otherwise be ready for a first round of review

@trinity-1686a trinity-1686a marked this pull request as ready for review July 31, 2024 15:02
@trinity-1686a trinity-1686a requested a review from fulmicoton July 31, 2024 15:02
@rdettai
Copy link
Contributor

rdettai commented Aug 21, 2024

@trinity-1686a trinity-1686a requested a review from guilload August 23, 2024 13:32
@trinity-1686a trinity-1686a force-pushed the trinity/update-restart-ingest branch from 5395840 to da62963 Compare August 27, 2024 10:52
@trinity-1686a trinity-1686a force-pushed the trinity/update-restart-ingest branch from da62963 to 61d3622 Compare August 27, 2024 11:03
@trinity-1686a
Copy link
Contributor Author

thinking about it, as we only restart indexing pipelines, but don't do anything about shards, shards from ingest-v2 could have a stall doc_mapping, and if validation is enabled, they could reject documents that are valid under the new mapping. Ideally we'd want to also close all shards for the index and open as many new ones.
We would then end up with a race condition where we said "this document is valid" at validation, run it against the new mapper, and throw it away as invalid then, without notifying the writer.

@trinity-1686a trinity-1686a enabled auto-merge (squash) September 3, 2024 12:37
cluster sandbox creation changed
@trinity-1686a trinity-1686a merged commit 97614af into main Sep 3, 2024
5 checks passed
@trinity-1686a trinity-1686a deleted the trinity/update-restart-ingest branch September 3, 2024 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants