-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Backport 2.x] Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility (#9262) #9367
[Backport 2.x] Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility (#9262) #9367
Conversation
A breaking change was merged into 2.x a couple of hours ago [1] - how can we give impacted teams time to stabilize before this change is merged? |
Stabilize what? Main? Or 2.x? The original change was merged to main five days ago. Backport PRs usually come quicker than that. |
@peternied I agree. The new compress library does technically make this breaking. Thx for labeling accordingly! |
Compatibility status:
|
Actually, it doesn't look like this PR breaks anything in security repo? It looks like Then I don't see security directly calling So what does this backport break? |
Thanks for looking into the security repo - doing that by hand seems painful. Seems like the compatibility check was broken, how would you feel about waiting till that is operational before we merge changes like this? Maybe we could invest a little their to build confidence in changes like this rather than starting at zero (such as I am). |
It doesn't look broken? The 2.x incompatibility is unrelated to this change. It reported incompatible because the Action class changes were only just merged to 2.x after compatibility checker was run on this PR. |
I'm refiring check compatibility just in case. |
I don't have the data, so let not hold off because I say so. What do you think about coming up with a prediction the downstream impact of this change, then so long as we [1] are ok with that impact, then lets move forward [2]?
How does apache foundation handle cases like this? |
It's varies by project. I can only speak for Lucene and Solr (and Elasticsearch, and even OpenSearch). Solr (and Elasticsearch, and OpenSearch core) absorbs upstream changes "on demand" the same way we do it here with Lucene. This is what I have been suggesting for a while. Instead of upstream core OpenSearch repo force feeding per-merge |
Compatibility status:
|
@nknize Thanks - can you speak to this during our next operations review for discussion and next steps? While I prefer the structure we currently have in place to prevent 'debt build up' and point in time alignment, the approach you called out to addresses stability that might be the change the project needs. |
I wonder if we could invent a "toggle" mechanism to allow downstreams to switch between the two approaches at will? Maybe there's a GHA to enable/disable "force fed" snapshots? This could prove useful during large scale refactors to clot the bleeding? Downstreams cut over to the manual snapshot consumption model, and switch back when the change storm has passed? |
a5ebcea
to
ed5e9cc
Compare
Compatibility status:Checks if related components are compatible with change 4cb5e66 Incompatible componentsIncompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git] Skipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git] |
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## 2.x #9367 +/- ##
============================================
- Coverage 70.92% 70.86% -0.06%
+ Complexity 57634 57605 -29
============================================
Files 4762 4764 +2
Lines 272085 272063 -22
Branches 40110 40109 -1
============================================
- Hits 192975 192800 -175
- Misses 62564 62751 +187
+ Partials 16546 16512 -34
|
With the need to cut a 2.9.1 patch release we should lean into getting this backport merged so we can easily and quickly track 2.x changes against the BlobStoreRepository Zstd compression logic. |
@peternied are you OK with that? |
@reta Thanks for checking in - speaking for the security plugin team, looks fine for us |
… extensibility (opensearch-project#9262) This commit refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields. --------- Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit c6b67e1)
ed5e9cc
to
00fb53c
Compare
Compatibility status:Checks if related components are compatible with change 1fdc08a Incompatible componentsIncompatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git] Skipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git] |
These CI/CD actions are not at all reliable. It's crazy how much time is wasted re-firing all of these checks... It only seems to be getting worse. Can we simplify all of this? Or find something more reliable?
|
Gradle Check (Jenkins) Run Completed with:
|
Backport c6b67e1 from #9262.