OpenSearch 2.9.0 includes a range of new features and enhancements to support search, analytics, observability, and security workloads, along with new codecs that offer performance improvements and increased index compression. A new experimental SDK lets you build your own extensions for OpenSearch.
- Search pipelines functionality is now generally available. Now, you can configure processors to transform search requests and responses and integrate functions like query rewriters and results rerankers.
- The neural search plugin is now generally available. These tools can help you build applications like semantic search more easily and take advantage of OpenSearch’s vector database capabilities.
- The machine learning (ML) framework is generally available in this release. This makes it easier to operationalize and integrate ML models by enabling users to upload their own models to OpenSearch.
- You can now integrate externally managed ML models with OpenSearch using new connectors. Connectors to external models can be created by defining a blueprint in JSON.
- Vector database functionality is enhanced with pre-filtering for queries using the Facebook AI Similarity Search (FAISS) engine, enabling the use of metadata to filter queries prior to performing nearest neighbors searches on k-NN indexes.
- Support for Lucene byte-sized vectors allows the use of vectors that have been quantized to the size of one byte per dimension instead of four for reduced storage and memory requirements.
- You can now view alerts and anomalies directly overlaid with the primary dashboards you use to monitor your environments and create alerting monitors or anomaly detectors directly from OpenSearch Dashboards.
- New composite monitors allow users to chain alerts generated by multiple individual monitors into a single workflow, enabling users to analyze data sources based on multiple criteria, create targeted notifications, and reduce alert volume.
- A new index compression codec enables performance improvements and reduced index sizes. The new codecs, zstd and zstd_no_dict, use the Zstandard compression algorithm for balanced compression and performance.
- Security Analytics supports a new ingestion schema for log data that follows the Open Cybersecurity Schema Framework (OCSF). A new correlation engine lets you create custom correlation rules to produce findings across different log sources.
- Geospatial tools add support for three types of aggregations using geoshapes data: geo_bounds, geo_hash, and geo_tile.
- CAT shards and CAT indices are added to the cluster health monitor metrics, allowing you to monitor the state of all primary and replica shards along with information related to index health and resource usage to support operational uptime.
OpenSearch 2.9.0 release introduces an experimental SDK that gives developers the tools to build extensions for OpenSearch. The SDK provides interfaces for communicating with OpenSearch, allows extensions to be decoupled from OpenSearch versions, and other changes designed to make it easier for developers to build, and users to deploy, extensions of OpenSearch’s functionality.
OpenSearch and OpenSearch Dashboards 2.9.0 includes the following feature, enhancement, bug fixes, infrastructure, documentation, maintenance and refactoring updates.
OpenSearch Release Notes.
OpenSearch Dashboards Release Notes.
- Adds transport layer actions for CRUD workflows. (#934)
- Added rest layer for the workflow. (#963)
- [BucketLevelMonitor] Multi-term agg support. (#964)
- Check if AD backend role is enabled. (#968)
- Add workflow_id field in alert mapping json. (#969)
- Adds chained alerts. (#976)
- Implemented support for configuring a cluster metrics monitor to call cat/indices, and cat/shards. (#992)
- Integrate with Vis Augmenter to augment Dashboard visualizations. (#596)
- Implemented support for configuring a cluster metrics monitor to call cat/indices, and cat/shards. (#609)
- Implemented composite monitors. (#611)
- Merge featureAnywhere branch into main (#539)
- Pick changes from featureAnywhere branch into main (#542)
- Adds Chained alerts triggers for workflows. (#456)
- Acknowledge chained alert request for workflow. (#459)
- Adds audit state in Alert. (#461)
- Add workflowId field in alert. ((#463)
- APIs for get workflow alerts and acknowledge chained alerts. (#472)
- Add auditDelegateMonitorAlerts flag. (#476)
- Implemented support for configuring a cluster metrics monitor to call cat/indices, and cat/shards. (#479)
- Add MacOS CI Workflow(#412)
- Move notification plugin into management section on left navigation menu (#46)
- Integrations Plugin (#317)
- Support for system index interface. (#789)
- Implemented alias action UX. (#754)
- Added support for Efficient Pre-filtering for Faiss Engine (#936)
- Add Support for Lucene Byte Sized Vector (#971)
- Remote inference: add connector; fine tune ML model and tensor class (#1051)
- Remote inference: add connector executor (#1052)
- Connector transport actions, requests and responses (#1053)
- Refactor predictable: add method to check if model is ready (#1057)
- Add basic connector access control classes (#1055)
- Connector transport actions and disable native memory CB (#1056)
- Restful connector actions and UT (#1065)
- Change connector access control creation allow empty list (#1069)
- Exclude remote models in admin UI overview page. (#225)
- Remove experiment banner and enable plugin by default. (#229)
- New Log Type JSON format. (#465)
- Correlation rule search, delete and edit API. (#476)
- Logtypes PR v2. (#482)
- Enforce DOCUMENT Replication for AD Indices and Adjust Primary Shards (#948)
- Remove invalid link (#470)
- Remove global OUI style override (#511)
- Fixing unit tests plus ran prettier (#546)
- Create model group automatically with first model version (#1063)
- Init master key automatically (#1075))
- Open external link in new blank page. (#211)
- Remove heap allocation rate as the input metric to HotShardClusterRca #411
- Set ThreadMetricsRca evaluation period from 12 seconds to 5 seconds #410
- Add unit tests for the REST layer in RCA Agent #436
- Use boucycastle PEM reader instead of reg expression (#2877)
- Adding field level security test cases for FlatFields (#2876) (#2893)
- Add password message to /dashboardsinfo endpoint (#2949) (#2955)
- Add .plugins-ml-connector to system index (#2947) (#2954)
- Parallel test jobs for CI (#2861) (#2936)
- Adds a check to skip serialization-deserialization if request is for same node (#2765) (#2973)
- Add workflow cluster permissions to alerting roles and add .plugins-ml-config in the system index (#2996)
- Add RESTAPI permissions and ISM control center permissions to dropdown (#1473)
- Move security plugin from Plugin to Management section on left navigation menu (#1495)
- Filter high level groups and action groups by cluster and index (#1508)
- Use password message from /dashboardsinfo (#1509)
- Pagination: Support WHERE clause, column list in SELECT clause and for functions and expressions in the query (#1500)
- Pagination: Support ORDER BY clauses and queries without FROM clause (#1599)
- Remove backticks on by field in stats (#1728)
- Support Array and ExprValue Parsing With Inner Hits (#1737)
- Add Support for Nested Function in Order By Clause (#1789)
- Add Support for Field Star in Nested Function (#1773)
- Guarantee datasource read api is strong consistent read (compatibility with segment replication) (#1815)
- Added new datetime functions and aliases to PPL (#1807)
- Support user-defined and incomplete date formats (#1821)
- Add _routing to SQL includes list (#1771)
- Disable read of plugins.query.datasources.encryption.masterkey from cluster settings GET API (#1825)
- Add EMR client to spark connector (#1790)
- Improved error codes in case of data source API security exception (#1753)
- Remove Default master encryption key from settings (#1851)
- Fix schema version in tests and delegate monitor metadata construction in tests. (#948)
- Fixed search monitor API to return alert counts. (#978)
- Resolve string issues from core. (#987)
- Fix getAlerts RBAC problem. (#991)
- Fix alert constructor with noop trigger to use execution id and workflow id. (#994)
- Fixed a bug impacting data filter configuration through the visual editor. (#574)
- Fixed a unit test. (#576)
- Temporary fix for blank monitor name on Monitors page issue. (#593)
- OpenSearch commons strings library dependency import. (#474)
- Update job details mapping (#391)
- Force DOCUMENT replication on lock index (#417)
- Fix bug time out of range (#409)
- Update packages according to a change in OpenSearch core (#422)
- Handle bug in Shard replication task assignment (#881)
- Update Leader checkpoint when shard replication task is reinitialized (#904)
- Correctly handle retention lease renewal (if the lease already exists) during bootstrap (#904)
- Clear persistent tasks from cluster state after STOP API is triggered (#905)
- Handle OpenSearchRejectExecuteException Exception during replay (#1004)
- Fix Resume replication flow in dedicated master node configuration (#1030)
- Add missing LICENSE file on 2.x branch (#68))
- Fix Log patterns UI regressions (#530)
- [BugFix] Application Analytics - Panels not displaying availability (#607)
- Upgrade tough-cookie and semver (#627)
- Upgrade tough-cookie and semver (#91)
- Upgrade tough-cookie and semver (#135)
- Bugfix/202 transform date add date conversion. (#622)
- Fix build errors after refactoring of Strings class in core. (#786)
- Fix LRON concurrent indexing throw ResourceAlreadyExists. (#831)
- ImmutableOpenMap to j.u.Map. (#838)
- Remove runtime notification settings. (#839)
- Fix: add validation on data.method when using transport.request. (#801)
- Add missing codes from pen test fix (#1060)
- Fix cannot specify model access control parameters error (#1068)
- Fix memory circuit breaker (#1072)
- PenTest fixes: error codes and update model group fix (#1074)
- Fix rare private ip address bypass SSRF issue (#1070)
- Leftover in the 404 Not Found return error (#1079)
- Modify error message when model group not unique is provided (#1078)
- Stash context before accessing ml config index (#1092)
- Fix init master key bug (#1094)
- Delete connector successfully if model index is missing (#1106)
- Fix encoding US-ASCII error (#1126)
- Remove refresh interval background in dark mode. (#209)
- Update to correct machine learning document address for experiment banner and empty screen. (#210)
- Fix update document with knnn_vector size not matching issue (#208)
- Fix NPE issue in ShardStateCollector, which was impacted by changes from upstream core #489
- Fix Mockito initialization issue #443
- Removing guava dependency to fix jarhell (#709)
- Fixed compile issues related to latest OS core repo changes. (#412)
- Switch to new tenant after loading a copied long URL (#1450) (#1477)
- Fix a bad import path (#1498) (#1499)
- Fixed bug of byte/short not handling 0 denominator in divide/modulus equations (#1716)
- Fix CSV/RAW output header being application/json rather than plain/text (#1779)
- Updated Maintainers and CODE_OWNERS list (#926)
- Bump guava version to 32.0.1 (#933)
- Bump scipy from 1.8.0 to 1.10.0 in /dataGeneration (#943)
- Fix main build - update import of Releasable and remove reference to BaseExceptionsHelper (#930)
- Merge featureAnywhere branch into main (#539)
- Adding an integration test for redeploying a model (#1016)
- Add unit test for connector class in commons (#1058)
- Remote inference: add unit test for model and register model input (#1059)
- Remote inference: add unit test for StringUtils and remote inference input (#1061)
- More UT for rest and trasport actions (#1066)
- Remote inference: add unit test for create connector request/response (#1067)
- Add more UT for remote inference classes (#1077)
- IT Security Tests for model access control (#1095)
- Add integration tests for the remote inference feature (#1073)
- Temporarily ignore OpenAI Tests (#1118)
- Increase sleep time on remote inference integration tests (#1124)
- Fix remote inference it: update job interval (#1129)
- Add null key check for remote inference integration tests (#1113)
- Reenable OpenAI Remote Inference Tests (#1127)
- Run publish maven snapshots on all branches matching pattern (#698)
- Strings compile fix due to core package change(#680)
- Update the BWC version to 2.8.0 #446
- Upgrade bcprov to bcprov-jdk15to18 in performance-analyzer #493
- Upgrade bcprov to bcprov-jdk15to18 in performance-analyzer-rca 439
- Upgrade bcpkix to bcpkix-jdk15to18 in performance-analyzer-rca 446
- Upgrade checkstyle version from 9.3 to 10.3.3 #495
- StopPrometheus task in doctest build.gradle now runs upon project failure in startOpenSearch (#1747)
- Bump guava to 32.0.1 (#1829)
- Disable CrossClusterSearchIT test (#1814)
- Fix flakytest when tests.locale=tr (#1827)
- Added 2.9 release notes. (#1010)
- Added 2.9.0 release notes. (#621)
- Updated Maintainers and CODE_OWNERS list (#926)
- Added 2.9 release notes. (#482)
- Drafted 2.9 release notes. (#71)
- Integraions Plugin Design RFC (#644)
- Added 2.9 release notes. (#851)
- Added 2.9 release notes. (#808)
- Add 2.9.0 release notes (#702)
- Added 2.9.0 release notes. (#486)
- Added 2.9 release notes. (#650)
- Updated documentation of round function return type (#1725)
- Updated
protocol.rstwith new wording for error message (#1662) - Updated documentation for temporal data types (#1826)
- Added 2.9.0 release notes.
- Increment version to 2.9.0-SNAPSHOT. (#950)
- Increment version to 2.9.0.0. (#580)
- Increment version to 2.9.0 (300)
- Increment version to 2.9.0-SNAPSHOT. (#444)
- Modify triggers to push snapshots on all branches. (#454)
Increment version to 2.9.0.0 (#426) Bump semver from 7.3.8 to 7.5.3 (#431) Bump tough cookie to ^4.1.3 (#439)
- [AUTO] Increment version to 2.9.0.0 (#61)
- Increment version to 2.9.0.0 (#85)
- Increment version to 2.9.0.0 (#129)
- Increment version to 2.9.0.0 (#192)
Increment version to 2.9.0-SNAPSHOT (#329)
- Increment version to 2.9.0-SNAPSHOT. (#805)
- Added to CODEOWNERS and MAINTAINERS files. (#810)
- Upgrade httpcomponent version after OpenSearch core's upgrade. (#847)
- Increment version to 2.9.0.0. (#795)
- Move index management from plugin section to management section. (#741)
- Increment version to 2.9.0-SNAPSHOT (#955)
- Manual CVE backport (#1008)
- Fix build. (#1018)
- Fix the refactor change brought by core backport (#1047)
- Change to compileOnly to avoid jarhell (#1062)
- Update ml CI workflow checkout ref to be PR commit (#1114)
- Add shell bash settings for windows github actions to export credentials (#1116)
Increment version to 2.9.0-SNAPSHOT (#191)
- [AUTO] Increment version to 2.9.0-SNAPSHOT (#690)
- Update build.gradle and github workflow to support 2.9 version #499
- Update licenses files for 2.9 #501
- Swap jboss annotation dependency for jakarta annotations #407
- Ensures compatibility check readiness #438
- Increment version to 2.9.0-SNAPSHOT (#712)
- Match version of zstd-jni from core (#2835)
- Add Andrey Pleskach (Willyborankin) to Maintainers (#2843)
- Updates bwc versions to latest release (#2849)
- Add search model group permission to ml_read_access role (#2855) (#2858)
- Format 2.x (#2878)
- Update snappy to 1.1.10.1 and guava to 32.0.1-jre (#2886) (#2889)
- Resolve ImmutableOpenMap issue from core refactor (#2908)
- Misc changes (#2902) (#2904)
- Bump BouncyCastle from jdk15on to jdk15to18 (#2901) (#2917)
- Fix the import org.opensearch.core.common.Strings; and import org.opensearch.core.common.logging.LoggerMessageFormat; (#2953)
- Remove commons-collections 3.2.2 (#2924) (#2957)
- Resolve CVE-2023-2976 by forcing use of Guava 32.0.1 (#2937) (#2974)
- Bump jaxb to 2.3.8 (#2977) (#2979)
- Update Gradle to 8.2.1 (#2978) (#2981)
- Changed maven repo location for compatibility check (#2988)
- Bump guava to 32.1.1-jre (#2976) (#2990)
- Use version from package.json for integration tests (#1465)
- Update permissions according to backend (#1480) (#1484)
- Use strong password in security test. (#933)
- Pass workflow id in alert constructors. (#465)
- Change package for Strings.hasText (#314)
- Add class for loading mapping templates in bulk (#1550)
- Use strong password in security test. (#452)
- Simplify OpenSearchIndexScanBuilder (#1738)