[Backport 2.5] Manual backports of several PRs

[AWSHurneyt]

Issue #, if available:

Description of changes:
This PR manually backports PRs #728, #729, #784, #861, and #900.

For context, the following PRs were identified as needing backport to the 2.5 branch. Some of them were able to be backported automatically; the backport PR is listed next to those PRs for reference. Some of the changes were already present in the 2.5 branch.

1. Added unwrapping exceptions from core #728
2. [Backport 2.x] QueryIndex rollover when field mapping limit is reached. #729
3. Fix setting default title only when no subject has been set #750 -> [Backport 2.5] Fix setting default title only when no subject has been set #786
4. [BUG] ExecuteMonitor inserting metadata doc during dry run #758 -> [Backport 2.5] [BUG] ExecuteMonitor inserting metadata doc during dry run #778
5. fix for ERROR alert state generation in doc-level monitors #768 -> [Backport 2.5] [Backport 2.x] fix for ERROR alert state generation in doc-level monitors #772
6. [Backport 2.x] Adjusting max field index setting dynamically for query index #779 -> [Backport 2.5] Adjusting max field index setting dynamically for query index #780
7. Multiple indices support in DocLevelMonitorInput #784
8. revert enabled field in source_to_query_index_mapping #812 -> changes present
9. Fixed transformation of document field names before running percolator search #845 -> doc transform 2.5 backport #854
10. Update config index schema if needed at the start of each monitor execution #849 -> [Backport 2.5] Update config index schema if needed at the start of each monitor execution #867
11. Mappings parsing fix #851 -> [Backport 2.5] Mappings parsing fix #921
12. [Backport 2.x] Notification security fix #861
13. [Backport 2.x] Notification security fix (#861) #863 -> changes present
14. Fix getAlerts API for standard Alerting monitors #870 -> [Backport 2.5] Fix getAlerts API for standard Alerting monitors #920
15. Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names. #894 -> [Backport 2.5] Fixed a bug that prevented alerts from being generated for doc level monitors that use wildcard characters in index names. #902
16. log error messages and clean up monitor when indexing doc level queries or metadata creation fails #900

CheckList:
[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[AWSHurneyt]

This test workflows will fail as the 2.5.1 artifacts for common utils and notifications are not available. ./gradlew build succeeds locally after manually publishing the respective artifacts to the local maven repository.

[AWSHurneyt]

Test output for visibility.

./gradlew clean && ./gradlew build
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Linux 5.4.241-160.348.amzn2int.x86_64 (amd64)
  JDK Version           : 14 (Oracle JDK)
  JAVA_HOME             : /local/home/hurneyt/jdk-14.0.2
  Random Testing Seed   : 3C516035428F198C
  In FIPS 140 mode      : false
=======================================

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 743ms
3 actionable tasks: 3 executed
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 7.4.2
  OS Info               : Linux 5.4.241-160.348.amzn2int.x86_64 (amd64)
  JDK Version           : 14 (Oracle JDK)
  JAVA_HOME             : /local/home/hurneyt/jdk-14.0.2
  Random Testing Seed   : 220F4EA34D4857FD
  In FIPS 140 mode      : false
=======================================

> Task :alerting-core:compileKotlin
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/core/src/main/kotlin/org/opensearch/alerting/core/action/node/ScheduledJobsStatsTransportAction.kt: (11, 44): 'BaseNodeRequest' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/core/src/main/kotlin/org/opensearch/alerting/core/action/node/ScheduledJobsStatsTransportAction.kt: (119, 39): 'BaseNodeRequest' is deprecated. Deprecated in Java

> Task :alerting:compileKotlin
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/BucketLevelMonitorRunner.kt: (62, 9): The corresponding parameter in the supertype 'MonitorRunner' is named 'dryRun'. This may cause problems when calling this function with named arguments.
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt: (68, 9): The corresponding parameter in the supertype 'MonitorRunner' is named 'dryRun'. This may cause problems when calling this function with named arguments.
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt: (102, 60): Unchecked cast: MutableMap<String, Any> to MutableMap<String, MutableMap<String, Any>>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt: (459, 59): Unnecessary non-null assertion (!!) on a non-null receiver of type ClusterService
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/DocumentLevelMonitorRunner.kt: (607, 57): Unchecked cast: Any to MutableMap<String, Any>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/MonitorMetadataService.kt: (167, 76): Unchecked cast: Map<String, Any> to MutableMap<String, MutableMap<String, Any>>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/QueryLevelMonitorRunner.kt: (28, 9): The corresponding parameter in the supertype 'MonitorRunner' is named 'dryRun'. This may cause problems when calling this function with named arguments.
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/alerts/AlertIndices.kt: (220, 42): 'getter for isLocalNodeElectedMaster: Boolean' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/alerts/AlertIndices.kt: (228, 42): 'getter for isLocalNodeElectedMaster: Boolean' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/alerts/AlertIndices.kt: (275, 60): Unnecessary non-null assertion (!!) on a non-null receiver of type String
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/alerts/AlertIndices.kt: (277, 43): Unnecessary non-null assertion (!!) on a non-null receiver of type String
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/model/DocumentLevelTriggerRunResult.kt: (34, 51): No cast needed
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/model/MonitorMetadata.kt: (41, 51): Unchecked cast: (Mutable)Map<String!, Any!>! to MutableMap<String, String>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/model/MonitorMetadata.kt: (51, 48): Unchecked cast: MutableMap<String, String> to MutableMap<String, Any>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/model/MonitorMetadata.kt: (61, 86): Unchecked cast: MutableMap<String, String> to MutableMap<String, Any>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/model/MonitorMetadata.kt: (101, 94): Unchecked cast: (Mutable)Map<String!, Any!>! to MutableMap<String, String>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/resthandler/RestIndexMonitorAction.kt: (87, 69): Unchecked cast: Any? to List<String>?
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/resthandler/RestIndexMonitorAction.kt: (93, 9): Non exhaustive 'when' statements on enum will be prohibited in 1.7, add 'CLUSTER_METRICS_MONITOR' branch or 'else' branch instead
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/resthandler/RestIndexMonitorAction.kt: (131, 13): Condition 'monitor.dataSources != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportDeleteMonitorAction.kt: (207, 25): Unsafe use of a nullable receiver of type TotalHits?
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/transport/TransportDeleteMonitorAction.kt: (218, 29): Variable 'response' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (169, 38): Unchecked cast: Any to MutableMap<String, Any>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (176, 80): Unchecked cast: Any to MutableMap<String, Any>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (180, 65): Unchecked cast: Any? to MutableMap<String, Any>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (213, 16): Unnecessary safe call on a non-null receiver of type List<String>. This expression will have nullable type in future releases
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (219, 29): Unchecked cast: Any? to MutableMap<String, Any>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (323, 37): Variable 'updateMappingResponse' initializer is redundant
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (342, 25): Name shadowed: updateMappingRequest
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (351, 25): Name shadowed: unwrappedException
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (364, 21): Name shadowed: unwrappedException
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/DocLevelMonitorQueries.kt: (394, 17): Variable 'updateSettingsResponse' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/main/kotlin/org/opensearch/alerting/util/destinationmigration/DestinationMigrationCoordinator.kt: (77, 45): 'getter for isLocalNodeElectedMaster: Boolean' is deprecated. Deprecated in Java

> Task :alerting:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :alerting:compileTestKotlin
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/AlertingRestTestCase.kt: (528, 52): Unchecked cast: Any? to Map<String, Any>
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (188, 13): Variable 'testDoc' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (523, 13): Variable 'searchResult' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (562, 13): Variable 'searchResult' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (615, 13): Variable 'searchResult' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (676, 13): Variable 'searchResult' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (697, 13): Variable 'testDoc' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/DocumentMonitorRunnerIT.kt: (747, 13): Variable 'testDoc' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (51, 8): This class shouldn't be used in Kotlin. Use kotlin.collections.Map or kotlin.collections.MutableMap instead.
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (325, 13): Variable 'mappingsResp' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (662, 38): Variable 'executeMonitorResponse' initializer is redundant
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (751, 13): Variable 'clusterStateResponse' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1080, 13): Condition 'updateMonitorResponse != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1081, 13): The value 'executeMonitor(updateMonitorResponse.monitor, monitorId, false)' assigned to 'var executeMonitorResponse: ExecuteMonitorResponse? defined in org.opensearch.alerting.MonitorDataSourcesIT.`test execute pre-existing monitor and update`' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1548, 27): Condition 'alerts != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1566, 27): Condition 'alerts != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1574, 27): Condition 'alerts != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1667, 27): Condition 'alerts != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1683, 27): Condition 'alerts != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1691, 27): Condition 'alerts != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1735, 36): Unnecessary safe call on a non-null receiver of type IndexMonitorResponse?. This expression will have nullable type in future releases
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1773, 9): The value 'monitorResponse!!.monitor' assigned to 'var monitor: Monitor defined in org.opensearch.alerting.MonitorDataSourcesIT.`test queryIndex gets increased max fields in mappings`' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/MonitorDataSourcesIT.kt: (1818, 31): Condition 'alerts != null' is always 'true'
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (105, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (165, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (186, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (229, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (273, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/aggregation/bucketselectorext/BucketSelectorExtAggregatorTests.kt: (321, 17): 'assertThat(T!, Matcher<in T!>!): Unit' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/resthandler/MonitorRestApiIT.kt: (831, 13): Variable 'alert' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/resthandler/MonitorRestApiIT.kt: (912, 17): Name shadowed: alerts
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/transport/AlertingSingleNodeTestCase.kt: (69, 9): The expression is unused
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/transport/AlertingSingleNodeTestCase.kt: (80, 9): 'createIndex(String!, Settings!, String!, vararg String!): IndexService!' is deprecated. Deprecated in Java
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/transport/AlertingSingleNodeTestCase.kt: (114, 13): Variable 'foundAlias' is never used
w: /local/home/hurneyt/workplace/OpenSearch-Alerting/alerting/src/test/kotlin/org/opensearch/alerting/transport/AlertingSingleNodeTestCase.kt: (125, 13): Variable 'foundAlias' is never used

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 10m 11s
45 actionable tasks: 43 executed, 2 up-to-date