Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update config index schema if needed at the start of each monitor execution #849

Merged
merged 5 commits into from
Apr 18, 2023
Merged

Update config index schema if needed at the start of each monitor execution #849

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d3b88ce
Update config index schema if needed at the start of each monitor exe…
lezzago Apr 4, 2023
ee99f5a
move logic at node startup
lezzago Apr 13, 2023
73c4f83
fix ktlint
lezzago Apr 13, 2023
d9ae868
undo other changes
lezzago Apr 13, 2023
cea9b9e
remove added logger
lezzago Apr 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions alerting/src/main/kotlin/org/opensearch/alerting/MonitorRunnerService.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,13 @@ import kotlinx.coroutines.Job
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.launch
import org.apache.logging.log4j.LogManager
import org.opensearch.action.ActionListener
import org.opensearch.action.bulk.BackoffPolicy
import org.opensearch.action.support.master.AcknowledgedResponse
import org.opensearch.alerting.alerts.AlertIndices
import org.opensearch.alerting.alerts.moveAlerts
import org.opensearch.alerting.core.JobRunner
import org.opensearch.alerting.core.ScheduledJobIndices
import org.opensearch.alerting.model.MonitorRunResult
import org.opensearch.alerting.model.destination.DestinationContextFactory
import org.opensearch.alerting.opensearchapi.retry
Expand All @@ -29,6 +32,7 @@ import org.opensearch.alerting.settings.DestinationSettings.Companion.ALLOW_LIST
import org.opensearch.alerting.settings.DestinationSettings.Companion.HOST_DENY_LIST
import org.opensearch.alerting.settings.DestinationSettings.Companion.loadDestinationSettings
import org.opensearch.alerting.util.DocLevelMonitorQueries
import org.opensearch.alerting.util.IndexUtils
import org.opensearch.alerting.util.isDocLevelMonitor
import org.opensearch.client.Client
import org.opensearch.cluster.metadata.IndexNameExpressionResolver
Expand Down Expand Up @@ -223,6 +227,23 @@ object MonitorRunnerService : JobRunner, CoroutineScope, AbstractLifecycleCompon
}

suspend fun runJob(job: ScheduledJob, periodStart: Instant, periodEnd: Instant, dryrun: Boolean): MonitorRunResult<*> {

// Updating the scheduled job index at the start of monitor execution runs for when there is an upgrade the the schema mapping
// has not been updated.
if (!IndexUtils.scheduledJobIndexUpdated && monitorCtx.clusterService != null && monitorCtx.client != null) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add tests to validate this execution flow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would have to be a part of this issue: #848.

We have no framework for this setup

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this not being done at node start up?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

scheduled jobs index schema shouldnt be handled inside a scheduled job. Tomorrow when we have workflows this might become a miss. Let's handle it at a holistic level

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only place if we do it at node startup would be in the Alerting plugin class, but it would not work there since to make some of the calls to check the index would not work as the node is still setting up services. I felt it was too much to startup another service from the Alerting plugin class to update the config schema if needed. I believe we can just have the same logic in the workflow runner service to handle this. Additionally the workflow runner service and monitor runner service have very similar logic so maybe parts of it can be refactored to reduce redundant code.

IndexUtils.updateIndexMapping(
ScheduledJob.SCHEDULED_JOBS_INDEX,
ScheduledJobIndices.scheduledJobMappings(), monitorCtx.clusterService!!.state(), monitorCtx.client!!.admin().indices(),
object : ActionListener<AcknowledgedResponse> {
override fun onResponse(response: AcknowledgedResponse) {
}
override fun onFailure(t: Exception) {
logger.error("Failed to update config index schema", t)
}
}
)
}

val monitor = job as Monitor
val runResult = if (monitor.isBucketLevelMonitor()) {
BucketLevelMonitorRunner.runMonitor(monitor, monitorCtx, periodStart, periodEnd, dryrun)
Expand Down