feat(sql): Optimize executions #4804

kirangodishala · 2024-11-24T19:04:13Z

This PR improves performance when using sql as the backend by optimizing the way the results are obtained for the api call /applications/{application}/pipelines?expand=false&limit=2 frequently initiated by deck.

Deck makes that call to Gate. Gate further forwards that call to Orca, specifically to /v2/applications/{application}/pipelines endpoint.

Orca calls front50 to get the pipeline and strategy config ids and then merges them. Then for each id in the merged list, it queries its own db (SQL) to handle the query parameters, such as limit=2 and filter by statuses. It is observed that these db operations are taking longer times and this PR fixes the slowness by optimizing the queries to the db.

Refactored TaskController to support externalized config properties

Optimize getPipelinesForApplication() by reducing the number of relevant pipeline config ids to query

Further optimize getPipelinesForApplication() by processing multiple pipeline config ids at a time and multi-threading each config id batch

Breaking changes:

These configuration properties

tasks:
   days-of-execution-history:  
   number-of-old-pipeline-executions-to-include:

are replaced by the below configuration along with few more newly added ones:

tasks:
   controller:
      days-of-execution-history: 
      number-of-old-pipeline-executions-to-include: 

      optimize-execution-retrieval: <boolean>
      max-execution-retrieval-threads:
      max-number-of-pipeline-executions-to-process: 
      execution-retrieval-timeout-seconds:

orca-web/orca-web.gradle

jasonmcintosh · 2024-12-03T19:29:17Z

...ore/src/main/java/com/netflix/spinnaker/orca/pipeline/persistence/DualExecutionRepository.kt

+    @Nonnull pipelineConfigIds: List<String>,
+    @Nonnull criteria: ExecutionCriteria
+  ): List<String> {
+    return primary.retrieveAndFilterPipelineExecutionIdsForApplication(application, pipelineConfigIds, criteria) +


MINOR: BUT ... this ordering DOES matter on the selection criteria and don't think we have a test for this (aka primary MUST come first or it'd select previous data potentially). Just an observation that this is missing a test .. but not sure how "critical" that test really is.

jasonmcintosh · 2024-12-03T20:02:46Z

...rc/main/kotlin/com/netflix/spinnaker/orca/sql/pipeline/persistence/SqlExecutionRepository.kt

+    var baseQueryPredicate = field("application").eq(application)
+      .and(field("config_id").`in`(*pipelineConfigIds.toTypedArray()))
+
+    var table = if (jooq.dialect() == SQLDialect.MYSQL) PIPELINE.tableName.forceIndex("pipeline_application_idx")


Question: DO we really want to force an index, or is another index on the combination needed? OR perhaps merging a couple of these indexes. NOT sure best solution but... we have a number of indexes with some of this data and wondering if we couldn't do this a touch cleaner. I'd EXPECT the optimizer to handle MOST of this without the need to force an index.

...rc/main/kotlin/com/netflix/spinnaker/orca/sql/pipeline/persistence/SqlExecutionRepository.kt

orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy

jasonmcintosh · 2024-12-03T20:39:55Z

orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy

+
+    // need to define a new executor service since we want a dedicated set of threads to be made available for every
+    // new request for performance reasons
+    ExecutorService executorService = Executors.newFixedThreadPool(


I gotta admin that having this in the request path generating a new thread pool PER request seems... dangerous though a bigger question might be what's the downstream impact on the DB, and the impact on running concurrent executions on the DB? Since these are all hitting the RW pod as a rule right now. I could see this adding a tremendous amount of load on the database as a result...

It's not quite a thread-per-request since the number of threads is set by a config prop (default 10).

#4803 comes next, with more to follow as we move requests to a read-only database replica.

Yeah saw the properties but generating a thread pool on the request even WITH limits is still backed by a queue ... It's still shutdown but this could absolutely hammer a database badly because of a potential exponential call graph.
E.g. a browser can repeat calls which then would make new calls on timeouts that'd overload this badly. The RO PR should help A LOT with that however, and IN THEORY this should improve performance. It's just a concern based upon how I've seen these API's get called.

Yup. We also ended up making changes in deck to improve things. Those are gonna be harder for us to upstream, but we'll try.

And yes, massive pipelines cause lots of load.

jasonmcintosh · 2024-12-09T22:49:19Z

orca-web/src/main/java/com/netflix/spinnaker/config/TaskControllerConfigurationProperties.java

+   * <p>It can be further tuned, depending on your setup, since 150 executions work well for some
+   * applications but a higher number may be appropriate for others.
+   */
+  int maxNumberOfPipelineExecutionsToProcess = 150;


A concern is this is probably TOO high depending upon capacity for "smaller" installations - as is probably the max execution threads. IF you allow for 3 concurrent API calls at a time, that's 50 threads that get spawned to meet this need, which is potentially 30 separate threads hitting the database. That's a LOT of concurrency against the database & databse connections & could impact pipeline operations as a result. And you don't want to make your connection pools TOO big (beyond CPU capacity) or it actually gets WORSE.

This is a configurable value and matters only when optimizeExecutionRetrieval = true which by default is false. The idea is that this feature is enabled only when the pipelines and/executions are high.

It would still be nice if the defaults are useful for more folks. What do you suggest here @jasonmcintosh ?

ponder so NORMAL recommendations on pool/concurrency size is tied to CPU's. I think most people run on smaller instances with limited cores, and with other background processes... I'd just go to smaller defaults? Probably maxExecutionRetrievalThreads set to 4 maybe. shrug it's a hard one to judge - m5.2xlarge has only 8 cpus (which... i'll ignore hyperthreading) so you could hit some concurrency issues. It's not IMPOSSIBLE... I just tend to vote "smaller, more of them" kinda stuff on this.

christosarvanitis · 2024-12-11T13:16:24Z

The whole idea seems like a good optimisation to me 👍 @dbyron-sf @kirangodishala how long have you been running this in your forks?

Also could we fit the same concept of optimisations in retrievePipelinesForPipelineConfigIdsBetweenBuildTimeBoundary in this PR?

dbyron-sf · 2024-12-11T20:02:26Z

...rc/main/kotlin/com/netflix/spinnaker/orca/sql/pipeline/persistence/SqlExecutionRepository.kt

+        .queryTimeout(queryTimeoutSeconds) // add an explicit timeout so that the query doesn't run forever
+        .fetch()
+
+      log.info("getting stage information for all the executions found so far")


This doesn't seem valuable enough to leave at info.

Suggested change

log.info("getting stage information for all the executions found so far")

log.debug("getting stage information for all the executions found so far")

dbyron-sf · 2024-12-11T20:03:14Z

orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy

    if (!expand) {
-      unexpandPipelineExecutions(allPipelines)
+      log.info("unexpanding pipeline executions")


Suggested change

log.info("unexpanding pipeline executions")

log.debug("unexpanding pipeline executions")

dbyron-sf · 2024-12-11T20:04:14Z

orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy

    }

-    return filterPipelinesByHistoryCutoff(allPipelines, limit)
+    log.info("filtering pipelines by history")


Suggested change

log.info("filtering pipelines by history")

log.debug("filtering pipelines by history")

…perties optimize getPipelinesForApplication() by reducing the number of relevant pipeline config ids to query This is a breaking change: TaskController previously looked at these properties: `tasks.days-of-execution-history` `tasks.number-of-old-pipeline-executions-to-include` and now they live in `tasks.controller`. fix(web): give java code access to groovy to fix below error: > Task :orca-web:compileJava FAILED ../orca/orca-web/src/main/java/com/netflix/spinnaker/config/TaskControllerConfigurationProperties.java:19: error: cannot find symbol import com.netflix.spinnaker.orca.controllers.TaskController; ^ symbol: class TaskController location: package com.netflix.spinnaker.orca.controllers 1 error

…ion() further by processing multiple pipeline config ids at a time and multi-threading each config id batch

…r request

refactor function and variable names to make them more descriptive

kirangodishala requested a review from dbyron-sf November 24, 2024 19:04

kirangodishala changed the title ~~Optimize executions~~ feat(sql): Optimize executions Nov 25, 2024

dbyron-sf reviewed Nov 25, 2024

View reviewed changes

orca-web/orca-web.gradle Outdated Show resolved Hide resolved

kirangodishala force-pushed the optimize_executions branch 2 times, most recently from fb4fd18 to 40fd05d Compare November 28, 2024 17:20

jasonmcintosh reviewed Dec 3, 2024

View reviewed changes

...rc/main/kotlin/com/netflix/spinnaker/orca/sql/pipeline/persistence/SqlExecutionRepository.kt Outdated Show resolved Hide resolved

jasonmcintosh reviewed Dec 3, 2024

View reviewed changes

...rc/main/kotlin/com/netflix/spinnaker/orca/sql/pipeline/persistence/SqlExecutionRepository.kt Show resolved Hide resolved

jasonmcintosh reviewed Dec 3, 2024

View reviewed changes

orca-web/src/main/groovy/com/netflix/spinnaker/orca/controllers/TaskController.groovy Outdated Show resolved Hide resolved

jasonmcintosh reviewed Dec 3, 2024

View reviewed changes

jasonmcintosh reviewed Dec 9, 2024

View reviewed changes

dbyron-sf reviewed Dec 11, 2024

View reviewed changes

kirangodishala and others added 7 commits December 13, 2024 22:12

feat(taskController): optimize TaskController.getPipelinesForApplicat…

213b308

…ion() further by processing multiple pipeline config ids at a time and multi-threading each config id batch

refactor(taskController): rework execution retrieval query optimizations

4aee645

feat(taskController): have a dedicated executor service threadpool pe…

471f2f9

…r request

fix(sql): add correct indexes

77d0b12

refactor function and variable names to make them more descriptive

feat(taskController): add query timeouts when retrieving execution body

3b3894b

fix(log): minor log refactor

ddf724e

kirangodishala force-pushed the optimize_executions branch from 48975d9 to ddf724e Compare December 13, 2024 16:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(sql): Optimize executions #4804

feat(sql): Optimize executions #4804

kirangodishala commented Nov 24, 2024 •

edited

Loading

jasonmcintosh Dec 3, 2024

jasonmcintosh Dec 3, 2024

jasonmcintosh Dec 3, 2024

dbyron-sf Dec 3, 2024

jasonmcintosh Dec 4, 2024

dbyron-sf Dec 4, 2024

dbyron-sf Dec 4, 2024

jasonmcintosh Dec 9, 2024

kirangodishala Dec 10, 2024

dbyron-sf Dec 11, 2024

jasonmcintosh Dec 13, 2024

christosarvanitis commented Dec 11, 2024

dbyron-sf Dec 11, 2024

dbyron-sf Dec 11, 2024

dbyron-sf Dec 11, 2024

	log.info("getting stage information for all the executions found so far")
	log.debug("getting stage information for all the executions found so far")

	log.info("unexpanding pipeline executions")
	log.debug("unexpanding pipeline executions")

	log.info("filtering pipelines by history")
	log.debug("filtering pipelines by history")

feat(sql): Optimize executions #4804

Are you sure you want to change the base?

feat(sql): Optimize executions #4804

Conversation

kirangodishala commented Nov 24, 2024 • edited Loading

Breaking changes:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christosarvanitis commented Dec 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kirangodishala commented Nov 24, 2024 •

edited

Loading