Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Add cluster idle HTTP api (backport #53850) #54010

Closed
wants to merge 1 commit into from

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Dec 17, 2024

Why I'm doing:

Add cluster idle api to help judge the cluster status.

What I'm doing:

Add a new daemon thread WarehouseIdleChecker to count all the tasks being executed and the time when the last task ended. If all the tasks being executed are 0 and the time when the last task ended has exceeded Config.warehouse_idle_check_interval_seconds, the system is considered idle.

The entry point for all SQL execution in the system is StmtExecutor. If it is synchronous SQL, we only need to count the SQL executed in StmtExecutor. If it is asynchronous SQL, such as broker load, we also need to count these asynchronously executed tasks: Stream Load, Broker Load, Spark Load, Routine Load, Backup/Restore, Schema Change.

In addition, for SQL executed within the system, such as statistics collection, we do not need to count them. Add the isInternal field in StmtExecutor to determine whether the SQL is initiated internally or by the user.

api request
/api/idle_status

api response

{
    "isClusterIdle": false,
    "clusterIdleTime": -1,
    "warehouses": [
        {
            "id": 0,
            "name": "default_warehouse",
            "isIdle": false,
            "idleTime": -1
        }
    ]
}

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
  • This is a backport pr

(cherry picked from commit 6cd9fbc)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/alter/LakeTableSchemaChangeJob.java
#	fe/fe-core/src/main/java/com/starrocks/alter/OnlineOptimizeJobV2.java
#	fe/fe-core/src/main/java/com/starrocks/alter/RollupJobV2.java
#	fe/fe-core/src/main/java/com/starrocks/alter/SchemaChangeJobV2.java
#	fe/fe-core/src/main/java/com/starrocks/backup/BackupJob.java
#	fe/fe-core/src/main/java/com/starrocks/backup/RestoreJob.java
#	fe/fe-core/src/main/java/com/starrocks/common/Config.java
#	fe/fe-core/src/main/java/com/starrocks/connector/metadata/MetadataExecutor.java
#	fe/fe-core/src/main/java/com/starrocks/datacache/DataCacheSelectExecutor.java
#	fe/fe-core/src/main/java/com/starrocks/load/routineload/RoutineLoadJob.java
#	fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadMgr.java
#	fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadTask.java
#	fe/fe-core/src/main/java/com/starrocks/qe/ConnectContext.java
#	fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java
#	fe/fe-core/src/main/java/com/starrocks/qe/feedback/PlanAdvisorExecutor.java
#	fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java
#	fe/fe-core/src/main/java/com/starrocks/server/WarehouseManager.java
#	fe/fe-core/src/main/java/com/starrocks/sql/util/CustomizedQueryExecutor.java
#	fe/fe-core/src/main/java/com/starrocks/statistic/HyperStatisticsCollectJob.java
#	fe/fe-core/src/test/java/com/starrocks/scheduler/MVRefreshTestBase.java
#	fe/fe-core/src/test/java/com/starrocks/sql/optimizer/rule/transformation/materialization/MvRewriteTestBase.java
@mergify mergify bot added the conflicts label Dec 17, 2024
Copy link
Contributor Author

mergify bot commented Dec 17, 2024

Cherry-pick of 6cd9fbc has failed:

On branch mergify/bp/branch-3.2/pr-53850
Your branch is up to date with 'origin/branch-3.2'.

You are currently cherry-picking commit 6cd9fbc95f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   fe/fe-core/src/main/java/com/starrocks/alter/AlterHandler.java
	modified:   fe/fe-core/src/main/java/com/starrocks/alter/AlterJobMgr.java
	modified:   fe/fe-core/src/main/java/com/starrocks/alter/AlterJobV2.java
	modified:   fe/fe-core/src/main/java/com/starrocks/backup/BackupHandler.java
	modified:   fe/fe-core/src/main/java/com/starrocks/http/HttpServer.java
	new file:   fe/fe-core/src/main/java/com/starrocks/http/rest/IdleAction.java
	modified:   fe/fe-core/src/main/java/com/starrocks/load/loadv2/BrokerLoadJob.java
	modified:   fe/fe-core/src/main/java/com/starrocks/load/loadv2/LoadMgr.java
	modified:   fe/fe-core/src/main/java/com/starrocks/load/loadv2/SparkLoadJob.java
	modified:   fe/fe-core/src/main/java/com/starrocks/load/pipe/filelist/RepoExecutor.java
	modified:   fe/fe-core/src/main/java/com/starrocks/load/routineload/RoutineLoadMgr.java
	modified:   fe/fe-core/src/main/java/com/starrocks/scheduler/PartitionBasedMvRefreshProcessor.java
	modified:   fe/fe-core/src/main/java/com/starrocks/scheduler/SqlTaskRunProcessor.java
	modified:   fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunExecutor.java
	modified:   fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunFIFOQueue.java
	modified:   fe/fe-core/src/main/java/com/starrocks/scheduler/TaskRunScheduler.java
	modified:   fe/fe-core/src/main/java/com/starrocks/sql/ast/UserVariable.java
	modified:   fe/fe-core/src/main/java/com/starrocks/statistic/ExternalFullStatisticsCollectJob.java
	modified:   fe/fe-core/src/main/java/com/starrocks/statistic/FullStatisticsCollectJob.java
	modified:   fe/fe-core/src/main/java/com/starrocks/statistic/StatisticExecutor.java
	modified:   fe/fe-core/src/main/java/com/starrocks/statistic/StatisticsCollectJob.java
	new file:   fe/fe-core/src/main/java/com/starrocks/warehouse/IdleStatus.java
	new file:   fe/fe-core/src/main/java/com/starrocks/warehouse/WarehouseIdleChecker.java
	modified:   fe/fe-core/src/test/java/com/starrocks/sql/plan/PlanTestNoneDBBase.java
	new file:   test/sql/test_warehouse/R/test_idle_status
	new file:   test/sql/test_warehouse/T/test_idle_status

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   fe/fe-core/src/main/java/com/starrocks/alter/LakeTableSchemaChangeJob.java
	deleted by us:   fe/fe-core/src/main/java/com/starrocks/alter/OnlineOptimizeJobV2.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/alter/RollupJobV2.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/alter/SchemaChangeJobV2.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/backup/BackupJob.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/backup/RestoreJob.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/common/Config.java
	deleted by us:   fe/fe-core/src/main/java/com/starrocks/connector/metadata/MetadataExecutor.java
	deleted by us:   fe/fe-core/src/main/java/com/starrocks/datacache/DataCacheSelectExecutor.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/load/routineload/RoutineLoadJob.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadMgr.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/load/streamload/StreamLoadTask.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/qe/ConnectContext.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java
	deleted by us:   fe/fe-core/src/main/java/com/starrocks/qe/feedback/PlanAdvisorExecutor.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/server/WarehouseManager.java
	deleted by them: fe/fe-core/src/main/java/com/starrocks/sql/util/CustomizedQueryExecutor.java
	deleted by us:   fe/fe-core/src/main/java/com/starrocks/statistic/HyperStatisticsCollectJob.java
	both modified:   fe/fe-core/src/test/java/com/starrocks/scheduler/MVRefreshTestBase.java
	both modified:   fe/fe-core/src/test/java/com/starrocks/sql/optimizer/rule/transformation/materialization/MvRewriteTestBase.java

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Copy link
Contributor Author

mergify bot commented Dec 17, 2024

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant