Skip to content

REST APIs

Jump to bottom
Hao Geng edited this page Apr 28, 2022 · 55 revisions

Contents

Asynchronous Endpoints

Please ensure proper use of UUIDs or cookies to interact with async endpoints.

Selected Cruise Control (CC) endpoints accept async calls to avoid blocking more than a configured period of time (via webserver.request.maxBlockTimeMs configuration). If the server-side processing of such requests takes more than this configured time, along with a with SC_ACCEPTED code, CC returns a progress response. This response contains both (1) a sessionId, and (2) a UUID corresponding to the client cookie and the request, respectively. The completed response of an in-progress request can be retrieved within a predefined time. Using cookies, this response can be retrieved before the timeout is configured via webserver.session.maxExpiryTimeMs. Using the UUID, the timeout is configured via relevant completed user task retention time configuration (see completed.kafka.monitor.user.task.retention.time.ms, completed.cruise.control.monitor.user.task.retention.time.ms, completed.kafka.admin.user.task.retention.time.ms, completed.cruise.control.admin.user.task.retention.time.ms, and completed.user.task.retention.time.ms ).

UUIDs

Here is an example how to use UUID with requests using cURL:

  1. Create a new request

curl -vv -X POST "http://CRUISE_CONTROL_HOST:9090/kafkacruisecontrol/remove_broker?brokerid=1234&dryrun=false"

  1. Retrieve the User-Task-ID from response header, e.g. User-Task-ID: 5ce7c299-53b3-48b6-b72e-6623e25bd9a8

  2. Specifying the User-Task-ID in request that has not completed

curl -vv -X POST -H "User-Task-ID: 5ce7c299-53b3-48b6-b72e-6623e25bd9a8" "http://CRUISE_CONTROL_HOST:9090/kafkacruisecontrol/remove_broker?brokerid=1234&dryrun=false"

Cookies

Here is an example of how to use cookies with requests using cURL:

  1. Create a cookie associated with a new request

curl -X POST -c /tmp/mycookie-jar.txt "http://CRUISE_CONTROL_HOST:9090/kafkacruisecontrol/remove_broker?brokerid=1234&dryrun=false"

  1. Use an existing cookie from the created file for a request that has not completed

curl -X POST -b /tmp/mycookie-jar.txt "http://CRUISE_CONTROL_HOST:9090/kafkacruisecontrol/remove_broker?brokerid=1234&dryrun=false"

Note that a User-Task-ID or a sessionId and is applicable for an entire URL, including its parameters. Hence, the same endpoint with different parameters would create and use a different User-Task-Id.

GET Requests

The GET requests in Kafka Cruise Control REST API are for read only operations, i.e. the operations that do not have any external impacts. The GET requests include the following operations:

Query the state of Cruise Control

User can query the state of Kafka Cruise Control at any time by issuing a HTTP GET request.

GET /kafkacruisecontrol/state

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
substates list substates for which to retrieve state from cruise-control, available substates are analyzer, monitor, executor and anomaly_detector all substates yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
super_verbose boolean return more detailed state information false yes
doAs string propagated user by the trusted proxy service null yes
reason string reason for the request "No reason provided" yes

The returned state contains the following information:

  • Monitor State:
    • State: NOT_STARTED / RUNNING / SAMPLING / PAUSED / BOOTSTRAPPING / TRAINING / LOADING,
    • Bootstrapping progress (If state is BOOTSTRAPPING)
    • Number of valid monitored windows / number of total monitored windows
    • Number of valid partitions out of the total number of partitions
    • Percentage of the partitions that are valid
    • Time and reason of recently sampling task pause/resume
  • Executor State:
    • State: NO_TASK_IN_PROGRESS / STARTING_EXECUTION / INTER_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS / INTRA_BROKER_REPLICA_MOVEMENT_TASK_IN_PROGRESS / LEADER_MOVEMENT_TASK_IN_PROGRESS / STOPPING_EXECUTION / INITIALIZING_PROPOSAL_EXECUTION / GENERATING_PROPOSALS_FOR_EXECUTION
    • Inter-broker replica movement progress (if state is INTER_BROKER_REPLICA_MOVEMENT_IN_PROGRESS)
    • Intra-broker replica movement progress (if state is INTRA_BROKER_REPLICA_MOVEMENT_IN_PROGRESS)
    • Leadership movement progress (if state is LEADERSHIP_MOVEMENT)
    • movement concurrency
    • UUID triggers the execution
  • Analyzer State:
    • isProposalReady: Is there a proposal cached
    • ReadyGoals: A list of goals that are ready for running
  • Anomaly Detector State:
    • selfHealingEnabled: Anomaly type for which self healing is enabled
    • selfHealingDisabled: Anomaly type for which self healing is disabled
    • recentGoalViolations: Recently detected goal violations
    • recentBrokerFailures: Recently detected broker failures
    • recentDiskFailures: Recently detected disk failures
    • recentMetricAnomalies: Recently detected metric anomalies

If verbose is set to true, the details about monitored windows and goals will be displayed. If super_verbose is set to true, the details about extrapolation made on metric samples will be displayed. If substates is not set, the full state will be displayed; if it is set to the specific substate(s), only state(s) of interest will be displayed and response will be returned faster.

Query the current cluster load

Once Cruise Control Load Monitor shows it is in the RUNNING state, Users can use the following HTTP GET to get the cluster load:

GET /kafkacruisecontrol/load

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
start long start time of the cluster load time of earliest valid window yes
end long end time of the cluster load current system time yes
time long end time of the cluster load current system time yes
allow_capacity_estimation boolean whether to allow broker capacity to be estimated from other broker in the cluster true yes
populate_disk_info boolean whether show the load of each disk of broker false yes
capacity_only boolean whether show only the cluster capacity or the utilization, as well. false yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
doAs string propagated user by the trusted proxy service null yes
reason string reason for the request "No reason provided" yes

If the number of workload snapshots for the given timestamp is not sufficient to generate a good load model, an exception will be returned.

Timestamp for start/end/time is in milliseconds since the epoch; what System.currentTimeMillis() returns. The time zone is the time zone of the Cruise Control server.

If allow_capacity_estimation is set to true, for brokers missing capacity information Cruise Control will make estimations based on other brokers in the cluster; otherwise an IllegalStateException will be thrown and shown in response.

The response contains both load-per-broker and load-per-host information. This is specifically useful when multiple brokers are hosted by the same machine.

NOTE: The load shown is only for the load from the valid partitions. i.e the partitions with enough metric samples. So please always check the LoadMonitor's state(via State endpoint) to decide whether the workload is representative enough.

Query partition resource utilization

The following GET request gives the partition load sorted by the utilization of a given resource:

GET /kafkacruisecontrol/partition_load

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
resource string resource type to sort partition load, available resources are DISK/CPU/NW_IN/NW_OUT DISK yes
start long the timestamp in millisecond of the earliest metric sample use to generate load time of earliest valid window yes
end long the timestamp in millisecond of the latest metric sample use to generate load current system time yes
entries integer number of partition load entries to report in response MAX_INT yes
json boolean return in JSON format or not false yes
allow_capacity_estimation boolean whether to allow capacity estimation when cruise-control is unable to obtain all per-broker capacity information true yes
max_load boolean whether report the max load for partition in windows false yes
avg_load boolean whether report the average load for partition in windows false yes
topic regex regular expression to filter partition load to report based on partition's topic null yes
partition integer/range partition number(e.g. 10) range(e.g. 1-10) to filter partition load to report null yes
min_valid_partition_ratio double minimal valid partition ratio requirement for cluster model null yes
brokerid int broker id to to filter partition load to report null yes
doAs string propagated user by the trusted proxy service null yes
reason string reason for the request "No reason provided" yes

The returned result would be a partition list sorted by the utilization of the specified resource in the time range specified by start and end. The resource can be CPU, NW_IN, NW_OUT and DISK. By default, the start is the earliest monitored time, the end is current wall clock time, resource is DISK, and entries is the all partitions in the cluster.

By specifying topic,partition and/or brokerid parameter, client can filter returned partition entries.

The min_valid_partition_ratio specifies minimal monitored valid partition percentage needed to calculate the partition load. If this parameter is not set in request, the config value min.valid.partition.ratio will be used.

The max_load parameter specifies whether report the maximal historical value or not. The avg_load parameter specifies whether report the average historical value or not. If both are not specified or specified as false, for DISK resource, latest value will be reported; for NW_IN/NW_OUT/CPU resource, average value will be reported.

Query partition and replica state

The following GET request gives partition healthiness on the cluster:

GET /kafkacruisecontrol/kafka_cluster_state

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
topic regex regular expression to filter partition state to report based on partition's topic null yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
doAs string propagated user by the trusted proxy service null yes
reason string reason for the request "No reason provided" yes

The returned result contains the following information

  • For each broker

    • Distribution of leader/follower/out-of-sync/offline replica information
    • Online/offline disks

  • For each partition

    • Distribution of leader/follower/in-sync/out-of-sync/offline replica information

Get optimization proposals

The following GET request returns the optimization proposals generated based on the workload model of the given timestamp. The workload summary before and after the optimization will also be returned.

GET /kafkacruisecontrol/proposals

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
ignore_proposal_cache boolean whether to ignore the cached proposal or not false yes
data_from string whether to calculate proposal from available valid partitions or valid windows VALID_WINDOWS yes
goals list list of goals used to generate proposal default goals yes
kafka_assigner boolean whether to use Kafka assigner mode to generate proposals false yes
allow_capacity_estimation boolean whether to allow broker capacity to be estimated true yes
excluded_topics regex regular expression to specify topics excluded from replica and leadership movement null yes
use_ready_default_goals boolean whether to use only ready goals to generate proposals false yes
exclude_recently_demoted_brokers boolean whether to allow leader replicas to be moved to recently demoted brokers false yes
exclude_recently_removed_brokers boolean whether allow replicas to be moved to recently removed broker false yes
destination_broker_ids list specify brokers to move replicas to available brokers yes
rebalance_disk boolean whether to balance load between disks within brokers (requires JBOD Kafka deployment) false yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
doAs string propagated user by the trusted proxy service null yes
fast_mode boolean true to compute proposals in fast mode, false otherwise true yes
reason string reason for the request "No reason provided" yes

Proposal can be generated based on valid_window or valid_partitions.

valid_windows: rebalance the cluster based on the information in the available valid snapshot windows. A valid snapshot window is a windows whose valid monitored partitions coverage meets the requirements of all the goals. (This is the default behavior)

valid_partitions: rebalance the cluster based on all the available valid partitions. All the snapshot windows will be included in this case.

Users can only specify either valid_windows or valid_partitions, but not both.

Kafka cruise control tries to precompute the optimization proposal in the background and caches the best proposal to serve when user queries. If users want to have a fresh proposal without reading it from the proposal cache, set the ignore_proposal_cache flag to true. The precomputing always uses available valid partitions to generate the proposals.

By default the proposal will be returned from the cache where all the pre-defined goals are used. Detailed information about the reliability of the proposals will also be returned. If users want to run with a different set of goals, they can specify the goals parameter with the goal names (simple class name).

If verbose is turned on, Cruise Control will return all the generated proposals. Otherwise a summary of the proposals will be returned.

If kafka_assigner is turned on, the proposals will be generated in Kafka Assigner mode,i.e. Cruise Control behaves like kafka assigner tool. This mode performs optimizations using the goals specific to Kafka Assigner -- i.e. goals with name KafkaAssigner*.

Users can specify excluded_topics to prevent certain topics' replicas from moving in the generated proposals.

If use_ready_default_goals is turned on, Cruise Control will use whatever ready goals(based on available metric data) to calculate the proposals.

Query the user request result

The following get request allows user to get a full list of all the active/completed(and not recycled) tasks inside Cruise Control, with their initial request detail(request time/IP address/request URL and parameter) and UUID information. User can then use the returned UUID and URL to fetch the original final result of the specific request.

GET /kafkacruisecontrol/user_tasks

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
user_task_ids list comma separated UUIDs to filter the task results Cruise Control report all tasks yes
client_ids list comma separated IP addresses to filter the task results Cruise Control report all users yes
entries integer number of partition load entries to report in response MAX_INT yes
endpoints list comma separated endpoints to filter the task results Cruise Control report all endpoints yes
types string comma separated HTTP request types to filter the task results Cruise Control report all request types yes
json boolean return in JSON format or not false yes
fetch_completed_task boolean whether return the original request's final response false yes
doAs string propagated user by the trusted proxy service null yes
reason string reason for the request "No reason provided" yes

User can use user_task_ids/client_ids/endpoints/types make Cruise Control only return requests they are interested. By default all the requests get returned.

If fetch_completed_task is set to true, the original response of each request will be returned. In the case where a task completed with errors the response will be CompletedWithError.

POST Requests

The post requests of Kafka Cruise Control REST API are operations that will have impact on the Kafka cluster. The post operations include:

Most of the POST actions has a dry-run mode, which only generate the proposals and estimated result but not really execute the proposals. To avoid accidentally triggering of data movement, by default all the POST actions are in dry run mode. To let Kafka Cruise Control actually move data, users need to explicitly set dryrun=false.

Trigger a workload balance

The following POST request will let Kafka Cruise Control rebalance a Kafka cluster

POST /kafkacruisecontrol/rebalance

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
dryrun boolean whether dry-run the request or not true yes
data_from string whether generate proposal from available valid partitions or valid windows VALID_WINDOW yes
goals list list of goals used to generate proposal default goals yes
kafka_assigner boolean whether to use Kafka assigner mode to generate proposals false yes
allow_capacity_estimation boolean whether to allow broker capacity to be estimated true yes
concurrent_partition_movements_per_broker integer upper bound of ongoing replica movements going into/out of each broker null yes
max_partition_movements_in_cluster integer upper bound of maximum replica movements going into/out of brokers across cluster default_config yes
concurrent_intra_broker_partition_movements integer upper bound of ongoing replica movements between disks within each broker null yes
concurrent_leader_movements integer upper bound of ongoing leadership movements null yes
skip_hard_goal_check boolean Whether allow hard goals be skipped in proposal generation false yes
excluded_topics regex regular expression to specify topics excluded from replica and leadership movement null yes
use_ready_default_goals boolean whether only use ready goals to generate proposal false yes
exclude_recently_demoted_brokers boolean whether to allow leader replicas to be moved to recently demoted brokers false yes
exclude_recently_removed_brokers boolean whether allow replicas to be moved to recently removed broker false yes
replica_movement_strategies string replica movement strategy to use null yes
ignore_proposal_cache boolean whether to ignore the cached proposal or not false yes
replication_throttle long upper bound on the bandwidth used to move replicas (in bytes per second) null yes
destination_broker_ids list specify brokers to move replicas to available brokers yes
rebalance_disk boolean whether to balance load between disks within brokers (requires JBOD Kafka deployment) false yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
reason string reason for the request "No reason provided" yes
doAs string propagated user by the trusted proxy service null yes
fast_mode boolean true to compute proposals in fast mode, false otherwise true yes

Similar to the GET interface for getting proposals, the rebalance can also be based on available valid windows or available valid partitions.

User can specify goals to use for rebalance via goals parameter. When goals is provided, the cached proposals will be ignored.

When rebalancing a cluster, all the brokers in the cluster(except recently removed/demoted brokers if exclude_recently_removed_brokers/exclude_recently_demoted_brokers set to true) are eligible to give / receive replicas. All the brokers will be throttled during the partition movement. The throttling can be set in two ways.

  • By throttling number of replica concurrently moving into a broker. It is gated by the request parameter concurrent_partition_movements_per_broker or config value num.concurrent.partition.movements.per.broker (if request parameter is not set), concurrent_intra_partition_movements or config value num.concurrent.intra.broker.partition.movements. Similarly, the number of concurrent partition leadership changes is gated by request parameter concurrent_leader_movements or config value num.concurrent.leader.movements.
  • By throttling the bandwidth used to move replicas into a broker. It is gated by the request parameter replication_throttle or config value default.replication.throttle (if request parameter is not set).

Add a list of new brokers to Kafka Cluster

The following POST request adds the given brokers to the Kafka cluster

POST /kafkacruisecontrol/add_broker?brokerid=[id1,id2...]

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
brokerid list list of ids of new broker added to the cluster N/A no
dryrun boolean whether dry-run the request or not true yes
data_from string whether generate proposal from available valid partitions or valid windows VALID_WINDOWS yes
goals list list of goals used to generate proposal default goals yes
kafka_assigner boolean whether to use Kafka assigner mode to generate proposals false yes
allow_capacity_estimation boolean whether to allow broker capacity to be estimated from other broker in the cluster true yes
concurrent_partition_movements_per_broker integer upper bound of ongoing replica movements going into/out of each broker null yes
max_partition_movements_in_cluster integer upper bound of maximum replica movements going into/out of brokers across cluster default_config yes
concurrent_leader_movements integer upper bound of ongoing leadership movements null yes
skip_hard_goal_check boolean whether allow hard goals be skipped in proposal generation false yes
excluded_topics regex regular expression to specify topics excluded from replica and leadership movement null yes
use_ready_default_goals boolean whether only use ready goals to generate proposal false yes
exclude_recently_demoted_brokers boolean whether to allow leader replicas to be moved to recently demoted brokers false yes
exclude_recently_removed_brokers boolean whether allow replicas to be moved to recently removed broker false yes
replica_movement_strategies string replica movement strategy to use null yes
replication_throttle long Upper bound on the bandwidth used to move replicas (in bytes per second) null yes
throttle_added_broker boolean whether throttle replica movement to new broker or not true yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
reason string reason for the request "No reason provided" yes
doAs string propagated user by the trusted proxy service null yes
fast_mode boolean true to compute proposals in fast mode, false otherwise true yes

When adding new brokers to a Kafka cluster, Cruise Control makes sure that the replicas will only be moved from the existing brokers to the provided new broker, but not moved among existing brokers.

Users can choose whether to throttle replica movement to the newly added broker via throttle_added_broker, in either case, the replica movement out of current broker are throttled, and the throttling can be set in the same way as rebalance request.

Decommission a list of brokers from the Kafka cluster

The following POST request removes a list of brokers from the Kafka cluster:

POST /kafkacruisecontrol/remove_broker?brokerid=[id1,id2...]

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
brokerid list list of ids of broker to be removed from the cluster N/A no
dryrun boolean whether dry-run the request or not true yes
data_from string specify either generate proposal from available valid partitions or valid windows VALID_WINDOW yes
goals list list of goals used to generate proposal default goals yes
kafka_assigner boolean whether to use Kafka assigner mode to generate proposals false yes
allow_capacity_estimation boolean whether to allow broker capacity to be estimated from other brokers in the cluster true yes
concurrent_partition_movements_per_broker integer upper bound of ongoing replica movements going into/out of each broker null yes
max_partition_movements_in_cluster integer upper bound of maximum replica movements going into/out of brokers across cluster default_config yes
concurrent_leader_movements integer upper bound of ongoing leadership movements null yes
skip_hard_goal_check boolean whether allow hard goals be skipped in proposal generation false yes
excluded_topics regex regular expression to specify topics excluded from replica and leadership movement null yes
use_ready_default_goals boolean whether only use ready goals to generate proposal false yes
exclude_recently_demoted_brokers boolean whether to allow leader replicas to be moved to recently demoted brokers false yes
exclude_recently_removed_brokers boolean whether allow replicas to be moved to recently removed broker false yes
replica_movement_strategies string replica movement strategy to use null yes
replication_throttle long upper bound on the bandwidth used to move replicas (in bytes per second) false yes
throttle_removed_broker boolean whether throttle replica movement out of the removed broker or not true yes
destination_broker_ids list specify brokers to move replicas to available brokers yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
reason string reason for the request "No reason provided" yes
doAs string propagated user by the trusted proxy service null yes
fast_mode boolean true to compute proposals in fast mode, false otherwise true yes

Similar to adding brokers to a cluster, removing brokers from a cluster will only move partitions from the brokers to be removed to the other existing brokers. There won't be partition movements among remaining brokers. And user can specify the destination broker for these replica movement via destination_broker_ids parameter.

Like adding brokers, users can choose whether to throttle the removed broker during the partition movement. If the removed brokers are throttled, the number of partitions concurrently moving out of a broker, number of concurrent partition leadership changes and bandwidth used for replica movement are gated in the same way as add_broker above.

Note if the topics specified in excluded_topics has replicas on the removed broker, the replicas will still get moved off the broker.

Fix offline replicas in Kafka cluster

The following POST request moves all the offline replicas from dead disks/brokers. This endpoint is not available in kafka_0_11_and_1_0 branch.

POST /kafkacruisecontrol/fix_offline_replicas

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
dryrun boolean whether dry-run the request or not true yes
data_from string specify either generate proposal from available valid partitions or valid windows VALID_WINDOW yes
goals list list of goals used to generate proposal default goals yes
kafka_assigner boolean whether to use Kafka assigner mode to generate proposals false yes
allow_capacity_estimation boolean whether to allow broker capacity to be estimated from other brokers in the cluster true yes
concurrent_partition_movements_per_broker integer upper bound of ongoing replica movements going into/out of each broker null yes
max_partition_movements_in_cluster integer upper bound of maximum replica movements going into/out of brokers across cluster default_config yes
concurrent_leader_movements integer upper bound of ongoing leadership movements null yes
skip_hard_goal_check boolean Whether allow hard goals be skipped in proposal generation false yes
excluded_topics regex regular expression to specify topics excluded from replica and leadership movement null yes
use_ready_default_goals boolean whether only use ready goals to generate proposal false yes
exclude_recently_demoted_brokers boolean whether to allow leader replicas to be moved to recently demoted brokers false yes
exclude_recently_removed_brokers boolean whether allow replicas to be moved to recently removed broker false yes
replica_movement_strategies string replica movement strategy to use null yes
replication_throttle long upper bound on the bandwidth used to move replicas (in bytes per second) null yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
reason string reason for the request "No reason provided" yes
doAs string propagated user by the trusted proxy service null yes
fast_mode boolean true to compute proposals in fast mode, false otherwise true yes

Likewise, users can throttle partition movement, the throttling can be set in the same way as rebalance request.

Note if the topics specified in excluded_topics has offline replicas, the replicas will still get moved to healthy brokers.

Demote a list of brokers from the Kafka cluster

The following POST request moves all the leader replicas away from a list of brokers.

POST /kafkacruisecontrol/demote_broker?brokerid=[id1, id2...]

User can also request to move all the leader replicas away from the a list of disks via

 POST /kafkacruisecontrol/demote_broker?brokerid_and_logdirs=[id1-logdir1, id2-logdir2...]

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
brokerid list list of ids of broker to be demoted in the cluster N/A no if brokerid_and_logdirs is not set
brokerid_and_logdirs list list of broker id and logdir pair to be demoted in the cluster N/A no if brokerids is not set
dryrun boolean whether dry-run the request or not true yes
allow_capacity_estimation boolean whether to allow broker capacity to be estimated true yes
concurrent_leader_movements integer upper bound of ongoing leadership movements null yes
skip_urp_demotion boolean whether skip demoting leader replicas for under replicated partitions true yes
exclude_follower_demotion boolean whether skip demoting follower replicas on the broker to be demoted true yes
exclude_recently_demoted_brokers boolean whether to allow leader replicas to be moved to recently demoted brokers false yes
replica_movement_strategies string replica movement strategy to use null yes
replication_throttle long upper bound on the bandwidth used to move replicas (in bytes per second) null yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
reason string reason for the request "No reason provided" yes
doAs string propagated user by the trusted proxy service null yes

Demoting a broker/disk is consist of tow steps.

  • Make all the replicas on given broker/disk the least preferred replicas for leadership election within their corresponding partitions
  • Trigger a preferred leader election on the partitions to migrate the leader replicas off the broker/disk

Set skip_urp_demotion to false will cancel outstanding operations if partitions stay under replicated; Set exclude_follower_demotion will skip operations on the partitions which only have follower replicas on the brokers/disks to be demoted. The purpose of the former is to prevent the URP recovery process from blocking the demotion execution, the latter ensures that the demotion operation is limited to leaders.

Stop the current proposal execution task

The following POST request will let Kafka Cruise Control stop an ongoing rebalance, add_broker, remove_broker, fix_offline_replica, topic_configuration or demote_broker operation:

POST /kafkacruisecontrol/stop_proposal_execution

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
force_stop boolean (not supported in Kafka 2.4 or above) stop an ongoing execution forcefully by deleting Kafka internal zNodes false yes
stop_external_agent boolean (required Kafka 2.4 or above) stop an ongoing execution even if it is started by an external agent true yes
review_id integer review id for 2-step verification N/A yes
json boolean return in JSON format or not false yes
get_response_schema boolean return JSON schema in response header or not false yes
doAs string propagated user by the trusted proxy service null yes
reason string reason for the request "No reason provided" yes

Note that Cruise Control does not wait for the ongoing batch to finish when it stops execution, i.e. the in-progress batch may still be running after Cruise Control stops the execution.

The stop_external_agent parameter is a no-op for CC versions before 2.4.73 and 2.5.65.

Pause metrics load sampling

The following POST request will let Kafka Cruise Control pause an ongoing metrics sampling process:

POST /kafkacruisecontrol/pause_sampling

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
reason string reason to pause sampling "No reason provided" yes
json boolean return in JSON format or not false yes
doAs string propagated user by the trusted proxy service null yes

The reason to pause sampling will be recorded and shows up in state endpoint(under LoadMonitor sub state).

Resume metrics load sampling

The following POST request will let Kafka Cruise Control resume a paused metrics sampling process:

POST /kafkacruisecontrol/resume_sampling
PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
reason string reason to resume sampling "No reason provided" yes
json boolean return in JSON format or not false yes
doAs string propagated user by the trusted proxy service null yes

The reason to resume sampling will be recorded and shows up in state endpoint(under LoadMonitor sub state).

Change Kafka topic configuration

Currently Cruise Control only supports changing topic's replication factor via topic_configuration endpoint. Ultimately we want make Cruise Control the central place to change any topic configurations(partition count, retention time etc.).

The following POST request can change topic's replication factor.

POST /kafkacruisecontrol/topic_configuration?topic=[topic_regex]&replication_factor=[target_replication_factor]

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
dryrun boolean whether dry-run the request or not true yes
topic regex regular expression to specify subject topics N/A no
replication_factor Integer target replication factor N/A no
data_from string whether generate proposal from available valid partitions or valid windows VALID_WINDOW yes
goals list list of goals used to generate proposal default goals yes
allow_capacity_estimation boolean whether to allow broker capacity to be estimated true yes
concurrent_partition_movements_per_broker integer upper bound of ongoing replica movements going into/out of each broker null yes
max_partition_movements_in_cluster integer upper bound of maximum replica movements going into/out of brokers across cluster default_config yes
concurrent_leader_movements integer upper bound of ongoing leadership movements null yes
skip_hard_goal_check boolean Whether allow hard goals be skipped in proposal generation false yes
exclude_recently_demoted_brokers Boolean Whether to allow leader replicas to be moved to recently demoted brokers false yes
exclude_recently_removed_brokers Boolean Whether allow replicas to be moved to recently removed broker false yes
replica_movement_strategies string replica movement strategy to use null yes
replication_throttle long upper bound on the bandwidth used to move replicas (in bytes per second) null yes
json boolean return in JSON format or not false yes
verbose boolean return detailed state information false yes
reason string reason for the request "No reason provided" yes
doAs string propagated user by the trusted proxy service null yes
fast_mode boolean true to compute proposals in fast mode, false otherwise true yes

Changing topic's replication factor will not move any existing replicas. goals are used to determine which replica to be deleted(to decrease topic's replication factor) and which broker to assign new replica (to increase topic's replication factor).

Note sometimes the topic regex can be too long to put at POST request head, in this case user can specify topic regex and target replication factor pairs in POST request body. For details, check Change-topic-replication-factor-through-Cruise-Control wiki page.

Change Cruise Control configuration

Some Cruise Control configs can be changed dynamically via admin endpoint, which includes

  • Dynamically change the partition and leadership concurrency and the interval between checking and updating (if needed) the progress of an ongoing execution.
  • Enable/disable self-healing for the specified anomaly types.
  • Drop selected recently removed/demoted brokers.
  • Enable/disable the specified concurrency adjusters.
  • Enable/disable the (At/Under)MinISR-based concurrency adjustment.

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
disable_self_healing_for list list of anomaly types to disable self-healing N/A yes
enable_self_healing_for list list of anomaly types to enable self-healing N/A yes
concurrent_partition_movements_per_broker integer upper bound of ongoing replica movements into/out of a broker N/A yes
max_partition_movements_in_cluster integer upper bound of max replica movements into/out of brokers across cluster N/A yes
concurrent_intra_broker_partition_movements integer upper bound of ongoing replica movements between disks within a broker N/A yes
concurrent_leader_movements integer upper bound of ongoing leadership movements N/A yes
drop_recently_removed_brokers list list of id of recently removed brokers to be dropped N/A yes
drop_recently_demoted_brokers list list of id of recently demoted brokers to be dropped N/A yes
doAs string propagated user by the trusted proxy service null yes
json boolean return in JSON format or not false yes
review_id integer review id for 2-step verification N/A yes
execution_progress_check_interval_ms long execution progress check interval in milliseconds N/A yes
get_response_schema boolean return JSON schema in response header or not false yes
disable_concurrency_adjuster_for list disable concurrency adjuster for given concurrency types N/A yes
enable_concurrency_adjuster_for list enable concurrency adjuster for given concurrency types N/A yes
min_isr_based_concurrency_adjustment boolean enable (true) or disable (false) MinISR-based concurrency adjustment N/A yes
reason string reason for the request "No reason provided" yes

To Enable/disable self-healing, send POST request like:

 POST /kafkacruisecontrol/admin?disable_self_healing_for=[anomaly_type]

To increase/decrease execution concurrency, send POST request like:

 POST /kafkacruisecontrol/admin?concurrent_partition_movements_per_broker=[integer]

To drop recently removed/demoted brokers, send POST request like:

 POSTs /kafkacruisecontrol/admin?drop_recently_removed_brokers=[broker_ids]

2-step Verification

2-step verification aims to help users verify the command they (or their peers) intend to run by letting them review requests explicitly to approve or discard them, and enable execution of only the approved requests. Read 2 step verification for POST requests for more detail.

Rightsize the cluster with the Provisioner

The following POST request can create a request to the provisioner to rightsize the broker or partition of a cluster.

POST /kafkacruisecontrol/rightsize

Supported parameters are:

PARAMETER TYPE DESCRIPTION DEFAULT OPTIONAL
num_brokers_to_add integer difference in broker count to rightsize towards -1 yes
partition_count integer target number of partitions to rightsize towards -1 yes
topic regex regular expression to specify subject topics null yes
doAs string propagated user by the trusted proxy service null yes
get_response_schema boolean return JSON schema in response header or not false yes
json boolean return in JSON format or not false yes
reason string reason for the request "No reason provided" yes

Contents

Clone this wiki locally