Clarify when changing number of cores is supported #850
Conversation
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
| @@ -133,6 +133,8 @@ Prior to Redpanda version 24.2, this meant that some cores on a broker could ina | |||
|
|
|||
| Starting in v24.2, topic-aware intra-broker partition balancing allows for dynamically reassigning partitions within a broker. Redpanda prioritizes an even distribution of a topic's partition replicas across all cores in a broker. If a broker's core count changes, when the broker starts back up, Redpanda can check partition assignments across the broker's cores and reassign partitions, so that a balanced assignment is maintained across all cores. Redpanda can also check partition assignments when partitions are added to or removed from a broker, and rebalance the remaining partitions between cores. | |||
|
|
|||
| NOTE: Changing the number of CPU cores in a running cluster is supported from v24.2 only, and requires that you enable `node_local_core_assignment` as described in the previous note. | |||
There was a problem hiding this comment.
@daisukebe @wzzzrd86 for this note are we referring to a manual change in number of cores? I am not sure if there is some process that can dynamically change the number and if so, if that's worth mentioning.
There was a problem hiding this comment.
@kbatuigas Yes, this is meant to be a manual change in number of cores. To be clear, the general step is below:
- Stop a broker, which originally uses 10 cores.
- Start the broker with specifying 9 cores (whatever).
This has not been working as expected (broker used to fail to start with decreasing the number of cores) since beginning but this just works today. Does this answer your question?
| @@ -133,6 +133,8 @@ Prior to Redpanda version 24.2, this meant that some cores on a broker could ina | |||
|
|
|||
| Starting in v24.2, topic-aware intra-broker partition balancing allows for dynamically reassigning partitions within a broker. Redpanda prioritizes an even distribution of a topic's partition replicas across all cores in a broker. If a broker's core count changes, when the broker starts back up, Redpanda can check partition assignments across the broker's cores and reassign partitions, so that a balanced assignment is maintained across all cores. Redpanda can also check partition assignments when partitions are added to or removed from a broker, and rebalance the remaining partitions between cores. | |||
|
|
|||
| NOTE: Changing the number of CPU cores in a running cluster is supported from v24.2 only, and requires that you enable `node_local_core_assignment` as described in the previous note. | |||
There was a problem hiding this comment.
Increasing has always been supported (but additional cores were initially idle).
Decreasing has not been supported until 24.2 (with the flag enabled).
Also, since 24.3 the flag will be enabled automatically.
kbatuigas
force-pushed
the
DOC-521-Document-core-change-capability-is-limited
branch
from
480a1a8
to
047809d
Compare
kbatuigas
force-pushed
the
DOC-521-Document-core-change-capability-is-limited
branch
from
047809d
to
72809b5
Compare
|
@daisukebe @wzzzrd86 I was able to confirm with @ztlpn that |
|
I've found other pages that mention "decreasing core count is not supported". We need to update these too @kbatuigas (not sure if you want to open a separate PR, anyway). https://docs.redpanda.com/current/reference/k-redpanda-helm-spec/#resources-cpu-cores |
Good spot @daisukebe thank you! @JakeSCahill , I went ahead and drafted something for the Manage Pod Resources doc, but I'm not sure about interacting with feature flags in K8s, would you be able to double check? Also, Is the helm spec page auto generated? I figured I'd leave it for you to update. |
|
|
||
| In v24.2, you must enable the `node_local_core_assignment` flag to be able to decrease the core count. | ||
|
|
||
| In v24.3 or later, decreasing CPU cores is enabled by default. |
There was a problem hiding this comment.
In v24.3 or later, the `node_local_core_assignment` flag is enabled and decreasing CPU cores is supported for production clusters.
| Redpanda does not support decreasing the CPU cores for brokers in an existing cluster. | ||
| Redpanda supports decreasing the CPU cores for brokers in an existing cluster as of version 24.2. | ||
|
|
||
| In v24.2, you must enable the `node_local_core_assignment` flag to be able to decrease the core count. |
There was a problem hiding this comment.
Can you add a note that decreasing CPU core count is not supported for production clusters?
Looks good. The Helm spec is autogenerated. The change is here: redpanda-data/helm-charts#1601 |
|
|
||
| In v24.2, you must enable the `node_local_core_assignment` flag to be able to decrease the core count. | ||
|
|
||
| In v24.3 or later, decreasing CPU cores is enabled by default. |
There was a problem hiding this comment.
I think we need a separate PR for the beta version of 24.3 docs that removes the limitations section.
We don't expect users to switch off feature flags.
In this PR, all we need is something like this (with links to intra-broker partition balancing and v24.3 docs):
Decreasing the number of CPU cores in a production cluster is not supported in this version of Redpanda. However, you can enable intra-broker partition balancing to reduce CPU cores in a development cluster.
Starting from version 24.3, decreasing the number of CPU cores in a production cluster is supported. Upgrade to version 24.3 or later to access this feature.
| Decreasing the number of CPU cores in a production cluster is not supported in this version of Redpanda. However, you can enable xref:manage:cluster-maintenance/cluster-balancing.adoc#intra-broker-partition-balancing[intra-broker partition balancing] to reduce CPU cores in a development cluster. | ||
|
|
||
| Starting from version 24.3, decreasing the number of CPU cores in a production cluster is supported. xref:24.3@ROOT:upgrade:index.adoc[Upgrade to version 24.3] or later to access this feature. |
There was a problem hiding this comment.
@daisukebe does this phrasing sound good to you?
There was a problem hiding this comment.
Yes, sounds good and correct.
modules/manage/pages/cluster-maintenance/cluster-balancing.adoc
Outdated
Show resolved
Hide resolved
| ==== | ||
|
|
||
| In Redpanda, every partition replica is assigned to a CPU core on a broker. While Redpanda's default <<partition-replica-balancing,partition balancing>> monitors cluster-level events, such as the addition of new brokers or broker failure to balance partition assignments, it does not account for the distribution of partitions _within_ an individual broker. | ||
|
|
||
| Prior to Redpanda version 24.2, this meant that some cores on a broker could inadvertently host many partitions of heavily-used topics and cause the CPU to be xref:manage:monitoring.adoc#cpu-usage[overburdened]. Additionally, when the partition rebalance moved some partitions away from a broker, the remaining partitions did not necessarily rebalance across the broker's cores. Or, if a broker's core count was increased, Redpanda did not assign any partitions to the new cores until new partitions were created or old partitions were moved out. | ||
| Prior to this version, this meant that some cores on a broker could inadvertently host many partitions of heavily-used topics and cause the CPU to be xref:manage:monitoring.adoc#cpu-usage[overburdened]. Additionally, when the partition rebalance moved some partitions away from a broker, the remaining partitions did not necessarily rebalance across the broker's cores. Or, if a broker's core count was increased, Redpanda did not assign any partitions to the new cores until new partitions were created or old partitions were moved out. |
There was a problem hiding this comment.
I think we should remove this. It’s something that should have been explained in release notes rather than being part of the doc. Users don’t typically care about what a previous version used to do unless they’re reading release notes.
Co-authored-by: Jake Cahill <[email protected]>
kbatuigas
deleted the
DOC-521-Document-core-change-capability-is-limited
branch
Description
node_local_core_assignmentis enabled - the same feature flag that enables intra-broker partition balancing.Resolves https://github.com/redpanda-data/documentation-private/issues/2706
Review deadline: 13 Nov
Page previews
Cluster balancing > Intra-broker partition balancing
Checks