diff --git a/modules/reference/pages/properties/broker-properties.adoc b/modules/reference/pages/properties/broker-properties.adoc index 94d39e230..ebcf4fab4 100644 --- a/modules/reference/pages/properties/broker-properties.adoc +++ b/modules/reference/pages/properties/broker-properties.adoc @@ -14,8 +14,6 @@ NOTE: All broker properties require that you restart Redpanda for any update to Network address for the glossterm:Admin API[] server. -*Optional:* Yes - *Visibility:* `user` *Default:* `127.0.0.1:9644` @@ -26,8 +24,6 @@ Network address for the glossterm:Admin API[] server. Path to the API specifications for the Admin API. -*Optional:* Yes - *Visibility:* `user` *Type:* string @@ -40,8 +36,6 @@ Path to the API specifications for the Admin API. Specifies the TLS configuration for the HTTP Admin API. -*Optional:* Yes - *Visibility:* `user` *Default:* `null` @@ -61,9 +55,7 @@ The crash-tracking logic is reset (to zero consecutive crashes) by any of the fo * The `redpanda.yaml` broker configuration file is updated. * The `startup_log` file in the broker's <> is manually deleted. -*Units*: number of consecutive crashes of a broker - -*Optional:* No +*Unit*: number of consecutive crashes of a broker *Visibility:* `user` @@ -79,8 +71,6 @@ The crash-tracking logic is reset (to zero consecutive crashes) by any of the fo Path to the directory for storing Redpanda's streaming data files. -*Optional:* Yes - *Visibility:* `user` *Type:* string @@ -93,9 +83,7 @@ Path to the directory for storing Redpanda's streaming data files. CAUTION: Enabling `developer_mode` isn't recommended for production use. -Flag to enable developer mode, which skips most of the checks performed at startup. - -*Optional:* Yes +Enable developer mode, which skips most of the checks performed at startup. *Visibility:* `tunable` @@ -109,8 +97,6 @@ Flag to enable developer mode, which skips most of the checks performed at start Override the cluster property xref:reference:properties/cluster-properties.adoc#data_transforms_enabled[`data_transforms_enabled`] and disable Wasm-powered data transforms. This is an emergency shutoff button. -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -127,8 +113,6 @@ Controls how a new cluster is formed. All brokers in a cluster must have the sam TIP: For backward compatibility, `true` is the default. Redpanda recommends using `false` in production environments to prevent accidental cluster formation. -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -147,8 +131,6 @@ Controls whether Redpanda starts in FIPS mode. This property allows for three v * Enabled - Redpanda verifies that the operating system is enabled for FIPS by checking `/proc/sys/crypto/fips_enabled`. If the file does not exist or does not return `1`, Redpanda immediately exits. -*Optional:* No - *Visibility:* `user` *Accepted values:* `0` (disabled), `1` (permissive), `2` (enabled) @@ -161,8 +143,6 @@ Controls whether Redpanda starts in FIPS mode. This property allows for three v IP address and port of the Kafka API endpoint that handles requests. -*Optional:* Yes - *Visibility:* `user` *Default:* `127.0.0.1:9092` @@ -173,8 +153,6 @@ IP address and port of the Kafka API endpoint that handles requests. Transport Layer Security (TLS) configuration for the Kafka API endpoint. -*Optional:* Yes - *Visibility:* `user` *Default:* `null` @@ -186,9 +164,6 @@ Transport Layer Security (TLS) configuration for the Kafka API endpoint. Threshold for log messages that contain a larger memory allocation than specified. *Unit:* bytes - -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -203,13 +178,10 @@ A number that uniquely identifies the broker within the cluster. If `null` (the CAUTION: The `node_id` property must not be changed after a broker joins the cluster. - *Accepted values:* [`0`, `4294967295`] *Type:* integer -*Optional:* No - *Visibility:* `user` *Default:* `null` @@ -220,8 +192,6 @@ CAUTION: The `node_id` property must not be changed after a broker joins the clu Path to the configuration file used by OpenSSL to properly load the FIPS-compliant module. -*Optional:* Yes - *Visibility:* `user` *Type:* string @@ -234,8 +204,6 @@ Path to the configuration file used by OpenSSL to properly load the FIPS-complia Path to the directory that contains the OpenSSL FIPS-compliant module. The filename that Redpanda looks for is `fips.so`. -*Optional:* Yes - *Visibility:* `user` *Type:* string @@ -248,8 +216,6 @@ Path to the directory that contains the OpenSSL FIPS-compliant module. The filen A label that identifies a failure zone. Apply the same label to all brokers in the same failure zone. When xref:./cluster-properties.adoc#enable_rack_awareness[enable_rack_awareness] is set to `true` at the cluster level, the system uses the rack labels to spread partition replicas across different failure zones. -*Optional:* No - *Visibility:* `user` *Default:* `null` @@ -260,8 +226,6 @@ A label that identifies a failure zone. Apply the same label to all brokers in t If `true`, start Redpanda in xref:manage:recovery-mode.adoc[recovery mode], where user partitions are not loaded and only administrative operations are allowed. -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -274,8 +238,6 @@ If `true`, start Redpanda in xref:manage:recovery-mode.adoc[recovery mode], wher IP address and port for the Remote Procedure Call (RPC) server. -*Optional:* Yes - *Visibility:* `user` *Default:* `127.0.0.1:33145` @@ -286,8 +248,6 @@ IP address and port for the Remote Procedure Call (RPC) server. TLS configuration for the RPC server. -*Optional:* Yes - *Visibility:* `user` --- @@ -310,8 +270,6 @@ Only one broker, the designated cluster root, should have an empty `seed_servers The `seed_servers` list must be consistent across all seed brokers to prevent cluster fragmentation and ensure stable cluster formation. ==== -*Optional:* Yes - *Visibility:* `user` *Type:* array @@ -324,8 +282,6 @@ The `seed_servers` list must be consistent across all seed brokers to prevent cl Path to the configuration file used for low level storage failure injection. -*Optional:* No - *Visibility:* `tunable` *Type:* string @@ -338,8 +294,6 @@ Path to the configuration file used for low level storage failure injection. If `true`, inject low level storage failures on the write path. Do _not_ use for production instances. -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -352,8 +306,6 @@ If `true`, inject low level storage failures on the write path. Do _not_ use for Whether to violate safety checks when starting a Redpanda version newer than the cluster's consensus version. -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -366,9 +318,7 @@ Whether to violate safety checks when starting a Redpanda version newer than the Maximum duration in seconds for verbose (`TRACE` or `DEBUG`) logging. Values configured above this will be clamped. If null (the default) there is no limit. Can be overridden in the Admin API on a per-request basis. -*Units:* seconds - -*Optional:* No +*Unit:* seconds *Visibility:* `tunable` @@ -388,15 +338,37 @@ The Schema Registry provides configuration properties to help you enable produce For information on how to edit broker properties for the Schema Registry, see xref:manage:cluster-maintenance/node-property-configuration.adoc[]. -=== schema_registry_api +=== api_doc_dir -Specifies the listener address and port in the Schema Registry API. +API doc directory. -*Optional:* Yes +*Visibility:* `user` -*Visibility:* `None` +*Type:* string -*Default:* `127.0.0.1:8081` +*Default:* `/usr/share/redpanda/proxy-api-doc` + +--- + +=== mode_mutability + +Enable modifications to the read-only `mode` of the Schema Registry. When set to `true`, the entire Schema Registry or its subjects can be switched to `READONLY` or `READWRITE`. This property is useful for preventing unwanted changes to the entire Schema Registry or specific subjects. + +*Visibility:* `user` + +*Type:* boolean + +*Default:* `true` + +--- + +=== schema_registry_api + +Schema Registry API listener address and port. + +*Visibility:* `user` + +*Default:* `0.0.0.0:8081` --- @@ -404,9 +376,7 @@ Specifies the listener address and port in the Schema Registry API. TLS configuration for Schema Registry API. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Default:* `null` @@ -416,9 +386,7 @@ TLS configuration for Schema Registry API. Replication factor for internal `_schemas` topic. If unset, defaults to `default_topic_replication`. -*Optional:* No - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -443,39 +411,9 @@ See xref:develop:http-proxy.adoc[] Network address for the HTTP Proxy API server to publish to clients. -*Optional:* Yes - -*Visibility:* `None` - -*Default:* `null` - ---- - -=== api_doc_dir - -Path to the API specifications for the HTTP Proxy API. - -*Optional:* Yes - -*Visibility:* `None` - -*Type:* string - -*Default:* `/usr/share/redpanda/proxy-api-doc` - ---- - -=== mode_mutability - -Enable modifications to the read-only `mode` of the Schema Registry. When set to `true`, the entire Schema Registry or its subjects can be switched to `READONLY` or `READWRITE`. This property is useful for preventing unwanted changes to the entire Schema Registry or specific subjects. - -*Nullable:* No - *Visibility:* `user` -*Type:* boolean - -*Default:* `true` +*Default:* `null` --- @@ -483,9 +421,7 @@ Enable modifications to the read-only `mode` of the Schema Registry. When set to The maximum number of Kafka client connections that Redpanda can cache in the LRU (least recently used) cache. The LRU cache helps optimize resource utilization by keeping the most recently used clients in memory, facilitating quicker reconnections for frequent clients while limiting memory usage. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -497,47 +433,41 @@ The maximum number of Kafka client connections that Redpanda can cache in the LR Time, in milliseconds, that an idle client connection may remain open to the HTTP Proxy API. -*Units* : milliseconds +*Unit:* milliseconds -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer *Accepted values:* [`-17592186044416`, `17592186044415`] -*Default:* `300000` +*Default:* `300000` (5min) --- -=== consumer_instance_timeout +=== consumer_instance_timeout_ms How long to wait for an idle consumer before removing it. A consumer is considered idle when it's not making requests or heartbeats. -*Units*: milliseconds +*Unit:* milliseconds -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer *Accepted values:* [`-17592186044416`, `17592186044415`] -*Default:* `300000` (5min) +*Default:* `300000` --- === pandaproxy_api -Specifies the listener address and port for the Rest API. - -*Optional:* Yes +Rest API listener address and port. -*Visibility:* `None` +*Visibility:* `user` -*Default:* `127.0.0.1:8082` +*Default:* `0.0.0.0:8082` --- @@ -545,9 +475,7 @@ Specifies the listener address and port for the Rest API. TLS configuration for Pandaproxy api. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Default:* `null` @@ -561,11 +489,7 @@ Configuration options for HTTP Proxy Client. TLS configuration for the Kafka API servers to which the HTTP Proxy client should connect. -*Optional:* Yes - -*Visibility:* `None` - -*Default:* `config::tls_config()` +*Visibility:* `user` --- @@ -573,13 +497,11 @@ TLS configuration for the Kafka API servers to which the HTTP Proxy client shoul Network addresses of the Kafka API servers to which the HTTP Proxy client should connect. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* array -*Default:* `["127.0.0.1:9092"]` +*Default:* `['127.0.0.1:9092']` --- @@ -587,9 +509,7 @@ Network addresses of the Kafka API servers to which the HTTP Proxy client should Custom identifier to include in the Kafka request header for the HTTP Proxy client. This identifier can help debug or monitor client activities. -*Optional:* No - -*Visibility:* `None` +*Visibility:* `user` *Type:* string @@ -597,15 +517,13 @@ Custom identifier to include in the Kafka request header for the HTTP Proxy clie --- -=== consumer_heartbeat_interval +=== consumer_heartbeat_interval_ms Interval (in milliseconds) for consumer heartbeats. -*Units*: milliseconds +*Unit:* milliseconds -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -615,21 +533,19 @@ Interval (in milliseconds) for consumer heartbeats. --- -=== consumer_rebalance_timeout +=== consumer_rebalance_timeout_ms Timeout (in milliseconds) for consumer rebalance. -*Units*: milliseconds +*Unit:* milliseconds -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer *Accepted values:* [`-17592186044416`, `17592186044415`] -*Default:* `200` +*Default:* `2000` --- @@ -637,11 +553,9 @@ Timeout (in milliseconds) for consumer rebalance. Maximum bytes to fetch per request. -*Units*: bytes - -*Optional:* Yes +*Unit:* bytes -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -655,11 +569,9 @@ Maximum bytes to fetch per request. Minimum bytes to fetch per request. -*Units*: bytes - -*Optional:* Yes +*Unit:* bytes -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -669,15 +581,13 @@ Minimum bytes to fetch per request. --- -=== consumer_request_timeout +=== consumer_request_timeout_ms Interval (in milliseconds) for consumer request timeout. -*Units*: milliseconds - -*Optional:* Yes +*Unit:* milliseconds -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -687,15 +597,13 @@ Interval (in milliseconds) for consumer request timeout. --- -=== consumer_session_timeout +=== consumer_session_timeout_ms Timeout (in milliseconds) for consumer session. -*Units*: milliseconds - -*Optional:* Yes +*Unit:* milliseconds -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -709,9 +617,7 @@ Timeout (in milliseconds) for consumer session. Number of acknowledgments the producer requires the leader to have received before considering a request complete. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -721,15 +627,13 @@ Number of acknowledgments the producer requires the leader to have received befo --- -=== produce_batch_delay +=== produce_batch_delay_ms Delay (in milliseconds) to wait before sending batch. -*Units*: milliseconds +*Unit:* milliseconds -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -743,9 +647,7 @@ Delay (in milliseconds) to wait before sending batch. Number of records to batch before sending to broker. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -759,11 +661,9 @@ Number of records to batch before sending to broker. Number of bytes to batch before sending to broker. -*Units*: bytes - -*Optional:* Yes +*Unit:* bytes -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -777,27 +677,21 @@ Number of bytes to batch before sending to broker. Enable or disable compression by the Kafka client. Specify `none` to disable compression or one of the supported types [gzip, snappy, lz4, zstd]. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* string -*Accepted values:* `gzip`, `snappy`, `lz4`, `zstd` - *Default:* `none` --- -=== produce_shutdown_delay +=== produce_shutdown_delay_ms Delay (in milliseconds) to allow for final flush of buffers before shutting down. -*Units*: milliseconds - -*Optional:* Yes +*Unit:* milliseconds -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -811,9 +705,7 @@ Delay (in milliseconds) to allow for final flush of buffers before shutting down Number of times to retry a request to a broker. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -821,15 +713,13 @@ Number of times to retry a request to a broker. --- -=== retry_base_backoff +=== retry_base_backoff_ms Delay (in milliseconds) for initial retry backoff. -*Units*: milliseconds - -*Optional:* Yes +*Unit:* milliseconds -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -843,9 +733,7 @@ Delay (in milliseconds) for initial retry backoff. The SASL mechanism to use when connecting. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* string @@ -857,9 +745,7 @@ The SASL mechanism to use when connecting. Password to use for SCRAM authentication mechanisms. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* string @@ -871,12 +757,10 @@ Password to use for SCRAM authentication mechanisms. Username to use for SCRAM authentication mechanisms. -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* string *Default:* `null` ---- +--- \ No newline at end of file diff --git a/modules/reference/pages/properties/cluster-properties.adoc b/modules/reference/pages/properties/cluster-properties.adoc index 48a128b42..c20839af1 100644 --- a/modules/reference/pages/properties/cluster-properties.adoc +++ b/modules/reference/pages/properties/cluster-properties.adoc @@ -14,11 +14,9 @@ NOTE: Some cluster properties require that you restart the cluster for any updat Capacity (in number of txns) of an abort index segment. -Each partition tracks the aborted transaction offset ranges to help service client requests. If the number transactions increase beyond this threshold, they are flushed to disk to easy memory pressure. Then they're loaded on demand. This configuration controls the maximum number of aborted transactions before they are flushed to disk. +Each partition tracks the aborted transaction offset ranges to help service client requests. If the number of transactions increases beyond this threshold, they are flushed to disk to ease memory pressure. Then they're loaded on demand. This configuration controls the maximum number of aborted transactions before they are flushed to disk. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -36,9 +34,7 @@ Interval, in milliseconds, at which Redpanda looks for inactive transactions and *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -56,8 +52,6 @@ Whether Admin API clients must provide HTTP basic authentication headers. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -72,9 +66,7 @@ Enable aggregation of metrics returned by the xref:reference:internal-metrics-re *Requires restart:* No -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* boolean @@ -90,8 +82,6 @@ The amount of time since the last broker status heartbeat. After this time, a br *Requires restart:* No -*Nullable:* No - *Visibility:* `tunable` *Type:* integer @@ -110,8 +100,6 @@ The duration, in milliseconds, that Redpanda waits for the replication of entrie *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -126,9 +114,7 @@ The duration, in milliseconds, that Redpanda waits for the replication of entrie Size of direct write operations to disk in bytes. A larger chunk size can improve performance for write-heavy workloads, but increase latency for these writes as more data is collected before each write operation. A smaller chunk size can decrease write latency, but potentially increase the number of disk I/O operations. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -144,8 +130,6 @@ Defines the number of bytes allocated by the internal audit client for audit mes *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -160,8 +144,6 @@ Enables or disables audit logging. When you set this to true, Redpanda checks fo *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -176,8 +158,6 @@ List of strings in JSON style identifying the event types to include in the audi *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* array @@ -192,8 +172,6 @@ List of user principals to exclude from auditing. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* array @@ -208,8 +186,6 @@ List of topics to exclude from auditing. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* array @@ -222,9 +198,9 @@ List of topics to exclude from auditing. Defines the number of partitions used by a newly-created audit topic. This configuration applies only to the audit log topic and may be different from the cluster or other topic configurations. This cannot be altered for existing audit log topics. -*Requires restart:* No +*Unit:* number of partitions per topic -*Optional:* Yes +*Requires restart:* No *Visibility:* `user` @@ -242,8 +218,6 @@ Defines the replication factor for a newly-created audit log topic. This configu *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -262,8 +236,6 @@ Interval, in milliseconds, at which Redpanda flushes the queued audit log messag *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -280,8 +252,6 @@ Defines the maximum amount of memory in bytes used by the audit buffer in each s *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -298,8 +268,6 @@ If you produce to a topic that doesn't exist, the topic will be created with def *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -314,9 +282,7 @@ Cluster identifier. *Requires restart:* No -*Optional:* No - -*Visibility:* `None` +*Visibility:* `user` *Type:* string @@ -330,8 +296,6 @@ Size (in bytes) for each compacted log segment. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -344,11 +308,9 @@ Size (in bytes) for each compacted log segment. === compaction_ctrl_backlog_size -Target backlog size for compaction controller. If not set the max backlog size is configured to 80% of total disk space available. - -*Requires restart:* No +Target backlog size for compaction controller. If not set the max backlog size is configured to 80% of total disk space available. -*Optional:* No +*Requires restart:* Yes *Visibility:* `tunable` @@ -362,9 +324,7 @@ Target backlog size for compaction controller. If not set the max backlog size i Derivative coefficient for compaction PID controller. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -378,9 +338,7 @@ Derivative coefficient for compaction PID controller. Integral coefficient for compaction PID controller. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -394,9 +352,7 @@ Integral coefficient for compaction PID controller. Maximum number of I/O and CPU shares that compaction process can use. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -412,9 +368,7 @@ Maximum number of I/O and CPU shares that compaction process can use. Minimum number of I/O and CPU shares that compaction process can use. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -430,9 +384,7 @@ Minimum number of I/O and CPU shares that compaction process can use. Proportional coefficient for compaction PID controller. This must be negative, because the compaction backlog should decrease when the number of compaction shares increases. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -450,8 +402,6 @@ Interval between iterations of controller backend housekeeping loop. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -468,8 +418,6 @@ Maximum capacity of rate limit accumulation in controller ACLs and users operati *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -484,8 +432,6 @@ Maximum capacity of rate limit accumulation in controller configuration operatio *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -500,8 +446,6 @@ Maximum capacity of rate limit accumulation in controller move operations limit. *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -516,8 +460,6 @@ Maximum capacity of rate limit accumulation in controller node management operat *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -532,8 +474,6 @@ Maximum capacity of rate limit accumulation in controller topic operations limit *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -546,12 +486,10 @@ Maximum capacity of rate limit accumulation in controller topic operations limit Maximum amount of time before Redpanda attempts to create a controller snapshot after a new controller command appears. -*Unit*: seconds +*Unit:* seconds *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -568,8 +506,6 @@ If set to `true`, move partitions between cores in runtime to maintain balanced *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* boolean @@ -586,15 +522,13 @@ Interval, in milliseconds, between trigger and invocation of core balancing. *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer *Accepted values:* [`-17592186044416`, `17592186044415`] -*Default:* `10000` +*Default:* `10000` (10s) --- @@ -604,8 +538,6 @@ If set to `true`, and if after a restart the number of cores changes, Redpanda w *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* boolean @@ -620,8 +552,6 @@ Enables CPU profiling for Redpanda. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -638,8 +568,6 @@ The sample period for the CPU profiler. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -658,8 +586,6 @@ Timeout, in milliseconds, to wait for new topic creation. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -676,8 +602,6 @@ The maximum size for a deployable WebAssembly binary that the broker can store. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -694,8 +618,6 @@ The commit interval at which data transforms progress. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -712,8 +634,6 @@ Enables WebAssembly-powered data transforms directly in the broker. When `data_t *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -726,9 +646,9 @@ Enables WebAssembly-powered data transforms directly in the broker. When `data_t Buffer capacity for transform logs, per shard. Buffer occupancy is calculated as the total size of buffered log messages; that is, logs emitted but not yet produced. -*Requires restart:* Yes +*Unit:* bytes -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -746,8 +666,6 @@ Flush interval for transform logs. When a timer expires, pending logs are collec *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -766,8 +684,6 @@ Transform log lines truncate to this length. Truncation occurs after any charact *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -782,8 +698,6 @@ The amount of memory to reserve per core for data transform (Wasm) virtual machi *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -798,8 +712,6 @@ The amount of memory to give an instance of a data transform (Wasm) virtual mach *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -816,8 +728,6 @@ The percentage of available memory in the transform subsystem to use for read bu *Requires restart:* Yes -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -834,8 +744,6 @@ The maximum amount of runtime to start up a data transform, and the time it take *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -854,8 +762,6 @@ The percentage of available memory in the transform subsystem to use for write b *Requires restart:* Yes -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -864,14 +770,44 @@ The percentage of available memory in the transform subsystem to use for write b --- +=== debug_bundle_auto_removal_seconds + +If set, how long debug bundles are kept in the debug bundle storage directory after they are created. If not set, debug bundles are kept indefinitely. + +*Unit:* seconds + +*Requires restart:* No + +*Visibility:* `user` + +*Type:* integer + +*Accepted values:* [`-17179869184`, `17179869183`] + +*Default:* `null` + +--- + +=== debug_bundle_storage_dir + +Path to the debug bundle storage directory. Note: Changing this path does not clean up existing debug bundles. If not set, the debug bundle is stored in the Redpanda data directory specified in the redpanda.yaml broker configuration file. + +*Requires restart:* No + +*Visibility:* `user` + +*Type:* string + +*Default:* `null` + +--- + === debug_load_slice_warning_depth The recursion depth after which debug logging is enabled automatically for the log reader. *Requires restart:* No -*Nullable:* Yes - *Visibility:* `tunable` *Type:* integer @@ -882,14 +818,24 @@ The recursion depth after which debug logging is enabled automatically for the l --- +=== default_leaders_preference + +Default settings for preferred location of topic partition leaders. It can be either "none" (no preference), or "racks:,,..." (prefer brokers with rack id from the list). + +*Requires restart:* No + +*Visibility:* `user` + +*Default:* `none` + +--- + === default_num_windows Default number of quota tracking windows. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -904,12 +850,10 @@ Default number of quota tracking windows. Default number of partitions per topic. -*Unit*: number of partitions per topic +*Unit:* number of partitions per topic *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -926,8 +870,6 @@ Default replication factor for new topics. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -948,8 +890,6 @@ Default quota tracking window size in milliseconds. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -960,13 +900,25 @@ Default quota tracking window size in milliseconds. --- -=== disable_batch_cache +=== development_enable_cloud_topics -Disable batch cache in log manager. +Enable cloud topics. *Requires restart:* No -*Optional:* Yes +*Visibility:* `user` + +*Type:* boolean + +*Default:* `false` + +--- + +=== disable_batch_cache + +Disable batch cache in log manager. + +*Requires restart:* Yes *Visibility:* `tunable` @@ -978,12 +930,12 @@ Disable batch cache in log manager. === disable_cluster_recovery_loop_for_tests -Disables the cluster recovery loop. This property is used to simplify testing and should not be set in production. +include::reference:partial$internal-use-property.adoc[] + +Disables the cluster recovery loop. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -996,11 +948,9 @@ Disables the cluster recovery loop. This property is used to simplify testing an Disable registering the metrics exposed on the internal `/metrics` endpoint. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes -*Visibility:* `None` +*Visibility:* `user` *Type:* boolean @@ -1012,11 +962,9 @@ Disable registering the metrics exposed on the internal `/metrics` endpoint. Disable registering the metrics exposed on the `/public_metrics` endpoint. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes -*Visibility:* `None` +*Visibility:* `user` *Type:* boolean @@ -1035,8 +983,6 @@ It is recommended to not run disks near capacity to avoid blocking I/O due to lo *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* number @@ -1051,8 +997,6 @@ Enables cluster metadata uploads. Required for xref:manage:whole-cluster-restore *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -1067,8 +1011,6 @@ Limits the write rate for the controller log. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -1081,9 +1023,7 @@ Limits the write rate for the controller log. Enable idempotent producers. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `user` @@ -1095,12 +1035,10 @@ Enable idempotent producers. === enable_leader_balancer -Enable automatic leadership rebalancing. Mode is set by <>. +Enable automatic leadership rebalancing. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -1123,8 +1061,6 @@ The cluster metrics of the metrics reporter are different from xref:manage:monit *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -1139,8 +1075,6 @@ Enable Redpanda extensions for MPX. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -1153,9 +1087,7 @@ Enable Redpanda extensions for MPX. Enable PID file. You should not need to change. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1171,8 +1103,6 @@ Enable rack-aware replica assignment. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -1187,8 +1117,6 @@ Enable SASL authentication for Kafka connections. Authorization is required to m *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -1207,8 +1135,6 @@ Mode to enable server-side schema ID validation. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Accepted Values*: @@ -1225,9 +1151,7 @@ Mode to enable server-side schema ID validation. Enable transactions (atomic writes). -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `user` @@ -1243,8 +1167,6 @@ Enables the usage tracking mechanism, storing windowed history of kafka/cloud_st *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -1259,8 +1181,6 @@ Whether new feature flags auto-activate after upgrades (true) or must wait for m *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -1273,12 +1193,10 @@ Whether new feature flags auto-activate after upgrades (true) or must wait for m Maximum number of bytes returned in a fetch request. -*Unit*: bytes +*Unit:* bytes *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -1287,6 +1205,82 @@ Maximum number of bytes returned in a fetch request. --- +=== fetch_pid_d_coeff + +Derivative coefficient for fetch PID controller. + +*Requires restart:* No + +*Visibility:* `tunable` + +*Type:* number + +*Default:* `0.0` + +--- + +=== fetch_pid_i_coeff + +Integral coefficient for fetch PID controller. + +*Requires restart:* No + +*Visibility:* `tunable` + +*Type:* number + +*Default:* `0.01` + +--- + +=== fetch_pid_max_debounce_ms + +The maximum debounce time the fetch PID controller will apply, in milliseconds. + +*Unit:* milliseconds + +*Requires restart:* No + +*Visibility:* `tunable` + +*Type:* integer + +*Accepted values:* [`-17592186044416`, `17592186044415`] + +*Default:* `100` + +--- + +=== fetch_pid_p_coeff + +Proportional coefficient for fetch PID controller. + +*Requires restart:* No + +*Visibility:* `tunable` + +*Type:* number + +*Default:* `100.0` + +--- + +=== fetch_pid_target_utilization_fraction + +A fraction, between 0 and 1, for the target reactor utilization of the fetch scheduling group. + +*Unit:* fraction + +*Requires restart:* No + +*Visibility:* `tunable` + +*Type:* number + +*Default:* `0.2` + +--- + === fetch_read_strategy The strategy used to fulfill fetch requests. @@ -1299,8 +1293,6 @@ The strategy used to fulfill fetch requests. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Accepted Values:* `polling`, `non_polling`, `non_polling_with_debounce` @@ -1317,8 +1309,6 @@ Time to wait for the next read in fetch requests when the requested minimum byte *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1335,9 +1325,7 @@ Time duration after which the inactive fetch session is removed from the fetch s *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1357,8 +1345,6 @@ Delay added to the rebalance phase to wait for new members. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1373,13 +1359,11 @@ Delay added to the rebalance phase to wait for new members. The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures. -*Unit*: milliseconds +*Unit:* milliseconds *Requires restart:* No -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -1393,13 +1377,11 @@ The maximum allowed session timeout for registered consumers. Longer timeouts gi The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating, which can overwhelm broker resources. -*Unit*: milliseconds +*Unit:* milliseconds *Requires restart:* No -*Optional:* Yes - -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -1417,8 +1399,6 @@ Timeout for new member joins. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1437,8 +1417,6 @@ Frequency rate at which the system should check for expired group offsets. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1457,8 +1435,6 @@ Consumer group offset retention seconds. To disable offset retention, set this t *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1473,9 +1449,9 @@ Consumer group offset retention seconds. To disable offset retention, set this t Number of partitions in the internal group membership topic. -*Requires restart:* No +*Unit:* number of partitions per topic -*Optional:* Yes +*Requires restart:* No *Visibility:* `tunable` @@ -1492,10 +1468,7 @@ Number of partitions in the internal group membership topic. How often the health manager runs. *Unit:* milliseconds - -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1515,8 +1488,6 @@ Maximum age of the metadata cached in the health monitor of a non-controller bro *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1533,8 +1504,6 @@ A list of supported HTTP authentication mechanisms. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* array @@ -1545,13 +1514,25 @@ A list of supported HTTP authentication mechanisms. --- +=== iceberg_enabled + +Enables the translation of topic data into Iceberg tables. Setting `iceberg_enabled` to `true` activates the feature at the cluster level, but each topic must also set the `redpanda.iceberg.enabled` topic-level property to `true` to use it. If `iceberg_enabled` is set to `false`, then the feature is disabled for all topics in the cluster, overriding any topic-level settings. + +*Requires restart:* Yes + +*Visibility:* `user` + +*Type:* boolean + +*Default:* `false` + +--- + === id_allocator_batch_size The ID allocator allocates messages in batches (each batch is a one log record) and then serves requests from memory without touching the log until the batch is exhausted. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1565,11 +1546,9 @@ The ID allocator allocates messages in batches (each batch is a one log record) === id_allocator_log_capacity -Capacity of the `id_allocator` log in number of batches. After it reaches `id_allocator_stm`, it truncates the log's prefix. - -*Requires restart:* No +Capacity of the `id_allocator` log in number of batches. After it reaches `id_allocator_stm`, it truncates the log's prefix. -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1589,8 +1568,6 @@ Initial local retention size target for partitions of topics with xref:manage:ti *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -1607,8 +1584,6 @@ Initial local retention time target for partitions of topics with xref:manage:ti *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -1624,10 +1599,7 @@ Initial local retention time target for partitions of topics with xref:manage:ti Target replication factor for internal topics. *Unit*: number of replicas per topic. - -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `user` @@ -1645,9 +1617,7 @@ Time between cluster join retries in milliseconds. *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1659,13 +1629,29 @@ Time between cluster join retries in milliseconds. --- +=== kafka_admin_topic_api_rate + +Target quota rate (partition mutations per default_window_sec). + +*Requires restart:* No + +*Visibility:* `user` + +*Type:* integer + +*Accepted values:* [`0`, `4294967295`] + +*Default:* `null` + +--- + === kafka_batch_max_bytes Maximum size of a batch processed by the server. If the batch is compressed, the limit applies to the compressed batch size. -*Requires restart:* No +*Unit:* bytes -*Optional:* Yes +*Requires restart:* No *Visibility:* `tunable` @@ -1677,19 +1663,35 @@ Maximum size of a batch processed by the server. If the batch is compressed, the --- -=== kafka_connection_rate_limit +=== kafka_client_group_byte_rate_quota -Maximum connections per second for one core. If `null` (the default), then the number of connections per second is unlimited. +Per-group target produce quota byte rate (bytes per second). Client is considered part of the group if client_id contains clients_prefix. -*Unit*: number of connections per second, per core +*Requires restart:* No -*Related topics*: +*Visibility:* `user` -* xref:manage:cluster-maintenance/configure-availability.adoc#limit-client-connections[Limit client connections] +*Default:* `null` + +--- + +=== kafka_client_group_fetch_byte_rate_quota + +Per-group target fetch quota byte rate (bytes per second). Client is considered part of the group if client_id contains clients_prefix. *Requires restart:* No -*Optional:* No +*Visibility:* `user` + +*Default:* `null` + +--- + +=== kafka_connection_rate_limit + +Maximum connections per second for one core. If `null` (the default), then the number of connections per second is unlimited. + +*Requires restart:* No *Visibility:* `user` @@ -1711,8 +1713,6 @@ Overrides the maximum connections per second for one core for the specified IP a *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* array @@ -1731,8 +1731,6 @@ Maximum number of Kafka client connections per broker. If `null`, the property i *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -1747,12 +1745,10 @@ Maximum number of Kafka client connections per broker. If `null`, the property i === kafka_connections_max_overrides -A list of IP addresses for which Kafka client connection limits are overridden and don't apply. For example, `(['127.0.0.1:90', '50.20.1.1:40']).` +A list of IP addresses for which Kafka client connection limits are overridden and don't apply. For example, `(['127.0.0.1:90', '50.20.1.1:40']).`. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* array @@ -1771,8 +1767,6 @@ Maximum number of Kafka client connections per IP address, per broker. If `null` *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -1793,8 +1787,6 @@ Flag to require authorization for Kafka connections. If `null`, the property is *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* boolean @@ -1820,8 +1812,6 @@ Whether to include Tiered Storage as a special remote:// directory in `DescribeL *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -1836,8 +1826,6 @@ Enable the Kafka partition reassignment API. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -1850,12 +1838,10 @@ Enable the Kafka partition reassignment API. Kafka group recovery timeout. -*Unit*: milliseconds +*Unit:* milliseconds *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -1872,8 +1858,6 @@ Limit fetch responses to this many bytes, even if the total of partition bytes l *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1888,8 +1872,6 @@ The size of the batch used to estimate memory consumption for fetch requests, in *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -1904,8 +1886,6 @@ The share of Kafka subsystem memory that can be used for fetch read buffers, as *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `user` *Type:* number @@ -1920,8 +1900,6 @@ Principal mapping rules for mTLS authentication on the Kafka API. If `null`, the *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* array @@ -1936,8 +1914,6 @@ A list of topics that are protected from deletion and configuration changes by K *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* string array @@ -1957,8 +1933,6 @@ A list of topics that are protected from being produced to by Kafka clients. Set *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* array @@ -1971,9 +1945,7 @@ A list of topics that are protected from being produced to by Kafka clients. Set Smoothing factor for Kafka queue depth control depth tracking. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1989,9 +1961,7 @@ Update frequency for Kafka queue depth control. *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -2009,8 +1979,6 @@ Enable Kafka queue depth control. *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -2023,9 +1991,7 @@ Enable Kafka queue depth control. Queue depth when idleness is detected in Kafka queue depth control. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -2039,9 +2005,7 @@ Queue depth when idleness is detected in Kafka queue depth control. Smoothing parameter for Kafka queue depth control latency tracking. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -2055,9 +2019,7 @@ Smoothing parameter for Kafka queue depth control latency tracking. Maximum queue depth used in Kafka queue depth control. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -2071,11 +2033,9 @@ Maximum queue depth used in Kafka queue depth control. Maximum latency threshold for Kafka queue depth control depth tracking. -*Unit*: milliseconds - -*Requires restart:* No +*Unit:* milliseconds -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `user` @@ -2091,9 +2051,7 @@ Maximum latency threshold for Kafka queue depth control depth tracking. Minimum queue depth used in Kafka queue depth control. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -2107,9 +2065,7 @@ Minimum queue depth used in Kafka queue depth control. Number of windows used in Kafka queue depth control latency tracking. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -2125,9 +2081,7 @@ Window size for Kafka queue depth control latency tracking. *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -2158,8 +2112,6 @@ If set to `0`, no minimum is enforced. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -2190,8 +2142,6 @@ If set to `0.0`, the minimum is disabled. If set to `1.0`, the balancer won't be *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* number @@ -2204,7 +2154,7 @@ If set to `0.0`, the minimum is disabled. If set to `1.0`, the balancer won't be --- -=== kafka_quota_balancer_node_period +=== kafka_quota_balancer_node_period_ms Intra-node throughput quota balancer invocation period, in milliseconds. When set to 0, the balancer is disabled and makes all the throughput quotas immutable. @@ -2212,8 +2162,6 @@ Intra-node throughput quota balancer invocation period, in milliseconds. When se *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -2224,7 +2172,7 @@ Intra-node throughput quota balancer invocation period, in milliseconds. When se --- -=== kafka_quota_balancer_window +=== kafka_quota_balancer_window_ms Time window used to average current throughput measurement for quota balancer, in milliseconds. @@ -2232,8 +2180,6 @@ Time window used to average current throughput measurement for quota balancer, i *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -2248,9 +2194,9 @@ Time window used to average current throughput measurement for quota balancer, i Maximum size of a single request processed using the Kafka API. -*Requires restart:* No +*Unit:* bytes -*Optional:* Yes +*Requires restart:* No *Visibility:* `tunable` @@ -2266,9 +2212,9 @@ Maximum size of a single request processed using the Kafka API. Maximum size of the user-space receive buffer. If `null`, this limit is not applied. -*Requires restart:* No +*Unit:* bytes -*Optional:* No +*Requires restart:* Yes *Visibility:* `tunable` @@ -2282,17 +2228,15 @@ Maximum size of the user-space receive buffer. If `null`, this limit is not appl Size of the Kafka server TCP receive buffer. If `null`, the property is disabled. -*Unit*: bytes - -*Requires restart:* No +*Unit:* bytes -*Optional:* No +*Requires restart:* Yes -*Visibility:* `None` +*Visibility:* `user` *Type:* integer -*Accepted values:* [`-2147483648`, `2147483647`] aligned to 4096 bytes +*Accepted values:* [`-2147483648`, `2147483647`] *Default:* `null` @@ -2302,13 +2246,11 @@ Size of the Kafka server TCP receive buffer. If `null`, the property is disabled Size of the Kafka server TCP transmit buffer. If `null`, the property is disabled. -*Unit*: bytes - -*Requires restart:* No +*Unit:* bytes -*Optional:* No +*Requires restart:* Yes -*Visibility:* `None` +*Visibility:* `user` *Type:* integer @@ -2328,8 +2270,6 @@ IMPORTANT: If this property is not set (or set to `null`), session expiry is dis *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -2346,8 +2286,6 @@ Per-shard capacity of the cache for validating schema IDs. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2356,7 +2294,7 @@ Per-shard capacity of the cache for validating schema IDs. --- -=== kafka_tcp_keepalive_idle_timeout_seconds +=== kafka_tcp_keepalive_timeout TCP keepalive idle timeout in seconds for Kafka connections. This describes the timeout between TCP keepalive probes that the remote site successfully acknowledged. Refers to the TCP_KEEPIDLE socket option. When changed, applies to new connections only. @@ -2364,8 +2302,6 @@ TCP keepalive idle timeout in seconds for Kafka connections. This describes the *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2384,8 +2320,6 @@ TCP keepalive probe interval in seconds for Kafka connections. This describes th *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2402,8 +2336,6 @@ TCP keepalive unacknowledged probes until the connection is considered dead for *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2433,8 +2365,6 @@ A connection is assigned the first matching group and is then excluded from thro *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* string array @@ -2455,8 +2385,6 @@ List of Kafka API keys that are subject to cluster-wide and node-wide throughput *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* list @@ -2467,16 +2395,12 @@ List of Kafka API keys that are subject to cluster-wide and node-wide throughput === kafka_throughput_limit_node_in_bps -The maximum rate of all ingress Kafka API traffic for a node. Includes all Kafka API traffic (requests, responses, headers, fetched data, produced data, etc.). - -If `null`, the property is disabled, and traffic is not limited. +The maximum rate of all ingress Kafka API traffic for a node. Includes all Kafka API traffic (requests, responses, headers, fetched data, produced data, etc.). If `null`, the property is disabled, and traffic is not limited. -*Unit*: bytes per second +*Unit:* bytes per second *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -2493,16 +2417,12 @@ If `null`, the property is disabled, and traffic is not limited. === kafka_throughput_limit_node_out_bps -The maximum rate of all egress Kafka traffic for a node. Includes all Kafka API traffic (requests, responses, headers, fetched data, produced data, etc.). +The maximum rate of all egress Kafka traffic for a node. Includes all Kafka API traffic (requests, responses, headers, fetched data, produced data, etc.). If `null`, the property is disabled, and traffic is not limited. -If `null`, the property is disabled, and traffic is not limited. - -*Unit*: bytes per second +*Unit:* bytes per second *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -2525,8 +2445,6 @@ This threshold is evaluated with each request for data. When the number of token *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -2549,8 +2467,6 @@ WARNING: Disabling this property is not recommended. It causes your Redpanda clu *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -2567,8 +2483,6 @@ Key-value store flush interval (in milliseconds). *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2583,9 +2497,7 @@ Key-value store flush interval (in milliseconds). Key-value maximum segment size (in bytes). -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -2603,8 +2515,6 @@ Leadership rebalancing idle timeout. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2615,27 +2525,6 @@ Leadership rebalancing idle timeout. --- -=== leader_balancer_mode - -Mode of the leader balancer for optimizing movements of leadership between shards (logical CPU cores). Enabled by <>. - -*Requires restart:* No - -*Optional:* Yes - -*Visibility:* `user` - -*Type:* `string` - -*Accepted Values*: - -* `random_hill_climbing`: a shard is randomly chosen and leadership is moved to it if the load on the original shard is reduced. -* `greedy_balanced_shards`: leadership movement is based on a greedy heuristic of moving leaders from the most loaded shard to the least loaded shard. - -*Default:* `random_hill_climbing` - ---- - === leader_balancer_mute_timeout Leadership rebalancing mute timeout. @@ -2644,8 +2533,6 @@ Leadership rebalancing mute timeout. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2656,7 +2543,7 @@ Leadership rebalancing mute timeout. --- -=== leader_balancer_node_mute_timeout +=== leader_balancer_mute_timeout Leadership rebalancing node mute timeout. @@ -2664,8 +2551,6 @@ Leadership rebalancing node mute timeout. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2682,8 +2567,6 @@ Per shard limit for in-progress leadership transfers. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2698,8 +2581,6 @@ Group offset retention is enabled by default starting in Redpanda version 23.1. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -2710,14 +2591,10 @@ Group offset retention is enabled by default starting in Redpanda version 23.1. === legacy_permit_unsafe_log_operation -Flag to enable a Redpanda cluster operator to use unsafe control characters within strings, such as consumer group names or user names. - -This flag applies only for Redpanda clusters that were originally on version 23.1 or earlier and have been upgraded to version 23.2 or later. Starting in version 23.2, newly-created Redpanda clusters ignore this property. +Flag to enable a Redpanda cluster operator to use unsafe control characters within strings, such as consumer group names or user names. This flag applies only for Redpanda clusters that were originally on version 23.1 or earlier and have been upgraded to version 23.2 or later. Starting in version 23.2, newly-created Redpanda clusters ignore this property. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* boolean @@ -2730,16 +2607,12 @@ This flag applies only for Redpanda clusters that were originally on version 23. === legacy_unsafe_log_warning_interval_sec -Period at which to log a warning about using unsafe strings containing control characters. - -If unsafe strings are permitted by <>, a warning will be logged at an interval specified by this property. +Period at which to log a warning about using unsafe strings containing control characters. If unsafe strings are permitted by `legacy_permit_unsafe_log_operation`, a warning will be logged at an interval specified by this property. -*Unit*: seconds +*Unit:* seconds *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -2760,8 +2633,6 @@ The topic property xref:./topic-properties.adoc#cleanuppolicy[`cleanup.policy`] *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Accepted Values*: `compact`, `delete`, `compact,delete` @@ -2774,12 +2645,10 @@ The topic property xref:./topic-properties.adoc#cleanuppolicy[`cleanup.policy`] How often to trigger background compaction. -*Unit*: milliseconds +*Unit:* milliseconds *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -2796,8 +2665,6 @@ Use sliding window compaction. *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -2814,8 +2681,6 @@ The topic property xref:./topic-properties.adoc#compressiontype[`compression.typ *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Accepted Values:* `gzip`, `snappy`, `lz4`, `zstd`, `producer`, `none`. @@ -2826,14 +2691,10 @@ The topic property xref:./topic-properties.adoc#compressiontype[`compression.typ === log_disable_housekeeping_for_tests -Disables the housekeeping loop for local storage. - -IMPORTANT: This property is used to simplify testing, and should not be set in production. +Disables the housekeeping loop for local storage. This property is used to simplify testing, and should not be set in production. *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -2850,8 +2711,6 @@ Threshold in milliseconds for alerting on messages with a timestamp after the br *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2870,8 +2729,6 @@ Threshold in milliseconds for alerting on messages with a timestamp before the b *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -2890,8 +2747,6 @@ The topic property xref:./topic-properties.adoc#messagetimestamptype[`message.ti *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Accepted Values:* `CreateTime`, `LogAppendTime`. @@ -2904,12 +2759,10 @@ The topic property xref:./topic-properties.adoc#messagetimestamptype[`message.ti The amount of time to keep a log file before deleting it (in milliseconds). If set to `-1`, no time limit is applied. This is a cluster-wide default when a topic does not set or disable xref:./topic-properties.adoc#retentionms[`retention.ms`]. -*Unit*: milliseconds +*Unit:* milliseconds *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Accepted values:* [`-17592186044416`, `17592186044415`] @@ -2924,12 +2777,10 @@ Default lifetime of log segments. If `null`, the property is disabled, and no de The topic property xref:./topic-properties.adoc#segmentms[`segment.ms`] overrides the value of `log_segment_ms` at the topic level. -*Unit*: milliseconds +*Unit:* milliseconds *Requires restart:* No -*Optional:* No - *Visibility:* `user` *Type:* integer @@ -2953,8 +2804,6 @@ Upper bound on topic `segment.ms`: higher values will be clamped to this value. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2973,8 +2822,6 @@ Lower bound on topic `segment.ms`: lower values will be clamped to this value. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -2991,8 +2838,6 @@ Default log segment size in bytes for topics which do not set `segment.bytes`. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3007,9 +2852,9 @@ Default log segment size in bytes for topics which do not set `segment.bytes`. Random variation to the segment size limit used for each partition. -*Requires restart:* Yes +*Unit:* percent -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -3027,8 +2872,6 @@ Upper bound on topic `segment.bytes`: higher values will be clamped to this limi *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -3045,8 +2888,6 @@ Lower bound on topic `segment.bytes`: lower values will be clamped to this limit *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -3063,8 +2904,6 @@ Disable reusable preallocated buffers for LZ4 decompression. *Requires restart:* Yes -*Nullable:* No - *Visibility:* `tunable` *Type:* boolean @@ -3079,8 +2918,6 @@ Maximum compacted segment size after consolidation. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3091,12 +2928,10 @@ Maximum compacted segment size after consolidation. === max_concurrent_producer_ids -Maximum number of the active producers sessions. When the threshold is passed, Redpanda terminates old sessions. When an idle producer corresponding to the terminated session wakes up and produces, its message batches are rejected, and an out of order sequence error is emitted. Consumers don't affect this setting. +Maximum number of active producer sessions. When the threshold is passed, Redpanda terminates old sessions. When an idle producer corresponding to the terminated session wakes up and produces, its message batches are rejected, and an out of order sequence error is emitted. Consumers don't affect this setting. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3113,8 +2948,6 @@ Maximum number of in-flight HTTP requests to HTTP Proxy permitted per shard. An *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3129,8 +2962,6 @@ Maximum number of in-flight HTTP requests to Schema Registry permitted per shard *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3147,8 +2978,6 @@ Fail-safe maximum throttle delay on Kafka requests. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3167,8 +2996,6 @@ For details, see xref:develop:transactions#transaction-usage-tips[Transaction us *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3185,9 +3012,7 @@ Time between members backend reconciliation loop retries. *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -3205,8 +3030,6 @@ If `true`, the Redpanda process will terminate immediately when an allocation ca *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -3219,11 +3042,9 @@ If `true`, the Redpanda process will terminate immediately when an allocation ca Interval for metadata dissemination batching. -*Unit*: milliseconds - -*Requires restart:* No +*Unit:* milliseconds -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -3237,13 +3058,9 @@ Interval for metadata dissemination batching. === metadata_dissemination_retries -Number of attempts to look up a topic's metadata-like shard before a request fails. - -This configuration controls the number of retries that request handlers perform when internal topic metadata (for topics like tx, consumer offsets, etc) is missing. These topics are usually created on demand when users try to use the cluster for the first time and it may take some time for the creation to happen and the metadata to propagate to all the brokers (particularly the broker handling the request). In the mean time Redpanda waits and retry. This configuration controls the number retries. - -*Requires restart:* No +Number of attempts to look up a topic's metadata-like shard before a request fails. This configuration controls the number of retries that request handlers perform when internal topic metadata (for topics like tx, consumer offsets, etc) is missing. These topics are usually created on demand when users try to use the cluster for the first time and it may take some time for the creation to happen and the metadata to propagate to all the brokers (particularly the broker handling the request). In the meantime Redpanda waits and retries. This configuration controls the number retries. -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -3261,9 +3078,7 @@ Delay before retrying a topic lookup in a shard or other meta tables. *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -3281,9 +3096,7 @@ Maximum time to wait in metadata request for cluster health to be refreshed. *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -3300,11 +3113,8 @@ Maximum time to wait in metadata request for cluster health to be refreshed. Cluster metrics reporter report interval. *Unit:* milliseconds - *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3323,8 +3133,6 @@ Cluster metrics reporter tick interval. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3341,8 +3149,6 @@ URL of the cluster metrics reporter. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* string @@ -3361,8 +3167,6 @@ If you change the `minimum_topic_replications` setting, the replication factor o *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -3379,8 +3183,6 @@ How long after the last heartbeat request a node will wait before considering it *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3397,9 +3199,7 @@ Timeout for executing node management operations. *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -3416,11 +3216,8 @@ Timeout for executing node management operations. Time interval between two node status messages. Node status messages establish liveness status outside of the Raft protocol. *Unit:* milliseconds - *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3439,8 +3236,6 @@ Maximum backoff (in milliseconds) to reconnect to an unresponsive peer during no *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -3459,8 +3254,6 @@ The amount of time (in seconds) to allow for when validating the expiry claim in *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -3477,8 +3270,6 @@ The URL pointing to the well-known discovery endpoint for the OIDC provider. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* string @@ -3495,8 +3286,6 @@ The frequency of refreshing the JSON Web Keys (JWKS) used to validate access tok *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -3513,8 +3302,6 @@ Rule for mapping JWT payload claim to a Redpanda user principal. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* string @@ -3529,8 +3316,6 @@ A string representing the intended recipient of the token. *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* string @@ -3545,8 +3330,6 @@ Number of partitions that can be reassigned at once. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -3565,8 +3348,6 @@ When the disk usage of a node exceeds this threshold, it triggers Redpanda to mo *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -3587,8 +3368,6 @@ Minimum size of partition that is going to be prioritized when rebalancing a clu *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -3603,8 +3382,6 @@ Mode of xref:manage:cluster-maintenance/cluster-balancing.adoc[partition balanci *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Accepted values:* @@ -3627,12 +3404,10 @@ NOTE: This property applies only when <= v23.3. + +*Requires restart:* No + +*Visibility:* `user` + +*Type:* boolean + +*Default:* `false` + +--- + === space_management_max_log_concurrency Maximum parallel logs inspected during space management process. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -4855,8 +4542,6 @@ Maximum parallel segments inspected during space management process. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -4873,8 +4558,6 @@ Maximum number of bytes that may be used on each shard by compaction index write *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -4891,8 +4574,6 @@ Maximum number of bytes that may be used on each shard by compaction key-offset *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -4911,8 +4592,6 @@ NOTE: Memory per shard is computed after <>. -*Unit*: milliseconds +*Unit:* milliseconds *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -5291,12 +4970,8 @@ Delete segments older than this age. To ensure transaction state is retained as The size (in bytes) each log segment should be. -*Unit:* bytes - *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -5311,9 +4986,9 @@ The size (in bytes) each log segment should be. Number of partitions for transactions coordinator. -*Requires restart:* No +*Unit:* number of partitions per topic -*Optional:* Yes +*Requires restart:* No *Visibility:* `tunable` @@ -5333,8 +5008,6 @@ The maximum allowed timeout for transactions. If a client-requested transaction *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -5349,12 +5022,10 @@ The maximum allowed timeout for transactions. If a client-requested transaction Expiration time of producer IDs. Measured starting from the time of the last write until now for a given ID. -*Unit*: milliseconds +*Unit:* milliseconds *Requires restart:* No -*Optional:* Yes - *Visibility:* `user` *Type:* integer @@ -5369,11 +5040,9 @@ Expiration time of producer IDs. Measured starting from the time of the last wri Delay before scheduling the next check for timed out transactions. -*Unit*: milliseconds - -*Requires restart:* No +*Unit:* milliseconds -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `user` @@ -5391,8 +5060,6 @@ Enables delete retention of consumer offsets topic. This is an internal-only con *Requires restart:* Yes -*Nullable:* No - *Visibility:* `user` *Type:* boolean @@ -5409,8 +5076,6 @@ The interval in which all usage stats are written to disk. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -5427,8 +5092,6 @@ The number of windows to persist in memory and disk. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -5445,8 +5108,6 @@ The width of a usage window, tracking cloud and kafka ingress/egress traffic eac *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -5463,8 +5124,6 @@ Use a separate scheduler group for fetch processing. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -5479,8 +5138,6 @@ Minimum number of active producers per virtual cluster. *Requires restart:* No -*Nullable:* No - *Visibility:* `tunable` *Type:* integer @@ -5499,8 +5156,6 @@ Timeout to wait for leadership in metadata cache. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -5519,8 +5174,6 @@ The `write_caching_default` cluster property can be overridden with the xref:top *Requires restart:* no -*Optional:* No - *Type*: string *Accepted values:* @@ -5541,9 +5194,9 @@ The `write_caching_default` cluster property can be overridden with the xref:top Size of the zstd decompression workspace. -*Requires restart:* No +*Unit:* bytes -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -5552,3 +5205,4 @@ Size of the zstd decompression workspace. *Default:* `8388608` --- + diff --git a/modules/reference/pages/properties/object-storage-properties.adoc b/modules/reference/pages/properties/object-storage-properties.adoc index 43c0090ac..0f8f0bfeb 100644 --- a/modules/reference/pages/properties/object-storage-properties.adoc +++ b/modules/reference/pages/properties/object-storage-properties.adoc @@ -11,14 +11,12 @@ Object storage properties should only be set if you enable xref:manage:tiered-st === cloud_storage_access_key -AWS or GCP access key. This access key is part of the credentials that Redpanda requires to authenticate with object storage services for Tiered Storage. This access key is used with the <> to form the complete credentials required for authentication. +AWS or GCP access key. This access key is part of the credentials that Redpanda requires to authenticate with object storage services for Tiered Storage. This access key is used with the <> to form the complete credentials required for authentication. -To authenticate using IAM roles, see <>. +To authenticate using IAM roles, see <>. *Requires restart:* Yes -*Optional:* No - *Visibility:* `user` *Type:* string @@ -51,287 +49,15 @@ Optional API endpoint. The only instance in which you must set this value is whe TLS port override. -*Requires restart:* No - -*Optional:* Yes - -*Visibility:* `user` - -*Type:* integer - -*Accepted values:* [`-32768`, `32767`] - -*Default:* `443` - ---- - -=== cloud_storage_bucket - -AWS or GCP bucket that should be used to store data. - -*Requires restart:* No - -*Optional:* No - -*Visibility:* `user` - -*Type:* string - -*Default:* `null` - ---- - - -=== cloud_storage_cache_size - -Maximum size of object storage cache. - -If both this property and <> are set, Redpanda uses the minimum of the two. - -*Units*: bytes - -*Requires restart:* No - -*Optional:* Yes - -*Visibility:* `user` - -*Type:* integer - -*Accepted values:* [`0`, `18446744073709551615`] - -*Default:* `0` - - ---- - -=== cloud_storage_cluster_metadata_upload_interval_ms - -Time interval to wait between cluster metadata uploads. - -*Units*: milliseconds - -*Requires restart:* No - -*Optional:* Yes - -*Visibility:* `tunable` - -*Type:* integer - -*Accepted values:* [`-17592186044416`, `17592186044415`] - -*Default:* `3600000` (1 hour) - ---- - -=== cloud_storage_credentials_source - -The source of credentials used to authenticate to object storage services. -Required for AWS or GCP authentication with IAM roles. - -To authenticate using access keys, see <>. - -*Accepted values*: `config_file`, `aws_instance_metadata`, `sts, gcp_instance_metadata`, `azure_vm_instance_metadata`, `azure_aks_oidc_federation` - -*Requires restart:* Yes - -*Optional:* Yes - -*Visibility:* `user` - -*Default:* `config_file` - ---- - -=== cloud_storage_crl_file - -Path to certificate revocation list for <>. - -*Requires restart:* No - -*Optional:* Yes - -*Visibility:* `user` - -*Type:* string - -*Default:* `null` - ---- - -=== cloud_storage_disable_archiver_manager - -Use legacy upload mode and do not start archiver_manager. - *Requires restart:* Yes -*Optional:* No - -*Visibility:* `user` - -*Type:* boolean - -*Default:* `true` - ---- - -=== cloud_storage_disable_tls - -Disable TLS for all object storage connections. - -*Type*: boolean - -*Default*: false - -*Restart required*: yes - ---- - -=== cloud_storage_enabled - -Enable object storage. Must be set to `true` to use Tiered Storage or Remote Read Replicas. - -*Requires restart:* No - -*Optional:* Yes - -*Visibility:* `user` - -*Type:* boolean - -*Default:* `false` - ---- - -=== cloud_storage_max_connections - -Maximum simultaneous object storage connections per shard, applicable to upload and download activities. - -*Units*: number of simultaneous connections - -*Requires restart:* No - -*Optional:* Yes - *Visibility:* `user` *Type:* integer *Accepted values:* [`-32768`, `32767`] -*Default:* `20` - ---- - -=== cloud_storage_recovery_topic_validation_mode - -Validation performed before recovering a topic from object storage. In case of failure, the reason for the failure appears as `ERROR` lines in the Redpanda application log. For each topic, this reports errors for all partitions, but for each partition, only the first error is reported. - -This property accepts the following parameters: - -- `no_check`: Skips the checks for topic recovery. -- `check_manifest_existence`: Runs an existence check on each `partition_manifest`. Fails if there are connection issues to the object storage. -- `check_manifest_and_segment_metadata`: Downloads the manifest and runs a consistency check, comparing the metadata with the cloud storage objects. The process fails if metadata references any missing cloud storage objects. - -Example: Redpanda validates the topic `kafka/panda-topic-recovery-NOT-OK` and stops due to a fatal error on partition 0: - -```bash -ERROR 2024-04-24 21:29:08,166 [shard 1:main] cluster - [fiber11|0|299996ms recovery validation of {kafka/panda-topic-recovery-NOT-OK/0}/24] - manifest metadata check: missing segment, validation not ok -ERROR 2024-04-24 21:29:08,166 [shard 1:main] cluster - topics_frontend.cc:519 - Stopping recovery of {kafka/panda-topic-recovery-NOT-OK} due to validation error -``` - -Each failing partition error message has the following format: - -```bash -ERROR .... [... recovery validation of {}...] - , validation not ok -``` - -At the end of the process, Redpanda outputs a final ERROR message: - -```bash -ERROR ... ... - Stopping recovery of {} due to validation error -``` - -*Requires restart:* No - -*Required:* No - -*Visibility:* `tunable` - -*Type:* string - -*Default:* `check_manifest_existence` - -*Accepted values:* [`no_check`, `check_manifest_existence`, `check_manifest_and_segment_metadata`] - -*Related topics:* xref:manage:whole-cluster-restore.adoc[] - ---- - -=== cloud_storage_recovery_topic_validation_depth - -Number of metadata segments to validate, from newest to oldest, when <> is set to `check_manifest_and_segment_metadata`. - -*Requires restart:* No - -*Required:* No - -*Visibility:* `tunable` - -*Type:* integer - -*Accepted values:* [`0`, `4294967295`] - -*Default:* `10` - ---- - -=== cloud_storage_region - -AWS or GCP region that houses the bucket or container used for storage. - -*Requires restart:* No - -*Optional:* No - -*Visibility:* `user` - -*Type:* string - -*Default:* `null` - ---- - -=== cloud_storage_secret_key - -AWS or GCP secret key. - -*Requires restart:* Yes - -*Optional:* No - -*Visibility:* `user` - -*Type:* string - -*Default:* `null` - ---- - -=== cloud_storage_trust_file - -Path to certificate that should be used to validate server certificate during TLS handshake. - -*Requires restart:* No - -*Optional:* No - -*Visibility:* `user` - -*Type:* string - -*Default:* `null` +*Default:* `443` --- @@ -341,8 +67,6 @@ When set to `true`, Redpanda automatically retrieves cluster metadata from a spe *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -359,8 +83,6 @@ If not set, this is automatically generated using `dfs.core.windows.net` and <>, and whichever limit is hit first will trigger trimming of the cache. +Maximum number of objects that may be held in the Tiered Storage cache. This applies simultaneously with <>, and whichever limit is hit first will trigger trimming of the cache. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -588,8 +305,6 @@ Divide the object storage cache across the specified number of buckets. This onl *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -600,17 +315,29 @@ Divide the object storage cache across the specified number of buckets. This onl --- -=== cloud_storage_cache_size_percent +=== cloud_storage_cache_size -Maximum size of the cloud cache as a percentage of unreserved disk space (see config_ref:disk_reservation_percent,true,cluster-properties[]). The default value for this option is tuned for a shared disk configuration. Consider increasing the value if using a dedicated cache disk. +Maximum size of object storage cache. If both this property and <> are set, Redpanda uses the minimum of the two. -The property <> controls the same limit expressed as a fixed number of bytes. If both `cloud_storage_cache_size` and `cloud_storage_cache_size_percent` are set, Redpanda uses the minimum of the two. +*Requires restart:* No -*Units*: percentage of total disk size. +*Visibility:* `user` -*Requires restart:* No +*Type:* integer + +*Accepted values:* [`0`, `18446744073709551615`] + +*Default:* `0` + +--- -*Optional:* No +=== cloud_storage_cache_size_percent + +Maximum size of the cloud cache as a percentage of unreserved disk space disk_reservation_percent. The default value for this option is tuned for a shared disk configuration. Consider increasing the value if using a dedicated cache disk. The property <> controls the same limit expressed as a fixed number of bytes. If both `cloud_storage_cache_size` and `cloud_storage_cache_size_percent` are set, Redpanda uses the minimum of the two. + +*Unit:* percent + +*Requires restart:* No *Visibility:* `user` @@ -628,8 +355,6 @@ Cache trimming is triggered when the number of objects in the cache reaches this *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* number @@ -646,8 +371,6 @@ Cache trimming is triggered when the cache size reaches this percentage relative *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* number @@ -662,8 +385,6 @@ The maximum number of concurrent tasks launched for traversing the directory str *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -680,8 +401,6 @@ Selects a strategy for evicting unused cache chunks. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Accepted values:* [`eager`, `capped`, `predictive`] @@ -696,8 +415,6 @@ Number of chunks to prefetch ahead of every downloaded chunk. Prefetching additi *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -710,12 +427,10 @@ Number of chunks to prefetch ahead of every downloaded chunk. Prefetching additi === cloud_storage_cluster_metadata_num_consumer_groups_per_upload -Number of groups to upload in a single snapshot object during consumer offsets upload. Setting a lower value means a larger number of smaller snapshots are uploaded. +Number of groups to upload in a single snapshot object during consumer offsets upload. Setting a lower value will mean a larger number of smaller snapshots are uploaded. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -730,8 +445,6 @@ Number of attempts metadata operations may be retried. *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -742,6 +455,24 @@ Number of attempts metadata operations may be retried. --- +=== cloud_storage_cluster_metadata_upload_interval_ms + +Time interval to wait between cluster metadata uploads. + +*Unit:* milliseconds + +*Requires restart:* No + +*Visibility:* `tunable` + +*Type:* integer + +*Accepted values:* [`-17592186044416`, `17592186044415`] + +*Default:* `3600000` + +--- + === cloud_storage_cluster_metadata_upload_timeout_ms Timeout for cluster metadata uploads. @@ -750,8 +481,6 @@ Timeout for cluster metadata uploads. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -764,12 +493,10 @@ Timeout for cluster metadata uploads. === cloud_storage_credentials_host -The hostname to connect to for retrieving role based credentials. Derived from <> if not set. Only required when using IAM role-based access. +The hostname to connect to for retrieving role based credentials. Derived from <> if not set. Only required when using IAM role based access. To authenticate using access keys, see <>. *Requires restart:* Yes -*Optional:* No - *Visibility:* `tunable` *Type:* @@ -780,14 +507,43 @@ The hostname to connect to for retrieving role based credentials. Derived from < --- +=== cloud_storage_credentials_source + +The source of credentials used to authenticate to object storage services. +Required for AWS or GCP authentication with IAM roles. + +To authenticate using access keys, see <>. + +*Accepted values*: `config_file`, `aws_instance_metadata`, `sts, gcp_instance_metadata`, `azure_vm_instance_metadata`, `azure_aks_oidc_federation` + +*Requires restart:* Yes + +*Visibility:* `user` + +*Default:* `config_file` + +--- + +=== cloud_storage_crl_file + +Path to certificate revocation list for <>. + +*Requires restart:* No + +*Visibility:* `user` + +*Type:* string + +*Default:* `null` + +--- + === cloud_storage_disable_chunk_reads Disable chunk reads and switch back to legacy mode where full segments are downloaded. When set to `true`, this option disables the more efficient chunk-based reads, causing Redpanda to download entire segments. This legacy behavior might be useful in specific scenarios where chunk-based fetching is not optimal. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -802,8 +558,6 @@ Disable all metadata consistency checks to allow Redpanda to replay logs with in *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -818,8 +572,6 @@ Begins the read replica sync loop in topic partitions with Tiered Storage enable *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -836,8 +588,6 @@ CAUTION: This property exists to simplify testing and shouldn't be set in produc *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* boolean @@ -846,14 +596,26 @@ CAUTION: This property exists to simplify testing and shouldn't be set in produc --- +=== cloud_storage_disable_tls + +Disable TLS for all object storage connections. + +*Requires restart:* Yes + +*Visibility:* `user` + +*Type:* boolean + +*Default:* `false` + +--- + === cloud_storage_disable_upload_consistency_checks Disable all upload consistency checks to allow Redpanda to upload logs with gaps and replicate metadata with consistency violations. Normally, this option should be disabled. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -868,8 +630,6 @@ Begins the upload loop in topic partitions with Tiered Storage enabled. The prop *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -885,8 +645,6 @@ When set to `true`, Redpanda can re-upload data for compacted topics to object s *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -902,8 +660,6 @@ When set to `true`, new topics are by default configured to allow reading data d *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -919,8 +675,6 @@ When set to `true`, new topics are by default configured to upload data to objec *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -935,8 +689,6 @@ Enable routine checks (scrubbing) of object storage partitions. The scrubber val *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -955,8 +707,6 @@ Enables adjacent segment merging. The segments are reuploaded if there is an opp *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* boolean @@ -965,6 +715,20 @@ Enables adjacent segment merging. The segments are reuploaded if there is an opp --- +=== cloud_storage_enabled + +Enable object storage. Must be set to `true` to use Tiered Storage or Remote Read Replicas. + +*Requires restart:* Yes + +*Visibility:* `user` + +*Type:* boolean + +*Default:* `false` + +--- + === cloud_storage_full_scrub_interval_ms Interval, in milliseconds, between a final scrub and the next scrub. @@ -973,15 +737,13 @@ Interval, in milliseconds, between a final scrub and the next scrub. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer *Accepted values:* [`-17592186044416`, `17592186044415`] -*Default:* `4320000000` (12h) +*Default:* `43200000` (12h) --- @@ -993,8 +755,6 @@ Timeout for running the cloud storage garbage collection, in milliseconds. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1013,8 +773,6 @@ Time limit on waiting for uploads to complete before a leadership transfer. If *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1033,15 +791,13 @@ Interval, in milliseconds, between object storage housekeeping tasks. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer *Accepted values:* [`-17592186044416`, `17592186044415`] -*Default:* `5000` +*Default:* `300000` --- @@ -1053,8 +809,6 @@ A segment is divided into chunks. Chunk hydration means downloading the chunk (w *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* number @@ -1075,8 +829,6 @@ Negative doesn't make sense, but it may not be checked-for/enforced. Large is su *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1093,8 +845,6 @@ The object storage request rate threshold for idle state detection. If the avera *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* number @@ -1111,8 +861,6 @@ The timeout, in milliseconds, used to detect the idle state of the object storag *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1131,8 +879,6 @@ Initial backoff time for exponential backoff algorithm (ms). *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1149,8 +895,6 @@ Scrubber uses the latest cloud storage inventory report, if available, to check *Requires restart:* Yes -*Optional:* No - *Visibility:* `tunable` *Type:* boolean @@ -1159,14 +903,12 @@ Scrubber uses the latest cloud storage inventory report, if available, to check --- -=== cloud_storage_inventory_hash_store +=== cloud_storage_inventory_hash_path_directory Directory to store inventory report hashes for use by cloud storage scrubber. *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `user` *Type:* string @@ -1181,8 +923,6 @@ The name of the scheduled inventory job created by Redpanda to generate bucket o *Requires restart:* Yes -*Optional:* No - *Visibility:* `tunable` *Type:* string @@ -1199,8 +939,6 @@ Maximum bytes of hashes held in memory before writing data to disk during invent *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1215,9 +953,9 @@ Maximum bytes of hashes held in memory before writing data to disk during invent Time interval between checks for a new inventory report in the cloud storage bucket or container. -*Requires restart:* Yes +*Unit:* milliseconds -*Optional:* No +*Requires restart:* Yes *Visibility:* `tunable` @@ -1235,8 +973,6 @@ The prefix to the path in the cloud storage bucket or container where inventory *Requires restart:* Yes -*Optional:* No - *Visibility:* `tunable` *Type:* string @@ -1251,8 +987,6 @@ If enabled, Redpanda will not attempt to create the scheduled report configurati *Requires restart:* Yes -*Optional:* No - *Visibility:* `tunable` *Type:* boolean @@ -1269,8 +1003,6 @@ Amount of memory that can be used to handle Tiered Storage metadata. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1279,7 +1011,7 @@ Amount of memory that can be used to handle Tiered Storage metadata. --- -=== cloud_storage_manifest_cache_ttl_ms +=== cloud_storage_materialized_manifest_ttl_ms The interval, in milliseconds, determines how long the materialized manifest can stay in the cache under contention. This setting is used for performance tuning. When the spillover manifest is materialized and stored in the cache, and the cache needs to evict it, it uses this value as a timeout. The cursor that uses the spillover manifest uses this value as a TTL interval, after which it stops referencing the manifest making it available for eviction. This only affects spillover manifests under contention. @@ -1287,8 +1019,6 @@ The interval, in milliseconds, determines how long the materialized manifest can *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1303,9 +1033,9 @@ The interval, in milliseconds, determines how long the materialized manifest can Minimum interval, in seconds, between partition manifest uploads. Actual time between uploads may be greater than this interval. If this is `null`, metadata is updated after each segment upload. -*Requires restart:* No +*Unit:* seconds -*Optional:* No +*Requires restart:* No *Visibility:* `tunable` @@ -1325,8 +1055,6 @@ Manifest upload timeout, in milliseconds. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1344,8 +1072,6 @@ Maximum concurrent segment hydrations of remote data per CPU core. If unset, va *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1363,9 +1089,7 @@ This setting reduces resource utilization by closing inactive connections. Adjus *Unit:* milliseconds -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1377,14 +1101,28 @@ This setting reduces resource utilization by closing inactive connections. Adjus --- +=== cloud_storage_max_connections + +Maximum simultaneous object storage connections per shard, applicable to upload and download activities. + +*Requires restart:* Yes + +*Visibility:* `user` + +*Type:* integer + +*Accepted values:* [`-32768`, `32767`] + +*Default:* `20` + +--- + === cloud_storage_max_segment_readers_per_shard Maximum concurrent I/O cursors of materialized remote segments per CPU core. If unset, the value of `topic_partitions_per_shard` is used, where one segment reader per partition is used if the shard is at its maximum partition capacity. These readers are cached across Kafka consume requests and store a readahead buffer. *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1401,8 +1139,6 @@ The per-partition limit for the number of segments pending deletion from the clo *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1418,8 +1154,6 @@ This setting limits the Tiered Storage subsystem's throughput per shard, facilit *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1436,8 +1170,6 @@ Timeout for xref:manage:tiered-storage.adoc[] metadata synchronization. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1454,8 +1186,6 @@ The minimum number of chunks per segment for trimming to be enabled. If the numb *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1474,8 +1204,6 @@ Time interval between two partial scrubs of the same partition. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1494,8 +1222,6 @@ Timeout to check if new data is available for partitions in object storage for r *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1512,8 +1238,6 @@ Retention in bytes for topics created during automated recovery. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1522,15 +1246,90 @@ Retention in bytes for topics created during automated recovery. --- -=== cloud_storage_roles_operation_timeout_ms +=== cloud_storage_recovery_topic_validation_depth -Timeout for IAM role related operations, in milliseconds. +Number of metadata segments to validate, from newest to oldest, when <> is set to `check_manifest_and_segment_metadata`. -*Unit:* milliseconds +*Requires restart:* No + +*Required:* No + +*Visibility:* `tunable` + +*Type:* integer + +*Accepted values:* [`0`, `4294967295`] + +*Default:* `10` + +--- + +=== cloud_storage_recovery_topic_validation_mode + +Validation performed before recovering a topic from object storage. In case of failure, the reason for the failure appears as `ERROR` lines in the Redpanda application log. For each topic, this reports errors for all partitions, but for each partition, only the first error is reported. + +This property accepts the following parameters: + +- `no_check`: Skips the checks for topic recovery. +- `check_manifest_existence`: Runs an existence check on each `partition_manifest`. Fails if there are connection issues to the object storage. +- `check_manifest_and_segment_metadata`: Downloads the manifest and runs a consistency check, comparing the metadata with the cloud storage objects. The process fails if metadata references any missing cloud storage objects. + +Example: Redpanda validates the topic `kafka/panda-topic-recovery-NOT-OK` and stops due to a fatal error on partition 0: + +```bash +ERROR 2024-04-24 21:29:08,166 [shard 1:main] cluster - [fiber11|0|299996ms recovery validation of {kafka/panda-topic-recovery-NOT-OK/0}/24] - manifest metadata check: missing segment, validation not ok +ERROR 2024-04-24 21:29:08,166 [shard 1:main] cluster - topics_frontend.cc:519 - Stopping recovery of {kafka/panda-topic-recovery-NOT-OK} due to validation error +``` + +Each failing partition error message has the following format: + +```bash +ERROR .... [... recovery validation of {}...] - , validation not ok +``` + +At the end of the process, Redpanda outputs a final ERROR message: + +```bash +ERROR ... ... - Stopping recovery of {} due to validation error +``` *Requires restart:* No -*Optional:* Yes +*Required:* No + +*Visibility:* `tunable` + +*Type:* string + +*Default:* `check_manifest_existence` + +*Accepted values:* [`no_check`, `check_manifest_existence`, `check_manifest_and_segment_metadata`] + +*Related topics:* xref:manage:whole-cluster-restore.adoc[] + +--- + +=== cloud_storage_region + +Cloud provider region that houses the bucket or container used for storage. + +*Requires restart:* Yes + +*Visibility:* `user` + +*Type:* string + +*Default:* `null` + +--- + +=== cloud_storage_roles_operation_timeout_ms + +Timeout for IAM role related operations (ms). + +*Unit:* milliseconds + +*Requires restart:* Yes *Visibility:* `tunable` @@ -1550,15 +1349,27 @@ Jitter applied to the object storage scrubbing interval. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer *Accepted values:* [`-17592186044416`, `17592186044415`] -*Default:* `600000` (10min) +*Default:* `600000` + +--- + +=== cloud_storage_secret_key + +Cloud provider secret key. + +*Requires restart:* Yes + +*Visibility:* `user` + +*Type:* string + +*Default:* `null` --- @@ -1566,9 +1377,9 @@ Jitter applied to the object storage scrubbing interval. Time that a segment can be kept locally without uploading it to the object storage, in seconds. -*Requires restart:* No +*Unit:* seconds -*Optional:* No +*Requires restart:* No *Visibility:* `tunable` @@ -1586,8 +1397,6 @@ Smallest acceptable segment size in the object storage. Default: `cloud_storage_ *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1604,8 +1413,6 @@ Desired segment size in the object storage. The default is set in the topic-leve *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1622,8 +1429,6 @@ Log segment upload timeout, in milliseconds. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1640,8 +1445,6 @@ Maximum number of segments in the spillover manifest that can be offloaded to th *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1656,8 +1459,6 @@ The size of the manifest which can be offloaded to the cloud. If the size of the *Requires restart:* No -*Optional:* No - *Visibility:* `tunable` *Type:* integer @@ -1670,9 +1471,9 @@ The size of the manifest which can be offloaded to the cloud. If the size of the Maximum throughput used by Tiered Storage per broker expressed as a percentage of the disk bandwidth. If the server has several disks, Redpanda uses the one that stores the Tiered Storage cache. Even if Tiered Storage is allowed to use the full bandwidth of the disk (100%), it won't necessarily use it in full. The actual usage depends on your workload and the state of the Tiered Storage cache. This setting is a safeguard that prevents Tiered Storage from using too many system resources: it is not a performance tuning knob. -*Requires restart:* No +*Unit:* percent -*Optional:* No +*Requires restart:* No *Visibility:* `tunable` @@ -1690,8 +1491,6 @@ Grace period during which the purger refuses to purge the topic. *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1702,13 +1501,25 @@ Grace period during which the purger refuses to purge the topic. --- +=== cloud_storage_trust_file + +Path to certificate that should be used to validate server certificate during TLS handshake. + +*Requires restart:* Yes + +*Visibility:* `user` + +*Type:* string + +*Default:* `null` + +--- + === cloud_storage_upload_ctrl_d_coeff Derivative coefficient for upload PID controller. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1722,9 +1533,7 @@ Derivative coefficient for upload PID controller. Maximum number of I/O and CPU shares that archival upload can use. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1740,9 +1549,7 @@ Maximum number of I/O and CPU shares that archival upload can use. Minimum number of I/O and CPU shares that archival upload can use. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1758,9 +1565,7 @@ Minimum number of I/O and CPU shares that archival upload can use. Proportional coefficient for upload PID controller. -*Requires restart:* No - -*Optional:* Yes +*Requires restart:* Yes *Visibility:* `tunable` @@ -1778,8 +1583,6 @@ Initial backoff interval when there is nothing to upload for a partition, in mil *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1798,8 +1601,6 @@ Maximum backoff interval when there is nothing to upload for a partition, in mil *Requires restart:* No -*Optional:* Yes - *Visibility:* `tunable` *Type:* integer @@ -1822,8 +1623,6 @@ If neither addressing style works, Redpanda terminates the startup, requiring ma *Requires restart:* Yes -*Optional:* Yes - *Visibility:* `user` *Accepted values:*