Skip to content

Commit

Permalink
K8S-380 Add Iceberg info to Topic CRD doc (#881)
Browse files Browse the repository at this point in the history
Co-authored-by: Paulo Borges <[email protected]>
Co-authored-by: Joyce Fee <[email protected]>
Co-authored-by: Gellért Peresztegi-Nagy <[email protected]>
Co-authored-by: Angela Simms <[email protected]>
Co-authored-by: Kat Batuigas <[email protected]>
Co-authored-by: Michele Cyran <[email protected]>
  • Loading branch information
7 people authored Dec 3, 2024
1 parent 3a10803 commit b1371d9
Show file tree
Hide file tree
Showing 4 changed files with 57 additions and 8 deletions.
1 change: 1 addition & 0 deletions local-antora-playbook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ antora:
extensions:
- require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-info'
- require: '@redpanda-data/docs-extensions-and-macros/extensions/unpublish-pages'
- require: '@redpanda-data/docs-extensions-and-macros/extensions/collect-bloblang-samples'
- require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-categories'
- require: '@redpanda-data/docs-extensions-and-macros/extensions/modify-redirects'
- require: '@redpanda-data/docs-extensions-and-macros/extensions/unlisted-pages'
Expand Down
10 changes: 5 additions & 5 deletions modules/get-started/pages/whats-new.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@

This topic includes new content added in version {page-component-version}. For a complete list of all product updates, see the https://github.com/redpanda-data/redpanda/releases/[Redpanda release notes^]. See also:

* xref:redpanda-cloud:get-started:whats-new-cloud.adoc[]
* xref:redpanda-cloud:get-started:whats-new-cloud.adoc[]
* xref:redpanda-cloud:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility]
== Iceberg topics

The xref:manage:topic-iceberg-integration.adoc[Iceberg integration for Redpanda] allows you to store topic data in the cloud in the Iceberg open table format. This makes your streaming data immediately available in downstream analytical systems without setting up and maintaining additional ETL pipelines. You can also integrate your data directly into commonly-used big data processing frameworks, standardizing and simplifying the consumption of streams as tables in a wide variety of data analytics pipelines.

== Leader pinning

For a Redpanda cluster deployed across multiple availability zones (AZs), xref:develop:produce-data/leader-pinning.adoc[leader pinning] ensures that a topic's partition leaders are geographically closer to clients. Leader pinning can lower networking costs and help guarantee lower latency by routing produce and consume requests to brokers located in certain AZs.

== Iceberg topics

The xref:manage:topic-iceberg-integration.adoc[Iceberg integration for Redpanda] allows you to store topic data in the cloud in the Iceberg open table format. This makes your streaming data immediately available in downstream analytical systems without setting up and maintaining additional ETL pipelines. You can also integrate your data directly into commonly-used big data processing frameworks, standardizing and simplifying the consumption of streams as tables in a wide variety of data analytics pipelines.

== Mountable topics

For topics with Tiered Storage enabled, you can unmount a topic to safely detach it from a cluster and keep the topic data in the cluster's object storage bucket or container. You can mount the detached topic to either the same origin cluster, or a different one. This allows you to hibernate a topic and free up system resources taken up by the topic, or migrate a topic to a different cluster. See xref:manage:mountable-topics.adoc[Mountable topics] for details.
Expand Down
47 changes: 47 additions & 0 deletions modules/manage/pages/kubernetes/k-manage-topics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,53 @@ The cleanup policy determines how to manage the partition log files when they re

If the cleanup policy is `delete` or `compact,delete` you can <<Delete records from a topic>>.

=== Configure Iceberg topics

Iceberg topics allow you to create a seamless integration with Apache Iceberg by writing records in Redpanda topics to object storage in Iceberg format.

[NOTE]
====
include::shared:partial$enterprise-license.adoc[]
====

In addition to the general prerequisites for managing topics, you must have the following:

- xref:manage:topic-iceberg-integration.adoc[Iceberg support] must be enabled on your Redpanda cluster.
- xref:manage:kubernetes/tiered-storage/k-tiered-storage.adoc[Tiered Storage] must be enabled on your Redpanda cluster.

To create an Iceberg topic, set the `redpanda.iceberg.mode` configuration in the `additionalConfig` property of the `Topic` resource.

[source,yaml]
----
apiVersion: cluster.redpanda.com/v1alpha2
kind: Topic
metadata:
name: iceberg-key-value
spec:
additionalConfig:
redpanda.iceberg.mode: "<mode>" <.>
kafkaApiSpec:
brokers:
- "redpanda-0.redpanda.<namespace>.svc.cluster.local:9093"
tls:
caCertSecretRef:
name: "redpanda-default-cert"
key: "ca.crt"
----

<.> Supported modes:

- `key_value`: Data is written with explicit keys and values.
- `value_schema_id_prefix`: Data is serialized using schemas managed by the Schema Registry, ensuring compatibility with schema-based systems.
- `disabled` (default)

Apply the resource to your Kubernetes cluster:

[source,bash]
----
kubectl apply -f iceberg-topic.yaml --namespace <namespace>
----

=== Configure write caching

Write caching is a relaxed mode of xref:develop:produce-data/configure-producers.adoc#acksall[`acks=all`] that provides better performance at the expense of durability. It acknowledges a message as soon as it is received and acknowledged on a majority of brokers, without waiting for it to be written to disk. This provides lower latency while still ensuring that a majority of brokers acknowledge the write.
Expand Down
7 changes: 4 additions & 3 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit b1371d9

Please sign in to comment.