Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
ocadaruma committed Aug 14, 2024
1 parent fbd8e03 commit 8afb496
Showing 1 changed file with 5 additions and 15 deletions.
20 changes: 5 additions & 15 deletions docs/consuming-any-topic.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,24 +3,14 @@ Consuming Arbitrary Topic
:base_version: 9.0.0
:modules: common,protocol,processor

This document covers an advanced usage of Decaton processor when you process topics containing records not produced by DecatonClient.
This document guides you how to consume and process topics containing records not produced by DecatonClient using Decaton processors.

[NOTE]
====
* From Decaton 9.0.0, DecatonClient no longer wraps tasks with DecatonTaskRequest protobuf message.
** The wrapper was a heritage from the time when Kafka didn't support record headers to store task metadata.
** Now task metadata is stored in record headers instead, so there are no differences in wire format between tasks produced by DecatonClient and by other clients.
* Hence, you can just use link:../processor/src/main/java/com/linecorp/decaton/processor/runtime/ProcessorsBuilder.java[ProcessorsBuilder#consuming(String topic, Deserializer<T> deserializer)] to process arbitrary topics in most cases.
** So the expected use cases of this guide are:
*** You need to apply custom task metadata extraction logic. (e.g. Set `scheduledTimeMillis` for delayed processing)
*** You need to access additional information (e.g. record headers) for deserialization
====

By default, Decaton assumes messages are produced by DecatonClient where task metadata is stored as link:../protocol/src/main/proto/decaton.proto[TaskMetadataProto] in record headers.
By default, Decaton assumes messages are produced `DecatonClient`, where task metadata are stored as link:../protocol/src/main/proto/decaton.proto[TaskMetadataProto] in record headers.
But Decaton has the capability to consume arbitrary topics other than topics produced by `DecatonClient`.

When you consume a topic not produced by DecatonClient, you can apply custom task metadata extraction logic.
This means you can use Decaton as a drop-in replacement for a vanilla KafkaConsumer to leverage powerful features like deferred completion, delayed processing and so on.

Through this guide, we assume the topic is JSON-serialized and use link:https://github.com/FasterXML/jackson-databind[jackson-databind] for deserialization, but it's trivial to apply to arbitrary formats other than JSON.
Through this guide, we assume the topic is JSON-serialized and use link:https://github.com/FasterXML/jackson-databind[jackson-databind] for deserialization, but it's trivial to consume arbitrary formats other than JSON.

== TaskExtractor

Expand Down

0 comments on commit 8afb496

Please sign in to comment.