-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
changefeedccl: support the various timestamp column types in Avro changefeeds #32472
Comments
Here's what BottledWater, a (now-unmaintained) Postgres-to-Kafka connector, does. And here's the Debezium Postgres-to-Avro type mapping. I'm starting to think ISO 8601-formatted timestamp strings might be the rational thing to do here, since they natively let you specify time zone offsets and various levels of precision, but am open to persuasion in any direction. |
@danhhz I see we now support Also, can we use the same encoding for |
This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with the `experimental_avro` CHANGEFEED format. It also adds a set of "golden" tests for what avro schema each of our sql column types map to. INTERVAL has a corresponding avro logical type, but is left unimplemented for now because our INTERVALs have a larger domain of values than avro's: 8 bytes vs 4. ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for customer demand. These are expected to be safe enough to backport if we need them. Closes cockroachdb#32472 Closes cockroachdb#34421 Closes cockroachdb#34417 Release note (enterprise change): the `CHANGEFEED` `experimental_avro` format now supports sql columns of type DATE, TIME, UUID, INET, and JSONB.
This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with the `experimental_avro` CHANGEFEED format. It also adds a set of "golden" tests for what avro schema each of our sql column types map to. Ditto for some example values for each type. INTERVAL has a corresponding avro logical type, but is left unimplemented for now because our INTERVALs have a larger domain of values than avro's: 8 bytes vs 4. ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for customer demand. These are expected to be safe enough to backport if we need them. Closes cockroachdb#32472 Closes cockroachdb#34421 Closes cockroachdb#34417 Release note (enterprise change): the `CHANGEFEED` `experimental_avro` format now supports sql columns of type DATE, TIME, UUID, INET, and JSONB.
34918: changefeedccl: support more columns types with avro r=nvanbenschoten a=danhhz This adds support for sql DATE, TIME, UUID, INET, and JSONB columns with the `experimental_avro` CHANGEFEED format. It also adds a set of "golden" tests for what avro schema each of our sql column types map to. INTERVAL has a corresponding avro logical type, but is left unimplemented for now because our INTERVALs have a larger domain of values than avro's: 8 bytes vs 4. ARRAY, BIT, and COLLATEDSTRING are a little tricker, so we'll wait for customer demand. These are expected to be safe enough to backport if we need them. Closes #32472 Closes #34421 Closes #34417 Release note (enterprise change): the `CHANGEFEED` `experimental_avro` format now supports sql columns of type DATE, TIME, UUID, INET, and JSONB. Co-authored-by: Daniel Harrison <[email protected]>
We need to support the various timestamp column types (specifically, the
ledger
workload depends ontimestamptz
, but we should figure out mappings for all of the SQL timestamp types we support). This likely needs additional research about common conventions used in the Avro/Confluent/Kafka ecosystem, since AFAIK Avro doesn't have a native/straightforward timestamp representation.The text was updated successfully, but these errors were encountered: