KLIP-3: WITH OUTPUT_SCHEMA

[TobiasHauck]

Description

Design proposal to support WITH (OUTPUT_SCHEMA='<schema-name>')

Related issues:

• Avro Schema conversion from "map" to array of records with namespace "io.confluent.connect.avro" and name "MapEntry" #2619: Avro Schema conversion from "map" to array of records with namespace "io.confluent.connect.avro" and name "MapEntry"
• KSQL should not silently overwrite Schemas in Registry with same name #2427: KSQL should not silently overwrite Schemas in Registry with same name
• KSQL forces Avro field names to upper case #2415: KSQL forces Avro field names to upper case
• Invalid Case expression. Schema for the default clause should be the same as schema for THEN clauses. Result scheme: Schema{INT32}. Schema for default expression is Schema{FLOAT64} #2405: Invalid Case expression. Schema for the default clause should be the same as schema for THEN clauses
• Allow statement to specify the casing (camel case, uppercase, etc) for field names when serialized to output topic #1039: Allow statement to specify the casing (camel case, uppercase, etc) for field names when serialized to output topic
• Simplify schemas in Serde classes #2603: Simplify schemas in Serde classes

Testing done

Reviewer checklist

• Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
• Ensure relevant issues are linked (description should include text like "Fixes #")

[ghost]

It looks like @TobiasHauck hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

[TobiasHauck]

[clabot:check]

[ghost]

@confluentinc It looks like @TobiasHauck just signed our Contributor License Agreement. 👍

Always at your service,

clabot

[apurvam]

Won't think require KSQL to validate that the selected schema is compatible with the actual data being produced? Is there a schema registry API to do this?

[TobiasHauck]

Yes it must be validated if the selected schema is be compatible to the actual data being produced. We can check the original schema against the selected one by using this method:

https://github.com/confluentinc/schema-registry/blob/master/client/src/main/java/io/confluent/kafka/schemaregistry/avro/AvroCompatibilityChecker.java#L84

[TobiasHauck]

@apurvam How shall we continue? Should I adjust the text?

[big-andy-coates]

This KLIP likely overlaps with #3634.

[andrewegel]

Causing CI to break due to the "unknown repository" - This needs to be closed to get around that. If this change is still required, please re-do it.