Skip to content

Commit

Permalink
Merge branch '2.19'
Browse files Browse the repository at this point in the history
  • Loading branch information
cowtowncoder committed Dec 24, 2024
2 parents 14006dc + 10fc1ba commit 429092f
Show file tree
Hide file tree
Showing 11 changed files with 129 additions and 30 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/dep_build_v2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ jobs:
env:
JAVA_OPTS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- name: Set up JDK
uses: actions/setup-java@99b8673ff64fbf99d8d325f52d9a5bdedb8483e9 # v4.2.1
uses: actions/setup-java@8df1039502a15bceb9433410b1a100fbe190c53b # v4.5.0
with:
distribution: 'temurin'
java-version: ${{ matrix.java_version }}
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/dep_build_v3.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ jobs:
env:
JAVA_OPTS: "-XX:+TieredCompilation -XX:TieredStopAtLevel=1"
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
ref: master
- name: Set up JDK
uses: actions/setup-java@99b8673ff64fbf99d8d325f52d9a5bdedb8483e9 # v4.2.1
uses: actions/setup-java@8df1039502a15bceb9433410b1a100fbe190c53b # v4.5.0
with:
distribution: 'temurin'
java-version: ${{ matrix.java_version }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ jobs:
run: ./mvnw -B -q -ff -ntp test
- name: Publish code coverage
if: ${{ matrix.release_build && github.event_name != 'pull_request' }}
uses: codecov/codecov-action@015f24e6818733317a2da2edd6290ab26238649a # v5.0.7
uses: codecov/codecov-action@7f8b4b4bde536c465e797be725718b88c5d95e0e # v5.1.1
with:
token: ${{ secrets.CODECOV_TOKEN }}
file: ./target/site/jacoco/jacoco.xml
Expand Down
40 changes: 21 additions & 19 deletions avro/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,32 +114,34 @@ and that's about it, for now.

## Avro Logical Types

Following is an extract from [Logical Types](http://avro.apache.org/docs/current/specification/_print/#logical-types) paragraph in
Avro schema specification:
The following is an excerpt from the [Logical Types](https://avro.apache.org/docs/1.11.1/specification/#logical-types) section of
the Avro schema specification:

> A logical type is an Avro primitive or complex type with extra attributes to represent a derived type. The attribute
> `logicalType` is always be present for a logical type, and is a string with the name of one of the logical types
> defined by Avro specification.
> `logicalType` must always be present for a logical type, and is a string with the name of one of the logical types
> listed later in this section. Other attributes may be defined for particular logical types.
Logical types are supported for a limited set of `java.time` classes and for 'java.util.UUID'. See the table below for more details.

Generation of logical types for limited set of `java.time` classes is supported at the moment. See a table bellow.
### Mapping to Logical Types

### Mapping to Logical Type
Mapping to Avro type and logical type involves these steps:

Mapping to Avro type and logical type works in few steps:
1. Serializer for particular Java type (or class) determines a Jackson type where the Java type will be serialized into.
2. `AvroSchemaGenerator` determines corresponding Avro type for that Jackson type.
2. If logical type generation is enabled, then `logicalType` is determined for the above combination of Java type and
Avro type.
1. The serializer for a Java type identifies the Jackson type it will serialize into.
2. The `AvroSchemaGenerator` maps that Jackson type to the corresponding Avro type.
3. `logicalType` value is combination of Java type and Jackson type.

#### Java type to Avro Logical Type mapping

| Java type | Serialization type | Generated Avro schema with Avro type and logical type
| ----------------------------- | ------------------ | -----------------------------------------------------
| `java.time.OffsetDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}`
| `java.time.ZonedDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}`
| `java.time.Instant` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}`
| `java.time.LocalDate` | NumberType.INT | `{"type": "int", "logicalType": "date"}`
| `java.time.LocalTime` | NumberType.INT | `{"type": "int", "logicalType": "time-millis"}`
| `java.time.LocalDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "local-timestamp-millis"}`
| Java type | Jackson type | Generated Avro schema with logical type |
|----------------------------|-----------------|---------------------------------------------------------------------------------------------------|
| `java.time.OffsetDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}` |
| `java.time.ZonedDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}` |
| `java.time.Instant` | NumberType.LONG | `{"type": "long", "logicalType": "timestamp-millis"}` |
| `java.time.LocalDate` | NumberType.INT | `{"type": "int", "logicalType": "date"}` |
| `java.time.LocalTime` | NumberType.INT | `{"type": "int", "logicalType": "time-millis"}` |
| `java.time.LocalDateTime` | NumberType.LONG | `{"type": "long", "logicalType": "local-timestamp-millis"}` |
| `java.util.UUID` (2.19+) | | `{"type": "fixed", "name": "UUID", "namespace": "java.util", "size": 16, "logicalType" : "uuid"}` |

_Provided Avro logical type generation is enabled._

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ public static Schema createEnumSchema(MapperConfig<?> config, JavaType enumType,
* @since 2.11
*/
public static Schema createUUIDSchema() {
return Schema.createFixed("UUID", "", "java.util", 16);
return Schema.createFixed("UUID", null, "java.util", 16);
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,6 @@ public Schema builtAvroSchema() {
// should we construct JavaType for `Character.class` in case of primitive or... ?
return AvroSchemaHelper.numericAvroSchema(NumberType.INT, _type);
}
// [dataformats-binary#179]: need special help with UUIDs, to coerce into Binary
// (could actually be
if (_type.hasRawClass(java.util.UUID.class)) {
return AvroSchemaHelper.createUUIDSchema();
}
AnnotatedClass annotations = _provider.introspectClassAnnotations(_type);
Schema schema = Schema.create(Schema.Type.STRING);
// Stringable classes need to include the type
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package tools.jackson.dataformat.avro.schema;

import java.util.Set;

import tools.jackson.databind.jsonFormatVisitors.JsonStringFormatVisitor;
import tools.jackson.databind.jsonFormatVisitors.JsonValueFormat;

import org.apache.avro.LogicalTypes;
import org.apache.avro.Schema;

/**
* Visitor for {@link java.util.UUID} type. When it is created with logicalTypesEnabled enabled,
* Avro schema is created with logical type uuid.
*
* @since 2.19
*/
public class UUIDVisitor extends JsonStringFormatVisitor.Base
implements SchemaBuilder {
protected boolean _logicalTypesEnabled = false;


public UUIDVisitor(boolean logicalTypesEnabled) {
_logicalTypesEnabled = logicalTypesEnabled;
}

@Override
public void format(JsonValueFormat format) {
// Ideally, we'd recognize UUIDs, Dates etc if need be, here...
}

@Override
public void enumTypes(Set<String> enums) {
// Do nothing
}

@Override
public Schema builtAvroSchema() {
// [dataformats-binary#179]: need special help with UUIDs, to coerce into Binary
// (could actually be
Schema schema = AvroSchemaHelper.createUUIDSchema();
return this._logicalTypesEnabled ? LogicalTypes.uuid().addToSchema(schema) : schema;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,12 @@ public JsonStringFormatVisitor expectStringFormat(JavaType type)
return v;
}

if (type.hasRawClass(java.util.UUID.class)) {
UUIDVisitor v = new UUIDVisitor(this._logicalTypesEnabled);
_builder = v;
return v;
}

StringVisitor v = new StringVisitor(_context, type);
_builder = v;
return v;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
package tools.jackson.dataformat.avro.schema;

import org.junit.Test;

import org.apache.avro.LogicalType;
import org.apache.avro.Schema;

import static org.assertj.core.api.Assertions.assertThat;

public class UUIDVisitor_builtAvroSchemaTest {

@Test
public void testLogicalTypesDisabled() {
// GIVEN
boolean logicalTypesEnabled = false;
UUIDVisitor uuidVisitor = new UUIDVisitor(logicalTypesEnabled);

// WHEN
Schema actualSchema = uuidVisitor.builtAvroSchema();

// THEN
assertThat(actualSchema.getType()).isEqualTo(Schema.Type.FIXED);
assertThat(actualSchema.getFixedSize()).isEqualTo(16);
assertThat(actualSchema.getName()).isEqualTo("UUID");
assertThat(actualSchema.getNamespace()).isEqualTo("java.util");
assertThat(actualSchema.getProp(LogicalType.LOGICAL_TYPE_PROP)).isNull();
}

@Test
public void testLogicalTypesEnabled() {
// GIVEN
boolean logicalTypesEnabled = true;
UUIDVisitor uuidVisitor = new UUIDVisitor(logicalTypesEnabled);

// WHEN
Schema actualSchema = uuidVisitor.builtAvroSchema();

// THEN
assertThat(actualSchema.getType()).isEqualTo(Schema.Type.FIXED);
assertThat(actualSchema.getFixedSize()).isEqualTo(16);
assertThat(actualSchema.getName()).isEqualTo("UUID");
assertThat(actualSchema.getNamespace()).isEqualTo("java.util");
assertThat(actualSchema.getProp(LogicalType.LOGICAL_TYPE_PROP)).isEqualTo("uuid");
}

}
2 changes: 2 additions & 0 deletions release-notes/CREDITS-2.x
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,8 @@ Michal Foksa (MichalFoksa@github)
* Contributed #494: Avro Schema generation: allow mapping Java Enum properties to
Avro String values
(2.18.0)
* Contributed #536: (avro) Add Logical Type support for `java.util.UUID`
(2.19.0)

Hunter Herman (hherman1@github)

Expand Down
5 changes: 5 additions & 0 deletions release-notes/VERSION-2.x
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@ Active maintainers:
=== Releases ===
------------------------------------------------------------------------

2.19.0 (not yet released)

#536: (avro) Add Logical Type support for `java.util.UUID`
(contributed by Michal F)

2.18.2 (27-Nov-2024)

No changes since 2.18.1
Expand Down

0 comments on commit 429092f

Please sign in to comment.