You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently AvroSource has its own hand-rolled avro parser, given the class is over 6 years old I assume this is due to it being written before Avro had DataFileReader, etc, with good support for partially reading an avro container file.
One of the downsides of this is that it doesn't share the same CodecFactory infrastructure as avro proper does, so users can't plug in custom CodecFactory instances using CodecFactory.addCodec.
It should be possible to refactor AvroSource to use DataFileReader rather than its own handmade parser, and in doing so would automatically get the benefit of using the normal CodecFactory infrastructure.
Issue Priority
Priority: 3
Issue Component
Component: io-java-avro
The text was updated successfully, but these errors were encountered:
From what I vaguely remember, the existing APIs at the time weren't able to support the splitting protocol that was necessary. As a quick sanity check you could compare the DataFileReader interface at the time to the current one to see if there are new APIs related to seeking to specific offsets and knowing what offset we are at.
Yeah that was what I figured, all the tests pass using DataFileReader so that's cool. I'm going to run a bigger test with our own internal test suite to make sure it works in the real world too, but the test coverage seems pretty complete.
What would you like to happen?
Currently AvroSource has its own hand-rolled avro parser, given the class is over 6 years old I assume this is due to it being written before Avro had DataFileReader, etc, with good support for partially reading an avro container file.
One of the downsides of this is that it doesn't share the same CodecFactory infrastructure as avro proper does, so users can't plug in custom CodecFactory instances using
CodecFactory.addCodec
.It should be possible to refactor AvroSource to use DataFileReader rather than its own handmade parser, and in doing so would automatically get the benefit of using the normal CodecFactory infrastructure.
Issue Priority
Priority: 3
Issue Component
Component: io-java-avro
The text was updated successfully, but these errors were encountered: