Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Serialization error: Unknown magic byte! #10

Open
retgoat opened this issue Jun 28, 2017 · 1 comment
Open

Serialization error: Unknown magic byte! #10

retgoat opened this issue Jun 28, 2017 · 1 comment

Comments

@retgoat
Copy link

retgoat commented Jun 28, 2017

Hello!
Guys, I have an error with Serialization of data.
Message serialized successfully pushed to kafka. But on the other side of Kafka another application reads the Kafka queue and deserializes my message.
But deserializarion causes an exception:

greenRiver[ERROR] Processing geronimo_entities offset 20
greenRiver[ERROR] message: Objavro.codenullavro.schema{"type":"record","name":"entities","fields":[{"name":"content","type":["string","null"]},{"name":"content_type_id","type":["int","null"]},{"name":"created_by","type":["int","null"]},{"name":"id","type":["int","null"]},{"name":"inserted_at","type":["string","null"]},{"name":"kind","type":["string","null"]},{"name":"schema_version","type":["string","null"]},{"name":"scope","type":["string","null"]},{"name":"storefront","type":["string","null"]},{"name":"updated_at","type":["string","null"]}],"namespace":"com.foxcommerce.geronimo"}���*k|^��Y_5x��{"title":"Some title foooooo","tags":["tag","another"],"body":"Lorem ipsum","author":"John Doe" ,2017-06-28T09:27:51.0ZBlogPost,2017-06-28T06:31:54.0Z1foo.bar,2017-06-28T09:27:51.0Z���*k|^��Y_5x
greenRiver[ERROR] Error deserializing avro message with key entities_16: error org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
greenRiver[ERROR] 	Objavro.codenullavro.schema{"type":"record","name":"entities","fields":[{"name":"content","type":["string","null"]},{"name":"content_type_id","type":["int","null"]},{"name":"created_by","type":["int","null"]},{"name":"id","type":["int","null"]},{"name":"inserted_at","type":["string","null"]},{"name":"kind","type":["string","null"]},{"name":"schema_version","type":["string","null"]},{"name":"scope","type":["string","null"]},{"name":"storefront","type":["string","null"]},{"name":"updated_at","type":["string","null"]}],"namespace":"com.foxcommerce.geronimo"}���*k|^��Y_5x��{"title":"Some title foooooo","tags":["tag","another"],"body":"Lorem ipsum","author":"John Doe" ,2017-06-28T09:27:51.0ZBlogPost,2017-06-28T06:31:54.0Z1foo.bar,2017-06-28T09:27:51.0Z���*k|^��Y_5x
greenRiver[ERROR] Not processed: geronimo_entities offset: 20
greenRiver[ERROR] Failure during processing geronimo_entities offset 20: org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
greenRiver[ERROR] org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1
greenRiver[ERROR] Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte!

Here is the schema

%{fields: [%{"name" => "content", "type" => ["string", "null"]},
   %{"name" => "content_type_id", "type" => ["int", "null"]},
   %{"name" => "created_by", "type" => ["int", "null"]},
   %{"name" => "id", "type" => ["int", "null"]},
   %{"name" => "inserted_at", "type" => ["string", "null"]},
   %{"name" => "kind", "type" => ["string", "null"]},
   %{"name" => "schema_version", "type" => ["string", "null"]},
   %{"name" => "scope", "type" => ["string", "null"]},
   %{"name" => "storefront", "type" => ["string", "null"]},
   %{"name" => "updated_at", "type" => ["string", "null"]}], name: "entities",
  namespace: "com.foxcommerce.geronimo", type: "record"}

Here is the message

%{"content" => "{\"title\":\"Some title foooooo\",\"tags\":[\"tag\",\"another\"],\"body\":\"Lorem ipsum\",\"author\":\"John Doe\"}",
  "content_type_id" => 1, "created_by" => 4, "id" => 16,
  "inserted_at" => "2017-06-28T09:27:51.0Z", "kind" => "BlogPost",
  "schema_version" => "2017-06-28T06:31:54.0Z", "scope" => "1",
  "storefront" => "foo.bar", "updated_at" => "2017-06-28T09:27:51.0Z"}```

Here is the encoded message

{:ok,
 <<79, 98, 106, 1, 3, 232, 8, 20, 97, 118, 114, 111, 46, 99, 111, 100, 101, 99,
   8, 110, 117, 108, 108, 22, 97, 118, 114, 111, 46, 115, 99, 104, 101, 109, 97,
   172, 8, 123, 34, 116, 121, 112, 101, 34, 58, 34, 114, 101, 99, ...>>}

We're using confluent platform

Any advices?

Many thanks!

@retgoat
Copy link
Author

retgoat commented Jun 29, 2017

It seems encoded message prepended with schema. This is the issue.
The message should be encoded as follows: http://docs.confluent.io/3.1.0/schema-registry/docs/serializer-formatter.html#wire-format

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant