Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Processing log: deserialization errors for keys #5627

Closed
big-andy-coates opened this issue Jun 16, 2020 · 1 comment · Fixed by #6109
Closed

Processing log: deserialization errors for keys #5627

big-andy-coates opened this issue Jun 16, 2020 · 1 comment · Fixed by #6109
Assignees
Milestone

Comments

@big-andy-coates
Copy link
Contributor

Currently, if a key fails to deserialize the message talks about the value format and probably includes the value in the recordB64.

e.g.

{
 "level": "ERROR",
 "logger": "processing.CSAS_S1_0.KsqlTopic.Source.deserializer",
 "time": 1592255634772,
 "message": {
   "type": 0,
   "deserializationError": {
     "errorMessage": "Error deserializing DELIMITED message from topic: ksql-test",
     "recordB64": "MTAw",
    "cause": [
      "Size of data received by IntegerDeserializer is not 4"
    ],
    "topic": "ksql-test"
  },
  "recordProcessingError": null,
  "productionError": null
}

We need:

  1. the errorMessage to make it clear if its a key or value deserialization error.
  2. the errorMessage to talk about the key format when the key fails to deserialize.
  3. the recordB64 to be the serialized key when the key fails to deserialize.
@big-andy-coates big-andy-coates added this to the 0.11.0 milestone Jun 16, 2020
@big-andy-coates big-andy-coates removed their assignment Jun 24, 2020
@big-andy-coates big-andy-coates removed this from the 0.11.0 milestone Jun 24, 2020
@vcrfxia vcrfxia self-assigned this Aug 26, 2020
@agavra agavra added this to the 0.13.0 milestone Aug 26, 2020
@vcrfxia
Copy link
Contributor

vcrfxia commented Aug 27, 2020

Had a look at this. Turns out the error messages that are thrown on deserialization errors are sane -- it's not the case that ksqlDB is always putting the value format into the error message, but there is a typo in the Kafka deserializer that explains the confusion in the example above:

"Error deserializing DELIMITED message from topic: " + topic, e);

The record contents in the processing log message is the correct one (either key or value based on where the failure was), but there's nothing in the message itself to indicate whether the failure corresponds to the key or value. I've opened a PR to add a new field ("component" -- name subject to change) with value either "key" or "value" to indicate so. We can close this ticket once merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants