1.17 is not a multiple of 0.0001

[visch]

13:41:07,19 INFO: stderr from second_pipe_cmd: b'jsonschema.exceptions.ValidationError: 1.17 is not a multiple of 0.0001\r\n'
13:41:07,20 INFO: stderr from second_pipe_cmd: b'\r\n'
13:41:07,20 INFO: stderr from second_pipe_cmd: b"Failed validating 'multipleOf' in schema['properties']['balance1']:\r\n"
13:41:07,21 INFO: stderr from second_pipe_cmd: b" {'inclusion': 'available',\r\n"
13:41:07,21 INFO: stderr from second_pipe_cmd: b" 'multipleOf': 0.0001,\r\n"
13:41:07,21 INFO: stderr from second_pipe_cmd: b" 'type': ['null', 'number']}\r\n"
13:41:07,22 INFO: stderr from second_pipe_cmd: b'\r\n'
13:41:07,22 INFO: stderr from second_pipe_cmd: b"On instance['balance1']:\r\n"
13:41:07,23 INFO: stderr from second_pipe_cmd: b' 1.17\r\n'

To replicate

Take this file

{"type": "SCHEMA", "stream": "stream", "schema": {"properties": {"primary_key": {"inclusion": "available", "minimum": -9223372036854775808, "maximum": 9223372036854775807, "type": ["null", "integer"]}, "testfield": {"inclusion": "available", "multipleOf": 0.0001, "type": ["null", "number"]}}, "type": "object"}, "key_properties": []}
{"type": "RECORD", "stream": "stream", "record": {"primary_key":123, "testfield": 1.1700}, "version": 1626119374393, "time_extracted": "2021-07-12T19:49:34.400771Z"}

And pipe it into the target-csv.

My guess this is something to do with decimal precision

[visch]

https://github.com/singer-io/target-csv/pull/25/files works as a hotfix for this.

[hsyyid]

@visch You are correct, adding a flag to allow users to prevent validation is a possible fix. My question is what is your use case that the schema provided doesn't match the record? I believe this behavior is intended to catch bugs in the tap being used.

[visch]

@visch You are correct, adding a flag to allow users to prevent validation is a possible fix. My question is what is your use case that the schema provided doesn't match the record? I believe this behavior is intended to catch bugs in the tap being used.

I have data coming from an Oracle tap that has data in this format. The thing is that the Schema is correct the incorrect portion is how the validation is validating the data.

0.0001 is a multiple of 1.1700 but it comes down to how precise decimal is being (I'm pretty sure, I'd have to dive into a fix I did for a different tap a while back)

[visch]

I have another use case from MSSQL as well with Balance information that has this issue. My workaround right now is in MSSQL just convert it to a string so the target doesn't see it as a number

[hsyyid]

@visch Spent some time looking into this today – looks like it's an issue that Singer Python already addressed by parsing with Decimal – singer-io/singer-python@ddab733#diff-c41c36e8946e31bf6ddedb837eb5a8eb284532852ad8b4afb7f72892afa9ba7cR186

Updating the singer-python version on target-csv seems to have solved this