fix: Allow value delimiter to be specified for datagen #5332
Conversation
| @@ -229,7 +233,7 @@ private static void usage() { | |||
| .put("value-format", (builder, arg) -> builder.valueFormat = parseFormat(arg)) | |||
| // "format" is maintained for backwards compatibility, but should be removed later. | |||
| .put("format", (builder, argVal) -> builder.valueFormat = parseFormat(argVal)) | |||
| .put("value_delimiter", | |||
| .put("valueDelimiter", | |||
There was a problem hiding this comment.
A note: The config is already weird in that it uses both snake case and camel case for arg names :(
So I had to choose one. I chose camel case because more properties seem to use that than snake case.
There was a problem hiding this comment.
Thanks @purplefox.
Am I right in thinking this is only a partitial fix? It doesn't seem to address the second part of the ticket:
Also, if value_delimiter is set to SPACE or TAB the code would use S or T as a delimiter as it stands.
Or did I miss something?
| @@ -50,8 +54,13 @@ static DataGenProducer getProducer( | |||
| final SerializerFactory<Struct> keySerializerFactory = | |||
| keySerializerFactory(keyFormat, ksqlConfig, srClient); | |||
|
|
|||
| final Map<String, String> formatInfoProperties = new HashMap<>(); | |||
| if (valueDelimiter != null) { | |||
| formatInfoProperties.put(DelimitedFormat.DELIMITER, valueDelimiter.toString()); | |||
There was a problem hiding this comment.
might want to check valueFormat is delimited before setting this, and throw an error if its not.
Should be fixed now. |
| if (valueDelimiter != null && !(valueFormat instanceof DelimitedFormat)) { | ||
| throw new IllegalArgumentException( | ||
| "valueDelimiter can only be specified with delimited format"); | ||
| } |
There was a problem hiding this comment.
nite: consider moving this into the if block below that already checks valueDelimiter != null
purplefox
force-pushed
the
datagen_value_delimiter
branch
from
3a75afb to
3de1fd0
Compare
Description
Fixes: #4481
Now value delimiter can be specified when using datagen with delimited format.
Testing done
Manually tested
Reviewer checklist