Filterx format csv

[bshifter]

add csv formatter function to filterx, helps re-format csv data parsed by parse_csv previously

• also accepts json_array and json_object as input which are the possible output types for parse_csv filter func
• supports optional named argument for delimiter character
• supports optional columns argument with json_object input type for column filtering/ordering
• ignores column names with json_array input
• uses json_object's key ordering when column names are not set (which is seemingly correct order)
• auto double-quotes column values when they contain delimiter character
• auto escapes double quotes when double-quoting a column value

examples:

filterx {
...
  # array to unnamed cols
  x = parse_csv($MSG); # returns json_array as '["foo","bar","baz"]'
  $MSG = format_csv(x, delimiter=","); # custom delimiter
...
}

filterx {
...
  # dict to unnamed cols
  cols = json_array(["future_use1","receive_time","serial_number","type"]);
  x = parse_csv($MSG, columns=cols); # returns json_object as '{"future_use1":"foo"...}'
  $MSG = format_csv(x); # uses json_oject's key ordering
...
}

filterx {
...
  #dict to named cols
  cols = json_array(["future_use1","receive_time","serial_number","type"]);
  x = parse_csv($MSG, columns=cols);
  cols2 = json_array(["type", "receive_time"]);
  $MSG = format_csv(x, delimiter=" ", columns=cols2); # filters and orders columns by cols2 
...
}

[bazsi]

the default delimiter for the parse_csv() function is a space, so I think we should match that here.

[bazsi]

we should return NULL here.

[bazsi]

I think we don't need to push the error here as filterx_expr_eval() would set that already.

[bazsi]

[optional] I'd extract the specializations to separate functions to make this a bit easier to follow.

[bazsi]

we should handle if columns is not a list (return NULL with an error)

[bazsi]

now, I can see that in that case, success is FALSE, so that's what we would do.

I think extracting the type specific specializations would have avoided my misunderstanding.

[bazsi]

we do the same in the epilogue part, as sucess is FALSE in this case.

[bazsi]

this is kind of the "dialect" parameter we already support on the parse_csv() side. the dialect would determine what kind of quotation we produce on the output side.

[bazsi]

This is looking good, I had a few comments, some of which would be worth addressing before merging this.

[jszigetvari]

@bshifter Just an idea:
If we try to format a set of fields to csv, and at a certain point all the remaining fields would be empty strings, then I think it would be safe to not print any more fields or separators from that point on. Thus we could stop the formatting of that message right at that place.

[jszigetvari]

@bshifter Just an idea: If we try to format a set of fields to csv, and at a certain point all the remaining fields would be empty strings, then I think it would be safe to not print any more fields or separators from that point on. Thus we could stop the formatting of that message right at that place.

After discussing this with @bshifter , it seems better to let this approach go.

[alltilla]

This is only a partial review. I will continue the review tomorrow.

[alltilla]

This leaks cols.

[alltilla]

[optional]

Usually we functions are written with early returns in axosyslog, as it is more readable. Can you reorganize this function? Thanks!