Make CSV encoder configurable

[scMarkus]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Currently the CSV encoder used in multiple places like the S3 sink is using a standard configuration in terms of column separator (,) and quotation escaping ("). This conforms to rfc 4180. Anyway we are currently using a setup where we are using a tab delimited (TSV: \t) variant of it. Furthermore the respective data is read via Apache Spark which uses \ as quotation escape character as its default. We would like to transparently move the logging pipeline towards vector without disrupting the downstream processes which would make said configuration options mandatory to us when writing the data.

Attempted Solutions

As far as I can see there is no documented configuration option to change the described behavior. Looking into the vector source it seams to me as the relevant part is:

vector/lib/codecs/src/encoding/format/csv.rs

Line 80 in 4b80c71

let mut wtr = csv::Writer::from_writer(buffer.writer());

This is using from_writer of the respective CSV library which sets default values. Alternatively a WriterBuilder of the same library could be used which has option to set a delimiter and quotation escape char as well as some other configs.

Proposal

What indeed does exists in the documentation already is the required encoding.csv.fields option. In my mind it would be a nice fit to extend encoding.csv by encoding.csv.delimiter and encoding.csv.quotation_escape and use those to configure the csv::WriterBuilder.

References

feat(codec): Add csv encoding for sinks #16603

Version

vector 0.29.0 (x86_64-unknown-linux-gnu 33b3868 2023-04-11 22:19:54.512366263)

[scMarkus]

Would the proposed approach be reasonable? I may find some time trying to implement this. If so would one be so kind and hint me at how to implement new config options?

[spencergilbert]

I'll see if I can get @vectordotdev/ux-team to respond by the end of the week - we have started adding some options on codecs like decoding.<codec>.<option> (ref).

So I could see the encoding.csv.<options> fitting in with that pattern.

[jszwedko]

Agreed, the options can be nested as suggested like:

encoding.csv.delimiter = ...
encoding.csv.quotes = ...

We'd be happy to see a PR for this if you feel motivated @scMarkus. The options would be added here:

vector/lib/codecs/src/encoding/format/csv.rs

Lines 51 to 60 in c21f892

	pub struct CsvSerializerOptions {
	/// Configures the fields that will be encoded, as well as the order in which they
	/// appear in the output.
	///
	/// If a field is not present in the event, the output will be an empty string.
	///
	/// Values of type `Array`, `Object`, and `Regex` are not supported and the
	/// output will be an empty string.
	pub fields: Vec<ConfigTargetPath>,
	}