augur curate passthru can add double quotations

[joverlee521]

Current Behavior

When using the --metadata input, field values with double quotes in them can result in additional double quotes in the output.

Since the metadata is read through csv.DictReader, we can probably tweak this behavior through the csv.Dialect attributes

augur/augur/io/metadata.py

Line 183 in 961cb00

metadata_reader = csv.DictReader(handle, dialect=dialect)

Additional context

This was first observed in nextstrain/mpox#179

[tsibley]

The escaped double quotes ("" for internal quotes) are more correct, no?

[joverlee521]

The escaped double quotes ("" for internal quotes) are more correct, no?

Hmm, yes. I guess the issue comes up when running through augur curate multiple times.
The double quotes keep getting added on with each pass. This is an example of a string with internal quotes that goes through curate 3 times:

SRC VB "Vector", Molecular Biology of Genomes
"SRC VB ""Vector"", Molecular Biology of Genomes"
"SRC VB ""Vector"""", Molecular Biology of Genomes"""
"SRC VB ""Vector"""""""", Molecular Biology of Genomes"""""""

[tsibley]

OH! Yeah, that's a misconfiguration of the parser/producer then.

[tsibley]

I think we want to revisit the change here, i.e. 915672e.

When I said:

OH! Yeah, that's a misconfiguration of the parser/producer then.

What I meant was, "we should fix the configuration of the parser/producer to not endlessly add additional quotes, but we should still use RFC-compliant CSV-like quoting in our TSVs rather than no quoting". With QUOTE_NONE, as added by this change, we've made it impossible for TSV field values to contain tabs or newlines; they'll error out.

Relatedly: #1563 (comment)

[tsibley]

For example:

$ augur curate passthru --output-metadata - <<<'{"x":"\u0009"}'
x
Traceback (most recent call last):
  File "/home/tom/nextstrain/augur/augur/__init__.py", line 69, in run
    return args.__command__.run(args)
  File "/home/tom/nextstrain/augur/augur/curate/__init__.py", line 243, in run
    write_records_to_tsv(validated_output_records, args.output_metadata)
  File "/home/tom/nextstrain/augur/augur/io/metadata.py", line 556, in write_records_to_tsv
    tsv_writer.writerow(first_record)
  File "/home/tom/.conda/envs/augur/lib/python3.9/csv.py", line 154, in writerow
    return self.writer.writerow(self._dict_to_list(rowdict))
_csv.Error: need to escape, but no escapechar set


An error occurred (see above) that has not been properly handled by Augur.
To report this, please open a new issue including the original command and the error above:
    <https://github.com/nextstrain/augur/issues/new/choose>

[joverlee521]

What I meant was, "we should fix the configuration of the parser/producer to not endlessly add additional quotes, but we should still use RFC-compliant CSV-like quoting in our TSVs rather than no quoting".

Ah gotcha. I was wondering why augur filter didn't have the same issue even though it uses QUOTE_MINIMAL and realized that setting to QUOTE_NONE fixed the bug but not the underlying issue. The reader is what adds the extra quotes, not the writer 🤦‍♀️