A wrong CSV mapped as valid?

[aborruso]

Hi,
I have this input CSV, that is wrong: It has two cells in two fields without field name.

But if I run mlr --ifs ";" --csv check input.csv, I have no error.

Is it normal?

Thank you

[johnkerl]

@aborruso as of #225 and #794 we have a field-name deduplicator.

$ cat x
a,b,b,a,a,b
1,2,3,4,5,6

$ mlr --csv check x

$ mlr --c2x cat x
a   1
b   2
b_2 3
a_2 4
a_3 5
b_3 6

In your example the first column named "" is left as-is while the second is deduplicated to "_2":

$ mlr --ifs semicolon --csv check input.csv

$ mlr --ifs semicolon --c2x head -n 1 input.csv
Period   w_44_2021
Zone_101 30.547875563
Zone_102 29.647857121
Zone_103 30.258906489
Zone_104 29.084443251
Zone_105 30.322373863
Zone_106 30.201716127
Zone_107 30.110808549
Zone_108 30.050539392
Zone_109 30.336472375
Zone_110 29.972881168
Zone_111 30.564672832
Zone_112 29.876034876
Zone_113 29.395016018
Zone_114 30.680595852
Zone_115 30.79686536
Zone_116 30.440053095
Zone_117 30.310936498
Zone_201
Zone_202
Zone_203
Zone_204
Zone_205

_2

We could prohibit "" in field names entirely, or have mlr check complain about that, but, in my experience getting CSVs from R/pandas/numpy it's quite common for the first field name to be "".

[aborruso]

Hi @johnkerl thank you.

In my opinion if you have 3 out of 20 columns with no names, these are not duplicates fields, but columns with no name.
Moreover, here, removing duplicates you have still a field with no name (the first).

The CSV is a format in which there are often a lot of ugliness, and I think that it's better to prohibit "" in field names entirely.

Moreover now check is not RFC 4180 compliant:

There maybe an optional header line appearing as the first line of the file with the same format as normal record lines. This
header will contain names corresponding to the fields in the file and should contain the same number of fields as the
records in the rest of the file (the presence or absence of the header line should be indicated via the optional "header"
parameter of this MIME type).

To be able to use check (also as an option) to run RFC 4180 check would be a great thing. Yesterday I wanted to use it to do this check.

[johnkerl]

@aborruso thanks! I would mention though that in the RFC I don't see a requirement that the column names be non-empty (or even unique).

[aborruso]

ok, thank you

[johnkerl]

Not to say we can't add a warning for it in mlr check!!! :)

[johnkerl]

@aborruso what do you think of #1330?

[aborruso]

@aborruso what do you think of #1330?

it's good! Thank you very much