support csv files ending with empty last line

[mef]

It is not uncommon to have files ending with an empty line.

Papaparse fails in handling these, returning errors about malformed file. How about adding support for this edge case?

[mholt]

This is intended behavior. An error on an empty last line is probably because you have header row enabled, and my guess is that your header row has more than 1 field. With a header row, all rows need to have the same number of fields.

The main problem is that a line break in CSV is significant. The standard says, "The last record in the file may or may not have an ending line break." - and the problem is that there's no way to know whether or not that file does actually have an ending line break or whether it doesn't and the last field is actually missing values.

I've erred on the side of caution in handling those cases.

Anyway, I guess we could add a config option, but if possible, I'd like to avoid adding more. Do you happen to have any suggestions for knowing whether the file has an ending line break?

[mef]

I haven't looked at the internals of papaParse, so I don't know if it's feasible, but how about adding a check when a line is not parsed correctly:

• if the line is empty and it's the last line of the file, just ignore the error and return the results
  • else: return the error (as it's currently done).

[mholt]

That assumes that the last line is supposed to be empty and isn't missing values. Suppose that a process generating the CSV file was about to write the next line but had some silent error and didn't finish (it happens).

I would be OK with doing what you suggest if it wasn't potentially covering up a problem with the CSV file. Then again, Papa Parse is not a CSV validator (which is a separate task entirely). Right now, the behavior of header is very consistent. If we add this exception, it'll complicate things.

So I'm not sure what to do yet. Maybe having the user assert the empty last line is OK is the best way to do it (via a config setting) -- although it could be annoying. I'll leave this issue open until I'm more decided or hear from more users that this would be the desired behavior.

[mef]

ok

[mef]

With a header row, all rows need to have the same number of fields

It might be worth adding this precision inside the documentation (The Parse Config Object > Config Options > header)

[mholt]

I think that's a good idea!

[bluej100]

skipEmptyLines would be one solution here, no?

[vbenso]

What if the file has an empty line before the header's row? First line empty? Papa parses the rest of the file giving the "TooManyFields" error to every single row.

[mholt]

Oh, yes, skipEmptyLines should do the trick. @vbenso I saw your issue #154 - we'll have to look into that to make sure it's a bug since combining header and skipEmptyLines should do what want.

I think that config option should do the trick then.

[jseabold]

FWIW, this happens not only on files that not end with an empty line but also on files that end with a linebreak as most files do (or should).

[mholt]

@jseabold Ending with an "empty line" and ending with a "line break" are the same thing.

[jseabold]

Sure. I guess I misread the example in the other issue. Isn't every line supposed to end with a newline? Anyway, fwiw, found this confusing for a bit until I came across these issues.

[s2131]

Hi, stumbled upon this topic while looking for different problem. Just have one comment for CSV parsing:
According to this CSV specification https://tools.ietf.org/html/rfc4180 in a valid csv:
"The last record in the file may or may not have an ending line break."

Therefore according to RFC4180 CSV standard whether last line is empty (previous line contains line ending) or last content line ends with just eof does not break CSV compatibility.

[pokoli]

@s2131 IIUC the last line is not empty as it contains the line values but not the line break. That's not an empty line.

[chrislong]

On Unix and Linux systems it has been standard for decades for the last character of a text-type file to be a newline. This behavior fails on such files, and, worse, fails to parse files that comply with the spec. I think the notion that a spec-compliant file might have missing data to be not at all sufficient for breaking the standard.

[pavloDeshko]

The last record in the file may or may not have an ending line break.

So according to format defenition, trailing line ending belongs to the previous record. So its presence (intentional or not) should't imply one more record. But after Papa Paprse it will be there - the empty record, header mode or not.