CSV Parse breaks on comment characters that are also in rows

[jlarmstrongiv]

Describe the bug

I am using [email protected] and it isn’t good at handling comments. For example, if a line starts with #, it should be a comment. Later on, rows with # fail to parse.

To Reproduce

Trying to parse https://download.geonames.org/export/dump/countryInfo.txt with the first 50 lines of comments. The Postal Code Format column has many # characters

const defaultParseOptions: ParseOptions = {
  bom: true,
  cast: true,
  columns: false,
  // comment: "#",
  // comment_no_infix: true,
  delimiter: "  ",
  escape: null,
  groupColumnsByName: false,
  quote: null,
  record_delimiter: ["\n", "\r", "\r\n"],
  relax_quotes: true,
  skip_empty_lines: true,
};

Additional context

Workaround

class RemoveCommentTransform extends Transform {
  override _transform(
    chunk: any,
    _encoding: BufferEncoding,
    callback: TransformCallback,
  ): void {
    const line = String(chunk);
    if (line.trim() !== "" && !line.startsWith("#")) {
      callback(null, line + "\n");
    } else {
      callback(null);
    }
  }
}

  const readableStream = fs.createReadStream(filePath);
  const readlineIterator = readline.createInterface({
    crlfDelay: Number.POSITIVE_INFINITY,
    input: readableStream,
  });
  const readlineStream = Readable.from(readlineIterator);
  const removeCommentTransform = new RemoveCommentTransform();

  const parser = parse({
    ...defaultParseOptions,
    ...parseOptions,
  });

  readlineStream.pipe(removeCommentTransform).pipe(parser);

[wdavidw]

I fetch the referenced file without much trouble unless I missed something. See 28088bc#diff-d920cfdd1103bd1f5bf21e5a0dea02fb7fdef4b77ce5ef5bae5abbc84d4b544d

[jlarmstrongiv]

relax_column_count should be false though, as all the rows have the same number of columns. Those columns have the wrong number of rows due to the incorrect comment detection

[wdavidw]

Hi @jlarmstrongiv, your last comment point me to the right direction. There was a bug with the comment_no_infix option. If you look at the sample now, relax_column_count is out and comment_no_infix is used instead. Version 5.5.5 of csv-parse ships the fix.

[jlarmstrongiv]

@wdavidw thank you! That’s great to hear