Consecutive Tab Delimiters Collapsed To A Single Character

[jetzerb]

When querying a tab-delimited file, if there's an empty column, the rest of the row shifts over one position. Works fine as long as there is data in every column.

echo 'a,b,c,d 1,2,3,4 5,,,8 9,A,B,C' | sed 's/,/\t/g; s/ /\n/g;' |trdsql -ih -id '\t' -oat 'select * from -'

+---+---+---+---+
| a | b | c | d |
+---+---+---+---+
| 1 | 2 | 3 | 4 |
| 5 | 8 |   |   | <-- value from column "d" is output under column "b"
| 9 | A | B | C |
+---+---+---+---+

For completeness, I tested all the ASCII values 1-126 as delimiters. The problem only occurs for a few values:

• 9 (Horizontal Tab)
• 11 (Vertical Tab)
• 12 (Form Feed)
• 32 (Space)

for ord in $(seq 1 126)
do
	[ $ord -eq 10 ] && continue; # linefeed
	[ $ord -eq 13 ] && continue; # carriage return
	[ $ord -eq 34 ] && continue; # double quote
	[ $ord -eq 39 ] && continue; # single quote
	[ $ord -eq 49 ] && continue; # the number 1 (our data below)

	delim=$(printf \\$(printf '%03o' $ord));
	[ $ord -eq 92 ] && delim="\\$delim";

	sedDelim=$delim;
	[ "$delim" = "/" ] && sedDelim="\/";
	[ "$delim" = "&" ] && sedDelim="\&";

	output=$(
		echo "1,,1" | sed "s/,/$sedDelim/g;" |
		trdsql -id "$delim" 'select * from -';
	);

	[ "$output" = "1,,1" ] || echo "$output for ord $ord";
done;

1,1 for ord 9
1,1 for ord 11
1,1 for ord 12
1,1 for ord 32

While I've never encountered a file delimited by VT, FF or Space, I regularly encounter tab delimited files, so this is a fairly important issue.

[noborus]

Thank you for a good issue.

I think this is because TrimLeadingSpace is set to true.
This removed unicode.IsSpace as well as space.
This is not the intended behavior and will be fixed.

[jetzerb]

That fixed the issue for all delimiters except space. But again, I don't think I've ever encountered a space-delimited file, so probably not that big a deal.

[noborus]

Thank you for your reply.
I cannot satisfy everyone, but I will merge pr#86.