strip() fails on string read from non-utf8 file

[swadey]

test.txt contains a single character:

\377

I get this error:

julia> f = open("test.txt")
IOStream(<file test.txt>)

julia> x = readline(f)
"\ufffd"

julia> strip(x)
ERROR: BoundsError()
 in getindex at ./array.jl:267
 in getindex at ./utf8.jl:111
 in lstrip at string.jl:1414
 in lstrip at string.jl:1410
 in strip at string.jl:1434

[stevengj]

Can you run readbytes(f) instead so that we can see the actual bytes in the file?

Is it UTF-16, or...?

[swadey]

Sure it's 0xff that causes the issue. It's from the 20 newsgroup dataset.

julia> x = readbytes(f)
2-element Array{Uint8,1}:
 0xff
 0x0a

[JeffBezanson]

see also #1792