Invalid byte sequence error when reading files with malformed utf-8 sequences #353

mrorii · 2013-01-13T04:07:15Z

For example, running linguist on this file throws an invalid byte sequence error:

$ wget https://raw.github.com/leoniedu/CongressoAberto/a4785785cb37e8095893dc411f0a030a57fd30f8/CongressoAbertoWP/wp-includes/js/swfupload/swfupload.js
$ linguist swfupload.js
/Users/orii/.rvm/gems/ruby-1.9.3-p286/gems/github-linguist-2.4.0/lib/linguist/blob_helper.rb:209:in `split': invalid byte sequence in UTF-8 (ArgumentError)
        from /Users/orii/.rvm/gems/ruby-1.9.3-p286/gems/github-linguist-2.4.0/lib/linguist/blob_helper.rb:209:in `lines'
        from /Users/orii/.rvm/gems/ruby-1.9.3-p286/gems/github-linguist-2.4.0/lib/linguist/blob_helper.rb:240:in `loc'
        from /Users/orii/.rvm/gems/ruby-1.9.3-p286/gems/github-linguist-2.4.0/bin/linguist:24:in `'
        from /Users/orii/.rvm/gems/ruby-1.9.3-p286/bin/linguist:23:in `load'
        from /Users/orii/.rvm/gems/ruby-1.9.3-p286/bin/linguist:23:in `'

andygrunwald · 2013-08-18T16:41:51Z

I can confirm this error.

gkze · 2013-10-03T14:00:19Z

Is this still a bug?

armw4 · 2013-10-17T20:39:52Z

Same thing for me...

Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/generated.rb:41:in `split': invalid byte sequence in UTF-8 (ArgumentError)
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/generated.rb:41:in `lines'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/generated.rb:100:in `compiled_coffeescript?'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/generated.rb:56:in `generated?'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/generated.rb:12:in `generated?'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/blob_helper.rb:277:in `generated?'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/repository.rb:74:in `block in compute_stats'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/repository.rb:69:in `each'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/repository.rb:69:in `compute_stats'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/lib/linguist/repository.rb:43:in `languages'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/gems/github-linguist-2.9.5/bin/linguist:14:in `<top (required)>'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/bin/linguist:23:in `load'
        from /Users/axw2/.rvm/gems/ruby-2.0.0-p247/bin/linguist:23:in `<main>'

rastkojokic · 2014-01-13T08:05:52Z

I can confirm this also...

ghost · 2014-01-22T00:23:17Z

I received this error as well after manually running linguist on my repo.

andygrunwald · 2014-04-06T18:18:09Z

I still can reproduce this with ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux] on a debian wheezy.

This seems to be the same like #241

pchaigno · 2014-06-11T11:53:31Z

I had the same error several times (last time on pdt-git/public).

I tried using force_encoding() and encode() on line 58 without effect.

arfon · 2014-10-31T22:02:32Z

Confirmed this is still an issue on 1.9.3-p484

arfon · 2015-01-25T15:28:32Z

I'm going to close this. Ruby 2.0 is nearly two years old now and I just don't see us investigating this any time soon sorry.

If anyone else wants to take a stab at this then please be my guest 😄

pchaigno · 2015-01-25T15:43:37Z

@arfon I get this error on the original file of this post with Ruby 2.2:

$ ruby --version
ruby 2.2.0p0 (2014-12-25 revision 49005) [x86_64-linux]

arfon · 2015-01-25T15:53:03Z

Well then.

echatman · 2015-02-02T20:31:09Z

I have also encountered this many times using ruby 2.2. Here is a recent stack trace:

$ linguist swfupload.js 
/var/lib/gems/2.2.0/gems/github-linguist-4.2.7/lib/linguist/blob_helper.rb:266:in `split': invalid byte sequence in UTF-8 (ArgumentError)
    from /var/lib/gems/2.2.0/gems/github-linguist-4.2.7/lib/linguist/blob_helper.rb:266:in `lines'
    from /var/lib/gems/2.2.0/gems/github-linguist-4.2.7/lib/linguist/blob_helper.rb:283:in `loc'
    from /var/lib/gems/2.2.0/gems/github-linguist-4.2.7/bin/linguist:51:in `<top (required)>'
    from /usr/local/bin/linguist:23:in `load'
    from /usr/local/bin/linguist:23:in `<main>'

ApsarasX · 2019-07-06T16:40:28Z

I still can confirm this error.
When I run the following command, my program crashed.

github-linguist Inderxer.asp.txt

BTW, The encoding for Inderxer.asp.txt is GB2312

Some files failed with "invalid byte sequence in UTF-8 (ArgumentError)" when BlobHelper#lines was called. Some problematic files include UTF-16LE samples such as test/fixtures/Data/utf16le. Errors were not present when computing stats from git repositories, since git blobs are always read as ASCII-8BIT and that was working correctly. However, when using FileBlob, encoding could be ASCII-8BIT, UTF-8 or other, depending on the runtime value of Encoding.external_encoding. Tests were not catching the error since they were forcing Encoding.external_encoding to be ASCII-8BIT (introduced in github-linguist#1211). So the error would only be seen in wild usage (see issue github-linguist#353). This commit forces ASCII-8BIT on File.read calls. The error is still present if using memory blobs with other encodings.

Some files failed with "invalid byte sequence in UTF-8 (ArgumentError)" when BlobHelper#lines was called. Some problematic files include UTF-16LE samples such as test/fixtures/Data/utf16le. Errors were not present when computing stats from git repositories, since git blobs are always read as ASCII-8BIT and that was working correctly. However, when using FileBlob, encoding could be ASCII-8BIT, UTF-8 or other, depending on the runtime value of Encoding.external_encoding. Tests were not catching the error since they were forcing Encoding.external_encoding to be ASCII-8BIT (introduced in #1211). So the error would only be seen in wild usage (see issue #353). This commit forces ASCII-8BIT on File.read calls. The error is still present if using memory blobs with other encodings.

* Prepare 7.9.0 release * Put back the v7.8.0 version We need this to ensure the versioning used during testing on GitHub.com doesn't cause caching problems in future * fix errors on non-UTF-8 encodings Some files failed with "invalid byte sequence in UTF-8 (ArgumentError)" when BlobHelper#lines was called. Some problematic files include UTF-16LE samples such as test/fixtures/Data/utf16le. Errors were not present when computing stats from git repositories, since git blobs are always read as ASCII-8BIT and that was working correctly. However, when using FileBlob, encoding could be ASCII-8BIT, UTF-8 or other, depending on the runtime value of Encoding.external_encoding. Tests were not catching the error since they were forcing Encoding.external_encoding to be ASCII-8BIT (introduced in #1211). So the error would only be seen in wild usage (see issue #353). This commit forces ASCII-8BIT on File.read calls. The error is still present if using memory blobs with other encodings. * Decrease expected error count * Set version to 7.9.0 Co-authored-by: Rick Winfrey <[email protected]> Co-authored-by: Santiago M. Mola <[email protected]>

lildude · 2020-03-19T15:27:07Z

This has been resolved by #4730 which is now live on GitHub.com. Closing.

bkeepers added the Triage label Apr 21, 2014

bkeepers removed the Triage label Dec 19, 2014

arfon referenced this issue Dec 23, 2014

Ruby 1.9 compatibility

fca8232

arfon closed this as completed Jan 25, 2015

arfon reopened this Jan 25, 2015

pchaigno mentioned this issue Apr 7, 2015

Mischaracterized as Smalltalk #2310

Closed

pchaigno mentioned this issue Jun 18, 2016

PAWN/SourcePawn misclassified as Pascal #3050

Closed

pchaigno changed the title ~~Invalid byte sequence error in Ruby 1.9 when reading files with malformed utf-8 sequences~~ Invalid byte sequence error when reading files with malformed utf-8 sequences Jun 28, 2016

pchaigno mentioned this issue Feb 15, 2017

C# code being displayed as Smalltalk in language statistics #3474

Closed

lildude added the Bug label Apr 27, 2017

pchaigno mentioned this issue Jul 31, 2017

[WIP] replace non-UTF-8 characters before tokenizing #3749

Closed

smola mentioned this issue Dec 1, 2019

fix errors on non-UTF-8 encodings #4730

Merged

lildude closed this as completed Mar 19, 2020

github-linguist locked as resolved and limited conversation to collaborators Jun 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Invalid byte sequence error when reading files with malformed utf-8 sequences #353

Invalid byte sequence error when reading files with malformed utf-8 sequences #353

mrorii commented Jan 13, 2013

andygrunwald commented Aug 18, 2013

gkze commented Oct 3, 2013

armw4 commented Oct 17, 2013

rastkojokic commented Jan 13, 2014

ghost commented Jan 22, 2014

andygrunwald commented Apr 6, 2014

pchaigno commented Jun 11, 2014

arfon commented Oct 31, 2014

arfon commented Jan 25, 2015

pchaigno commented Jan 25, 2015

arfon commented Jan 25, 2015

echatman commented Feb 2, 2015

ApsarasX commented Jul 6, 2019

lildude commented Mar 19, 2020

Invalid byte sequence error when reading files with malformed utf-8 sequences #353

Invalid byte sequence error when reading files with malformed utf-8 sequences #353

Comments

mrorii commented Jan 13, 2013

andygrunwald commented Aug 18, 2013

gkze commented Oct 3, 2013

armw4 commented Oct 17, 2013

rastkojokic commented Jan 13, 2014

ghost commented Jan 22, 2014

andygrunwald commented Apr 6, 2014

pchaigno commented Jun 11, 2014

arfon commented Oct 31, 2014

arfon commented Jan 25, 2015

pchaigno commented Jan 25, 2015

arfon commented Jan 25, 2015

echatman commented Feb 2, 2015

ApsarasX commented Jul 6, 2019

lildude commented Mar 19, 2020