Encoding::UndefinedConversionError from UTF-8 to US-ASCII while minifying JS in production #76

kgrz · 2013-02-04T08:50:15Z

The assets part of my app:

assets {                                                                                                                                                                                       
  serve '/javascripts', from: 'public/javascripts'
  serve '/stylesheets', from: 'public/stylesheets'
  serve '/images', from: 'public/images'

  js :foundation, '/javascripts/foundation.js',[
    '/javascripts/foundation/*.js'
  ]

  js :application, '/javascripts/application.js',[
    '/javascripts/application.js'
  ]

  css :application, '/stylesheets/application.css', [
    '/stylesheets/app.css',
  ]

  js_compression :yui
  css_compression :yui

I'm using slim as my templating engine. The app is a "modular" Sinatra app. Not sure if these change the behaviour.

The offending part is at the line https://github.com/rstacruz/sinatra-assetpack/blob/master/lib/sinatra/assetpack/package.rb#L101

This is unrelated to the yui compressor since changing that to the default compressor also triggers this error. The issue with this error is when compiling/compressing the assets on a per-request basis, the javascripts files won't get sent to the browser:

Changing the entire method to the below snippet corrects this behaviour.

if result.body.respond_to?(:force_encoding)
  result.body.force_encoding("ISO-8859-1").encode("UTF-8")  if result.status == 200                                                                                    
else
  result.body  if result.status == 200
end

Tested this with 1.8.7 (Mac system ruby), 1.9.3-p327, rubinius 1.2.4. Here's the commit in my repo kgrz@4810b71 Will send a PR if this gets tested thoroughly. The tests pass without any error but there are failures which were there before changing this method.

I'm guessing the reason for this happening only in production is that the combined method is not used in other environments.

Another point to note is that this setup ( assets in public directory) cannot used if you want to build the assets first since by default, the built assets get copied into public directory.

Here's the stacktrace:

Encoding::UndefinedConversionError - U+FEFF from UTF-8 to US-ASCII:
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:101:in `encode'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:101:in `block in combined'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:97:in `map'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:97:in `combined'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:83:in `minify'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/class_methods.rb:28:in `block (3 levels) in add_compressed_routes!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/tilt-1.3.3/lib/tilt.rb:127:in `fetch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/class_methods.rb:27:in `block (2 levels) in add_compressed_routes!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:1293:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:1293:in `block in compile!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `[]'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `block (3 levels) in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:876:in `route_eval'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `block (2 levels) in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:897:in `block in process_route'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:895:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:895:in `process_route'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:859:in `block in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:858:in `each'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:858:in `route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:963:in `block in dispatch!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `block in invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:960:in `dispatch!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:794:in `block in call!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `block in invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:794:in `call!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:780:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/xss_header.rb:27:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/path_traversal.rb:16:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/json_csrf.rb:17:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/base.rb:48:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/base.rb:48:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/xss_header.rb:27:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/nulllogger.rb:9:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/head.rb:11:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:124:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:81:in `block in pre_process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in `pre_process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:54:in `process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:39:in `receive_data'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run_machine'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/handler/thin.rb:16:in `run'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/server.rb:264:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/server.rb:141:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/bin/rackup:4:in `<top (required)>'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/bin/rackup:23:in `load'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/bin/rackup:23:in `<main>'

The text was updated successfully, but these errors were encountered:

j15e · 2013-02-04T14:20:22Z

Thanks for the detailed report.

I think the encoding issue might be related to querying via http to get assets, we should get them from the disk directly when possible. Might be related to #68 too

j15e · 2013-02-04T14:21:15Z

(and or detect & deal with encoding properly, I think your fix will work only for files encoded in ISO-8859-1)

kgrz · 2013-02-05T06:04:17Z

ASCII-8BIT to UTF-8 conversion can be done this way without errors. So, files encoded in ISO-8859-1, US-ASCII, ASCII-8BIT and UTF-8 can be converted to UTF-8 encoding without errors.

j15e · 2013-03-08T18:42:10Z

Unfortunately force_encoding is not (1.8.7 compatible](http://ruby-doc.org/core-1.8.7/String.html) and I doubt doing this is the real solution as force_encoding does not change anything. We should add a test I adress this issue with a cleaner solution.

shioyama · 2013-06-05T13:52:50Z

Just bit by the same bug. Any progress?

j15e · 2013-06-05T15:03:37Z

I haven't work on this issue, but would sure accept a pull request on it. I doubt replacing the package snippet with force_encoding would solve all cases.

doits · 2013-07-23T16:32:42Z

just stumbled upon this right now, too. why is it trying to convert from utf-8 to ascii anyway? everything is written in utf-8 on my server (hopefully all vendor js, too), output to clients is utf-8. why does it want to convert from utf-8 to ascii?

kgrz · 2013-07-23T16:56:37Z

@doits I suppose this has to do with backwards compatibility. (@j15e mentioned this earlier :)

j15e · 2013-07-23T17:06:07Z

I do not have any update on this issue but it should indeed be resolved and we have to find a clean solution for this issue.

doits · 2013-07-23T19:28:25Z

I could get rid of the error by setting the correct LANG env var. By doing LANG=en_US.UTF-8 it changed Encoding.default_external to UTF-8. Therefore, assetpack does not try to convert things to ASCII, but to UTF-8 and everything works.

I could do this for apache with phusion passenger by SetEnv LANG en_US.UTF-8.

Soulou · 2013-11-14T22:18:37Z

I'm also encountering this issue, is there something new? (Setting the env var LANG didn't work in my case.

jarredholman · 2014-07-04T05:49:56Z

Can an option be provided to specify which encoding should be used instead of forcing it to the Encoding.default_external?

willmoss · 2014-08-29T22:14:00Z

I encountered this issue today, it would be nice to get a fix thanks

j15e · 2014-09-02T13:52:03Z

Could you report your full stacktrace @willmoss ? Is it exactly the same as the original issue?

papanikge · 2017-06-02T10:53:10Z

Just encountered this in production as well.

j15e · 2017-06-02T14:29:58Z

@papanikge unfortunately, this repo is not maintained anymore. see #197 there is many other better up to date assets manager for sinatra

jamesrwhite mentioned this issue Apr 16, 2014

UTF-8 error jamesrwhite/minicron#64

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encoding::UndefinedConversionError from UTF-8 to US-ASCII while minifying JS in production #76

Encoding::UndefinedConversionError from UTF-8 to US-ASCII while minifying JS in production #76

kgrz commented Feb 4, 2013

j15e commented Feb 4, 2013

j15e commented Feb 4, 2013

kgrz commented Feb 5, 2013

j15e commented Mar 8, 2013

shioyama commented Jun 5, 2013

j15e commented Jun 5, 2013

doits commented Jul 23, 2013

kgrz commented Jul 23, 2013

j15e commented Jul 23, 2013

doits commented Jul 23, 2013

Soulou commented Nov 14, 2013

jarredholman commented Jul 4, 2014

willmoss commented Aug 29, 2014

j15e commented Sep 2, 2014

papanikge commented Jun 2, 2017

j15e commented Jun 2, 2017

Encoding::UndefinedConversionError from UTF-8 to US-ASCII while minifying JS in production #76

Encoding::UndefinedConversionError from UTF-8 to US-ASCII while minifying JS in production #76

Comments

kgrz commented Feb 4, 2013

j15e commented Feb 4, 2013

j15e commented Feb 4, 2013

kgrz commented Feb 5, 2013

j15e commented Mar 8, 2013

shioyama commented Jun 5, 2013

j15e commented Jun 5, 2013

doits commented Jul 23, 2013

kgrz commented Jul 23, 2013

j15e commented Jul 23, 2013

doits commented Jul 23, 2013

Soulou commented Nov 14, 2013

jarredholman commented Jul 4, 2014

willmoss commented Aug 29, 2014

j15e commented Sep 2, 2014

papanikge commented Jun 2, 2017

j15e commented Jun 2, 2017