Reintroduce gzip file generation

This change re-introduces compressed file generation and parallel file writing.

Gzip file generation was taken out in this PR: sstephenson#589

This was then discussed in this issue: #26

It was decided that compressing assets to maximize compression ratio at compile time was a valuable feature and within the scope of sprockets. This is one possible implementation.

Assets are written to disk in parallel using `Concurrent::Future`, since the gzip file cannot be generated until the original file is written to disk, it must process that file first. Speed impacts of writing files in parallel vary based on the number of assets being written to disk, disk speed, and IO contention.

Gzipping can be turned off at the environment level.

cc @fxn

CHANGELOG.md

@@ -1,3 +1,7 @@
+**Master**
+
+* Reintroduce Gzip file generation for non-binary assets.
+
 **3.4.1** (November 25, 2015)
 
 * PathUtils::Entries will no longer error on an empty directory.

guides/building_an_asset_processing_framework.md

@@ -1,9 +1,32 @@
 # Building an Asset Processing Framework
 
-This guide is for using a Sprockets::Environment to process assets. You would use this class directly if you were building a feature similar to Rail's asset pipeline. If you aren't building an asset processing frameworks, you will want to refer to the [End User Asset Generation](end_user_asset_generation.md) guide instead. For a reference use of `Sprockets::Environemnt` see [sprockets-rails](github.com/rails/sprockets-rails).
+This guide is for using a Sprockets::Environment to process assets. You would use this class directly if you were building a feature similar to Rail's asset pipeline. If you aren't building an asset processing frameworks, you will want to refer to the [End User Asset Generation](end_user_asset_generation.md) guide instead. For a reference use of `Sprockets::Environemnt` see [sprockets-rails](github.com/rails/Sprockets-rails).
+
+## Gzip
+
+By default when Sprockets generates a compiled asset file it will also produce a gzipped copy of that file. Sprockets only gzips non-binary files such as CSS, JavaScript, and SVG files.
+
+For example if Sprockets is generating
+
+```
+application-12345.css
+```
+
+Then it will also generate a compressed copy in
+
+```
+application-12345.css.gz
+```
+
+You can disable this behavior `Sprockets::Environemnt#gzip=` to something falsey for example:
+
+```ruby
+env = Sprockets::Environment.new(".")
+env.gzip = false
+```
 
 ## WIP
 
-This guide is a work in progress. There are many different groups of people who interact with sprockets. Some only need to know directive syntax to put in their asset files, some are building features like the Rails asset pipeline, and some are plugging into sprockets and writing things like preprocessors. The goal of these guides are to provide task specific guidance to make the expected behavior explicit. If you are using sprockets and you find missing information in these guides, please consider submitting a pull request with updated information.
+This guide is a work in progress. There are many different groups of people who interact with Sprockets. Some only need to know directive syntax to put in their asset files, some are building features like the Rails asset pipeline, and some are plugging into Sprockets and writing things like preprocessors. The goal of these guides are to provide task specific guidance to make the expected behavior explicit. If you are using Sprockets and you find missing information in these guides, please consider submitting a pull request with updated information.
 
 These guides live in [guides](/guides).

guides/end_user_asset_generation.md

@@ -11,13 +11,13 @@ write assets in languages like CoffeeScript, Sass and SCSS.
 
 ## Behavior Overview
 
-You can interact through sprockets primarily through directives and file extensions. This section covers how to use each of these things, and the defaults that ship with sprockets.
+You can interact through Sprockets primarily through directives and file extensions. This section covers how to use each of these things, and the defaults that ship with Sprockets.
 
-Since you are likely using sprockets through another framework (like the Rails asset pipeline), there will configuration options you can toggle that will change behavior such as what directories or files get compiled. For that documentation you should see your framework's documentation.
+Since you are likely using Sprockets through another framework (like the Rails asset pipeline), there will configuration options you can toggle that will change behavior such as what directories or files get compiled. For that documentation you should see your framework's documentation.
 
 ### Directives
 
-Directives are special comments in your asset file and the main way of interacting with processors. What kind of interactions? You can use these directives to tell sprockets to load other files, or specify dependencies on other assets.
+Directives are special comments in your asset file and the main way of interacting with processors. What kind of interactions? You can use these directives to tell Sprockets to load other files, or specify dependencies on other assets.
 
 For example, let's say you have custom JavaScript that you've written. You put this javascript in a file called `beta.js`. The javascript makes heavy use of jQuery, so you need to load that before your code executes. You could add this directive to the top of `beta.js`:
 
@@ -70,7 +70,7 @@ Sprockets uses the filename extensions to determine what processors to run on yo
 application.scss
 ```
 
-Then sprockets will by default run the sass processor (which implements scss). The output file will be converted to css.
+Then Sprockets will by default run the sass processor (which implements scss). The output file will be converted to css.
 
 You can specify multiple processors by specifying multiple file extensions. For example you can use Ruby's [ERB template language](http://ruby-doc.org/stdlib-2.2.3/libdoc/erb/rdoc/ERB.html) to embed content in your doc before running the sass processor. To accomplish this you would need to name your file
 
@@ -80,7 +80,7 @@ application.scss.erb
 
 Processors are run from right to left, so in the above example the processor associated with `erb` will be run before the processor associated with `scss` extension.
 
-For a description of the processors that sprockets has by default see the "default processors" section below. Other libraries may register additional processors.
+For a description of the processors that Sprockets has by default see the "default processors" section below. Other libraries may register additional processors.
 
 ## File Order Processing
 
@@ -114,7 +114,7 @@ Or you can use index files to proxy your folders
 
 ### Index files are proxies for folders
 
-In sprockets index files such as `index.js` or `index.css` files inside of a folder will generate a file with the folder's name. So if you have a `foo/index.js` file it will compile down to `foo.js`. This is similar to NPM's behavior of using [folders as modules](https://nodejs.org/api/modules.html#modules_folders_as_modules). It is also somewhat similar to the way that a file in `public/my_folder/index.html` can be reached by a request to `/my_folder`. This means that you cannot directly use an index file. For example this would not work:
+In Sprockets index files such as `index.js` or `index.css` files inside of a folder will generate a file with the folder's name. So if you have a `foo/index.js` file it will compile down to `foo.js`. This is similar to NPM's behavior of using [folders as modules](https://nodejs.org/api/modules.html#modules_folders_as_modules). It is also somewhat similar to the way that a file in `public/my_folder/index.html` can be reached by a request to `/my_folder`. This means that you cannot directly use an index file. For example this would not work:
 
 ```
 <%= asset_path("foo/index.js") %>
@@ -126,13 +126,13 @@ Instead you would need to use:
 <%= asset_path("foo.js") %>
 ```
 
-Why would you want to use this behavior?  It is common behavior where you might want to include an entire directory of files in a top level JavaScript. You can do this in sprockets using `require_tree .`
+Why would you want to use this behavior?  It is common behavior where you might want to include an entire directory of files in a top level JavaScript. You can do this in Sprockets using `require_tree .`
 
 ```
 //= require_tree .
 ```
 
-This has the problem that files are required alphabetically. If your directory has `jquery-ui.js` and `jquery.min.js` then sprockets will require `jquery-ui.js` before `jquery` is required which won't work (because jquery-ui depends on jquery). Previously the only way to get the correct ordering would be to rename your files, something like `0-jquery-ui.js`. Instead of doing that you can use an index file.
+This has the problem that files are required alphabetically. If your directory has `jquery-ui.js` and `jquery.min.js` then Sprockets will require `jquery-ui.js` before `jquery` is required which won't work (because jquery-ui depends on jquery). Previously the only way to get the correct ordering would be to rename your files, something like `0-jquery-ui.js`. Instead of doing that you can use an index file.
 
 For example, if you have an `application.js` and want all the files in the `foo/` folder you could do this:
 
@@ -165,7 +165,7 @@ TODO:
 
 ## Output
 
-This section details the default output of sprockets. This may have been modified by the frameworks you're using, so you will want to verify behavior with their docs.
+This section details the default output of Sprockets. This may have been modified by the frameworks you're using, so you will want to verify behavior with their docs.
 
 Processors and compressors will affect individual file output contents. Refer to the default processors and compressor section how processors for your asset may have modified your file.
 
@@ -177,8 +177,26 @@ TODO: Explain contents, location, and name of a manifest file.
 
 TODO: Explain default fingerprinting/digest behavior
 
+### Gzip
+
+By default when Sprockets generates a compiled asset file it will also produce a gzipped copy of that file. Sprockets only gzips non-binary files such as CSS, javascript, and SVG files.
+
+For example if Sprockets is generating
+
+```
+application-12345.css
+```
+
+Then it will also generate a compressed copy in
+
+```
+application-12345.css.gz
+```
+
+This behavior can be disabled, refer to your framework specific documentation.
+
 ## WIP
 
-This guide is a work in progress. There are many different groups of people who interact with sprockets. Some only need to know directive syntax to put in their asset files, some are building features like the Rails asset pipeline, and some are plugging into sprockets and writing things like preprocessors. The goal of these guides are to provide task specific guidance to make the expected behavior explicit. If you are using sprockets and you find missing information in these guides, please consider submitting a pull request with updated information.
+This guide is a work in progress. There are many different groups of people who interact with Sprockets. Some only need to know directive syntax to put in their asset files, some are building features like the Rails asset pipeline, and some are plugging into Sprockets and writing things like preprocessors. The goal of these guides are to provide task specific guidance to make the expected behavior explicit. If you are using Sprockets and you find missing information in these guides, please consider submitting a pull request with updated information.
 
 These guides live in [guides](/guides).

lib/sprockets/compressing.rb

@@ -51,6 +51,21 @@ def js_compressor
       end
     end
 
+    def gzip?
+      return @gzip if defined?(@gzip)
+      true
+    end
+
+    def skip_gzip?
+      !gzip?
+    end
+
+    # Enable or disable the creation of gzip files,
+    # on by default.
+    def gzip=(gzip)
+      @gzip = gzip
+    end
+
     # Assign a compressor to run on `application/javascript` assets.
     #
     # The compressor object must respond to `compress`.

lib/sprockets/manifest.rb

@@ -1,6 +1,10 @@
 require 'json'
 require 'time'
+
+require 'concurrent/future'
+
 require 'sprockets/manifest_utils'
+require 'sprockets/utils/gzip'
 
 module Sprockets
   # The Manifest logs the contents of assets compiled to a single directory. It
@@ -157,7 +161,9 @@ def compile(*args)
         raise Error, "manifest requires environment for compilation"
       end
 
-      filenames = []
+      filenames              = []
+      concurrent_compressors = []
+      concurrent_writers     = []
 
       find(*args) do |asset|
         files[asset.digest_path] = {
@@ -183,12 +189,26 @@ def compile(*args)
           logger.debug "Skipping #{target}, already exists"
         else
           logger.info "Writing #{target}"
-          asset.write_to target
+          write_file = Concurrent::Future.execute { asset.write_to target }
+          concurrent_writers << write_file
         end
-
         filenames << asset.filename
+
+        next if environment.skip_gzip?
+        gzip = Utils::Gzip.new(asset)
+        next if gzip.cannot_compress?(environment.mime_types)
+
+        if File.exist?("#{target}.gz")
+          logger.debug "Skipping #{target}.gz, already exists"
+        else
+          logger.info "Writing #{target}.gz"
+          concurrent_compressors << Concurrent::Future.execute { write_file.wait; gzip.compress(target) }
+        end
+
       end
-      save
+      concurrent_writers.each(&:wait)
+      concurrent_compressors.each(&:wait)
+      Concurrent::Future.execute { self.save }.wait
 
       filenames
     end
@@ -200,6 +220,7 @@ def compile(*args)
     #
     def remove(filename)
       path = File.join(dir, filename)
+      gzip = "#{path}.gz"
       logical_path = files[filename]['logical_path']
 
       if assets[logical_path] == filename
@@ -208,6 +229,7 @@ def remove(filename)
 
       files.delete(filename)
       FileUtils.rm(path) if File.exist?(path)
+      FileUtils.rm(gzip) if File.exist?(gzip)
 
       save
 

lib/sprockets/utils/gzip.rb

@@ -0,0 +1,56 @@
+module Sprockets
+  module Utils
+    class Gzip
+      # Private: Generates a gzipped file based off of reference file.
+      def initialize(asset)
+        @content_type  = asset.content_type
+        @mtime         = asset.mtime
+        @source        = asset.source
+        @charset       = asset.charset
+      end
+
+      # Private: Returns whether or not an asset can be compressed.
+      #
+      # We want to compress any file that is text based.
+      # You do not want to compress binary
+      # files as they may already be compressed and running them
+      # through a compression algorithm would make them larger.
+      #
+      # Return Boolean.
+      def can_compress?(mime_types)
+        # The "charset" of a mime type is present if the value is
+        # encoded text. We can check this value to see if the asset
+        # can be compressed.
+        #
+        # SVG images are text but do not have
+        # a charset defined, this is special cased.
+        @charset || @content_type == "image/svg+xml".freeze
+      end
+
+      # Private: Opposite of `can_compress?`.
+      #
+      # Returns Boolean.
+      def cannot_compress?(mime_types)
+        !can_compress?(mime_types)
+      end
+
+      # Private: Generates a gzipped file based off of reference asset.
+      #
+      # Compresses the target asset's contents and puts it into a file with
+      # the same name plus a `.gz` extension in the same folder as the original.
+      # Does not modify the target asset.
+      #
+      # Returns nothing.
+      def compress(target)
+        PathUtils.atomic_write("#{target}.gz") do |f|
+          gz = Zlib::GzipWriter.new(f, Zlib::BEST_COMPRESSION)
+          gz.mtime = @mtime.to_i
+          gz.write(@source)
+          gz.close
+        end
+
+        nil
+      end
+    end
+  end
+end

sprockets.gemspec

@@ -11,7 +11,8 @@ Gem::Specification.new do |s|
   s.files = Dir["README.md", "CHANGELOG.md", "LICENSE", "lib/**/*.rb"]
   s.executables = ["sprockets"]
 
-  s.add_dependency "rack", "> 1", "< 3"
+  s.add_dependency "rack",            "> 1", "< 3"
+  s.add_dependency "concurrent-ruby", "~> 1.0"
 
   s.add_development_dependency "closure-compiler", "~> 1.1"
   s.add_development_dependency "coffee-script-source", "~> 1.6"

test/test_manifest.rb

@@ -629,4 +629,32 @@ def teardown
     assert paths.include?("mobile/b.js")
     assert !paths.include?("application.js")
   end
+
+  test "compress non-binary assets" do
+    manifest = Sprockets::Manifest.new(@env, @dir)
+    %W{ gallery.css application.js logo.svg }.each do |file_name|
+      original_path = @env[file_name].digest_path
+      manifest.compile(file_name)
+      assert File.exist?("#{@dir}/#{original_path}.gz"), "Expecting '#{original_path}' to generate gzipped file: '#{original_path}.gz' but it did not"
+    end
+  end
+
+  test "disable file gzip" do
+    @env.gzip = false
+    manifest = Sprockets::Manifest.new(@env, @dir)
+    %W{ gallery.css application.js logo.svg }.each do |file_name|
+      original_path = @env[file_name].digest_path
+      manifest.compile(file_name)
+      refute File.exist?("#{@dir}/#{original_path}.gz"), "Expecting '#{original_path}' to not generate gzipped file: '#{original_path}.gz' but it did"
+    end
+  end
+
+  test "do not compress binary assets" do
+    manifest = Sprockets::Manifest.new(@env, @dir)
+    %W{ blank.gif }.each do |file_name|
+      original_path = @env[file_name].digest_path
+      manifest.compile(file_name)
+      refute File.exist?("#{@dir}/#{original_path}.gz"), "Expecting '#{original_path}' to not generate gzipped file: '#{original_path}.gz' but it did"
+    end
+  end
 end