Skip to content

Commit

Permalink
[close #96] Remove all absolute paths from cache
Browse files Browse the repository at this point in the history 
Targeted at sprockets 4.x

The patch in #92 was incomplete, it converted all cache keys to use relative paths, but didn't fully remove all absolute paths from cache contents. The test case was accidentally passing since we didn't check to make sure any of the paths from cache were different from the original ones stored. This patch eliminates these absolute paths stored in the cache:

- [x] dependency paths
- [x] filename
- [x] asset uris
- [x] "included" paths (no idea what these are)
- [x] load paths

This patch works by introducing a utility class URITar which can "compress" or "expand" a given uri. A "compressed" uri can either be one that is relative to the root, or an absolute path if it is outside of the root. An expanded uri will always be an absolute path.

A uri that is relative to the root will be compressed with no beginning slash

    file://relative/to/root/file.js

A uri that is outside of the root will be compressed with a beginning slash

    file:///absolute/path/to/file.js

Even though I'm using "uri" here the URITar class can also operate on file paths without a valid URI scheme name. Like:

    relative/to/root/file.js

and

    /absolute/path/to/file.js

The UnloadedAsset class was moved to it's own file and refactored to use the new URITar class.

Before putting anything in the cache, we will "compress" all uris  and paths so that no absolute paths are in the cache (unless they're not relative to the root which would indicate they're somewhere global e.g. from a gem or shared directory).

Upon loading an asset in memory, we "expand" all uris since sprockets relies on absolute paths for just about everything.

Almost all the business logic is limited to the loader, so the rest of sprockets has no clue if relative or absolute paths were used to build the asset.

We are also compressing the "environment-paths" so that dependencies in different paths will differ. I think this is needed, but the tests don't fail when it's taken out.

ATP
  • Loading branch information
@schneems
schneems committed Aug 19, 2015
1 parent 343ceb5 commit 436e2d6
Show file tree
Hide file tree
Showing 6 changed files with 320 additions and 173 deletions.
2 changes: 1 addition & 1 deletion lib/sprockets.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ module Sprockets
env.version
end
register_dependency_resolver 'environment-paths' do |env|
env.paths
env.paths.map {|path| env.compress_from_root(path) }
end
register_dependency_resolver 'file-digest' do |env, str|
env.file_digest(env.parse_file_digest_uri(str))
Expand Down
9 changes: 9 additions & 0 deletions lib/sprockets/base.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
require 'sprockets/server'
require 'sprockets/source_map_utils'
require 'sprockets/loader'
require 'sprockets/uri_tar'

module Sprockets
# `Base` class for `Environment` and `Cached`.
Expand Down Expand Up @@ -98,5 +99,13 @@ def inspect
"root=#{root.to_s.inspect}, " +
"paths=#{paths.inspect}>"
end

def compress_from_root(uri)
URITar.new(uri, self).compress
end

def expand_from_root(uri)
URITar.new(uri, self).expand
end
end
end
233 changes: 62 additions & 171 deletions lib/sprockets/loader.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,156 +9,10 @@
require 'sprockets/resolve'
require 'sprockets/transformers'
require 'sprockets/uri_utils'
require 'sprockets/unloaded_asset'

module Sprockets

# Internal: Used to parse and store the URI to an unloaded asset
# Generates keys used to store and retrieve items from cache
class UnloadedAsset

# Internal: Initialize object for generating cache keys
#
# uri - A String containing complete URI to a file including schema
# and full path such as
# "file:///Path/app/assets/js/app.js?type=application/javascript"
# env - The current "environment" that assets are being loaded into.
# We need it so we know where the +root+ (directory where sprockets
# is being invoked). We also need for the `file_digest` method,
# since, for some strange reason, memoization is provided by
# overriding methods such as `stat` in the `PathUtils` module.
#
# Returns UnloadedAsset.
def initialize(uri, env)
@uri = uri
@env = env
@root = env.root
@relative_path = get_relative_path_from_uri
@params = nil # lazy loaded
@filename = nil # lazy loaded
end
attr_reader :relative_path, :root, :uri


# Internal: Full file path without schema
#
# This returns a string containing the full path to the asset without the schema.
# Information is loaded lazilly since we want `UnloadedAsset.new(dep, self).relative_path`
# to be fast. Calling this method the first time allocates an array and a hash.
#
# Example
#
# If the URI is `file:///Full/path/app/assets/javascripts/application.js"` then the
# filename would be `"/Full/path/app/assets/javascripts/application.js"`
#
# Returns a String.
def filename
unless @filename
load_file_params
end
@filename
end

# Internal: Hash of param values
#
# This information is generated and used internally by sprockets.
# Known keys include `:type` which store the asset's mime-type, `:id` which is a fully resolved
# digest for the asset (includes dependency digest as opposed to a digest of only file contents)
# and `:pipeline`. Hash may be empty.
#
# Example
#
# If the URI is `file:///Full/path/app/assets/javascripts/application.js"type=application/javascript`
# Then the params would be `{type: "application/javascript"}`
#
# Returns a Hash.
def params
unless @params
load_file_params
end
@params
end

# Internal: Key of asset
#
# Used to retrieve an asset from the cache based on relative path to asset
#
# Returns a String.
def asset_key
"asset-uri:#{relative_path}"
end

# Public: Dependency History key
#
# Used to retrieve an array of "histories" each of which contain a set of stored dependencies
# for a given asset path and filename digest.
#
# A dependency can refer to either an asset i.e. index.js
# may rely on jquery.js (so jquery.js is a dependency), or other factors that may affect
# compilation, such as the VERSION of sprockets (i.e. the environment) and what "processors"
# are used.
#
# For example a history array with one Set of dependencies may look like:
#
# [["environment-version", "environment-paths", "processors:type=text/css&file_type=text/css",
# "file-digest:///Full/path/app/assets/stylesheets/application.css",
# "processors:type=text/css&file_type=text/css&pipeline=self",
# "file-digest:///Full/path/app/assets/stylesheets"]]
#
# This method of asset lookup is used to ensure that none of the dependencies have been modified
# since last lookup. If one of them has, the key will be different and a new entry must be stored.
#
# URI depndencies are later converted to relative paths
#
# Returns a String.
def dependency_history_key
"asset-uri-cache-dependencies:#{relative_path}:#{ @env.file_digest(filename) }"
end

# Internal: Digest key
#
# Used to retrieve a string containing the relative path to an asset based on
# a digest. The digest is generated from dependencies stored via information stored in
# the `dependency_history_key` after each of the "dependencies" is "resolved" for example
# "environment-version" may be resolved to "environment-1.0-3.2.0" for version "3.2.0" of sprockets
#
# Returns a String.
def digest_key(digest)
"asset-uri-digest:#{relative_path}:#{digest}"
end

# Internal: File digest key
#
# The digest for a given file won't change if the path and the stat time hasn't changed
# We can save time by not re-computing this information and storing it in the cache
#
# Returns a String.
def file_digest_key(stat)
"file_digest:#{relative_path}:#{stat}"
end

private
# Internal: Parses uri into filename and params hash
#
# Returns Array with filename and params hash
def load_file_params
@filename, @params = URIUtils.parse_asset_uri(uri)
end

# Internal: Converts uri to a relative path
#
# Returns a relative path if given URI is in the `@env.root` of where sprockets
# is running. Otherwise it returns a string of the absolute path
#
# Returns a String.
def get_relative_path_from_uri
path = uri.sub(/\Afile:\/\//, "".freeze)
if relative_path = PathUtils.split_subpath(root, path)
relative_path
else
path
end
end
end
# The loader phase takes a asset URI location and returns a constructed Asset
# object.
module Loader
Expand All @@ -177,7 +31,7 @@ module Loader
def load(uri)
unloaded = UnloadedAsset.new(uri, self)
if unloaded.params.key?(:id)
unless asset = cache.get(unloaded.asset_key, true)
unless asset = asset_from_cache(unloaded.asset_key)
id = unloaded.params.delete(:id)
uri_without_id = build_asset_uri(unloaded.filename, unloaded.params)
asset = load_from_unloaded(UnloadedAsset.new(uri_without_id, self))
Expand All @@ -199,7 +53,7 @@ def load(uri)
if paths
digest = DigestUtils.digest(resolve_dependencies(paths))
if uri_from_cache = cache.get(unloaded.digest_key(digest), true)
cache.get(UnloadedAsset.new(uri_from_cache, self).asset_key, true)
asset_from_cache(UnloadedAsset.new(uri_from_cache, self).asset_key)
end
else
load_from_unloaded(unloaded)
Expand All @@ -211,6 +65,24 @@ def load(uri)

private

# Internal: Load asset hash from cache
#
# key - A String containing lookup information for an asset
#
# This method converts all "compressed" paths to absolute paths.
# Returns a hash of values representing an asset
def asset_from_cache(key)
asset = cache.get(key, true)
if asset
asset[:uri] = expand_from_root(asset[:uri])
asset[:load_path] = expand_from_root(asset[:load_path])
asset[:filename] = expand_from_root(asset[:filename])
asset[:metadata][:included].map! { |uri| expand_from_root(uri) } if asset[:metadata][:included]
asset[:metadata][:dependencies].map! { |uri| uri.start_with?("file-digest://") ? expand_from_root(uri) : uri } if asset[:metadata][:dependencies]
end
asset
end

# Internal: Loads an asset and saves it to cache
#
# unloaded - An UnloadedAsset
Expand Down Expand Up @@ -304,16 +176,46 @@ def load_from_unloaded(unloaded)
asset[:id] = pack_hexdigest(digest(asset))
asset[:uri] = build_asset_uri(unloaded.filename, unloaded.params.merge(id: asset[:id]))

# Unloaded asset and stored_asset now have a different URI
stored_asset = UnloadedAsset.new(asset[:uri], self)
store_asset(asset, unloaded)
asset
end

# Internal: Save a given asset to the cache
#
# asset - A hash containing values of loaded asset
# unloaded - The UnloadedAsset used to lookup the `asset`
#
# This method converts all absolute paths to "compressed" paths
# which are relative if they're in the root.
def store_asset(asset, unloaded)
# Save the asset in the cache under the new URI
cache.set(stored_asset.asset_key, asset, true)
cached_asset = asset.dup
cached_asset[:uri] = compress_from_root(asset[:uri])
cached_asset[:filename] = compress_from_root(asset[:filename])
cached_asset[:load_path] = compress_from_root(asset[:load_path])

if cached_asset[:metadata]
# Deep dup to avoid modifying `asset`
cached_asset[:metadata] = cached_asset[:metadata].dup
if cached_asset[:metadata][:included]
cached_asset[:metadata][:included] = cached_asset[:metadata][:included].dup
cached_asset[:metadata][:included] = cached_asset[:metadata][:included].map {|uri| compress_from_root(uri) }
end

# Save the new relative path for the digest key of the unloaded asset
cache.set(unloaded.digest_key(asset[:dependencies_digest]), stored_asset.relative_path, true) # wat
if cached_asset[:metadata][:dependencies]
cached_asset[:metadata][:dependencies] = cached_asset[:metadata][:dependencies].dup
cached_asset[:metadata][:dependencies].map! do |uri|
uri.start_with?("file-digest://".freeze) ? compress_from_root(uri) : uri
end
end
end

asset
# Unloaded asset and stored_asset now have a different URI
stored_asset = UnloadedAsset.new(asset[:uri], self)
cache.set(stored_asset.asset_key, cached_asset, true)

# Save the new relative path for the digest key of the unloaded asset
cache.set(unloaded.digest_key(asset[:dependencies_digest]), stored_asset.compressed_path, true)
end


Expand All @@ -335,21 +237,7 @@ def load_from_unloaded(unloaded)
#
# Returns array of resolved dependencies
def resolve_dependencies(uris)
uris.map do |uri|
dependency = resolve_dependency(uri)
case dependency
when Array
dependency.map do |dep|
if dep && dep.is_a?(String)
UnloadedAsset.new(dep, self).relative_path
else
dep
end
end
else
dependency
end
end
uris.map { |uri| resolve_dependency(uri) }
end

# Internal: Retrieves an asset based on its digest
Expand Down Expand Up @@ -389,14 +277,17 @@ def fetch_asset_from_dependency_cache(unloaded, limit = 3)

history = cache.get(key) || []
history.each_with_index do |deps, index|
deps = deps.map { |path| path.start_with?("file-digest://") ? expand_from_root(path) : path }
if asset = yield(deps)
cache.set(key, history.rotate!(index)) if index > 0
return asset
end
end

asset = yield
deps = asset[:metadata][:dependencies]
deps = asset[:metadata][:dependencies].map do |uri|
uri.start_with?("file-digest://") ? compress_from_root(uri) : uri
end
cache.set(key, history.unshift(deps).take(limit))
asset
end
Expand Down
Loading

1 comment on commit 436e2d6

@fakefarm
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👓 📖

Please sign in to comment.