Loading webpack bundles with hashed filenames

[jhnns]

This may be a dumb question, but I dont find a simple solution for that. How can I load webpack bundles with hashed filenames without adjusting the <script>-tag all the time?

So in my index.html I would like to write

    <script src="/app.js"></script>

which loads something like /app.a76gs0ad5dh45sdh9aa.js.

Is there a simpler solution than parsing the index.html and replacing the script-tag?

[sokra]

The idea behind the whole "hash to file" story is that you want to have caching, but still want to be able to change the file.

Typical implementation is: The html file is generated dynamically and the up-to-date hash is injected.

<script src="/app.<?= hash ?>.js"></script>

This is common because more dynamic stuff is injected here, i.e. session or user info

All app.*.js files can be cached infinite.

---

But there are other solutions possible:

If the html is static (forever) it can contain:

<script src="/app.js"></script>

And the server redirect /app.js to "/app." + hash + ".js".

All app.*.js files and the html files can be cached infinite, but /app.js cannot be cached.

---

A simple but less efficient solution is to only add the hash to the chunks not to the entry bundle. I use this for github hosted files, because I'm to lazy to change (or regenerate) the html file.

[jhnns]

K thx, I think generating the html dynamically is still the best solution. Forwarding all request of app.js to the hashed version is not ideal since the first chunk is typically the biggest.

[jhnns]

Now I'm trying to write a WebpackPlugin that generates an index.html after compilation. Unfortunately fs.writeFile throws an EPERM, Operation not permitted-error. I guess this is because the code is executed by a child-process. What's the "official" way to emit files within a plugin? I also tried compiler.outputFileSystem.writeFile().

[sokra]

I wonder why it throws an error...

You should be able to use the fs.writeFile function. compiler.outputFileSystem.writeFile() is useful if you want the file to be in-memory with webpack-dev-middleware. Attaching the plugin to after-emit may be a good idea.

There is an array compiler.assets with all the files emitted at the end. emit is the last chance to add stuff to it.

compiler.plugin("after-emit", function(compilation, callback) {
  fs.writeFile(xxxPath, yyyString, "utf-8", callback);
});
// node.js only

compiler.plugin("after-emit", function(compilation, callback) {
  this.outputFileSystem.writeFile(xxxPath, yyyBuffer, callback);
});
// any output file system... (i.e. in-memory)

compiler.plugin("emit", function(compilation, callback) {
  compilation.assets[xxxPathRelativeToOutputPath] = {
    source: function() {
      return yyyStringOrBuffer;
    }
  };
});
// any output file system...
// appear in assets list
// parallel with the other assets emitted

[jhnns]

Using fs.writeFile on "after-emit" is fine ... I'll take a closer look tomorrow

[ampedandwired]

Not sure if anything came of this discussion, but I couldn't find any plugins that solved this problem, so I wrote one myself: https://github.com/ampedandwired/html-webpack-plugin

[sokra]

@ampedandwired Great.

Just some notes regarding your plugin:

• Allow to emit HTML for each entry points: "[name].html"
• Use fs.readFile instead of fs.readFileSync
• Allow to pass the template content in addition to the template filename.

[ampedandwired]

Cool, thanks for the feedback @sokra, I'll have a look at these.

[jhnns]

@ampedandwired awesome ... and shame to me. I've written basically the same plugin, but it's closed source. Haven't found the time to remove internal stuff... 😒

[skuridin]

@sokra Can you describe how to embed hash into HTML without plugins?

UPD: found answer but it works only with real fs, not dev server memory fs =( Any ideas how to do it?

plugins: [
    function() {
      this.plugin("done", function(stats) {
        var htmlPath = path.join(__dirname, 'dist', 'index.html');
        var template = fs.readFileSync(htmlPath, 'utf8');
        var html = _.template(template)({hash: stats.hash});
        fs.writeFile(htmlPath, html);
      });
    }
  ],

[DanielJomphe]

I spent a few hours searching for standard solutions, and ended up coding this plugin, which I only run in production mode.
When I'm not in production mode, no hashes are used and everything works, including in watch mode and dev modes (hot or not).

if (process.env.NODE_ENV === 'production') {
    // Hashing of this kind happens only in prod.
    config.output.filename = "bundle-[hash].js"; // In dev's config, it's "bundle.js"
    config.plugins.push(
        // To rewrite stuff like `bundle.js` to `bundle-[hash].js` in files that refer to it, I tried and
        // didn't like the following plugin: https://github.com/ampedandwired/html-webpack-plugin
        // for 2 reasons:
        //    1. because it didn't work with HMR dev mode...
        //    2. because it's centered around HTML files but I also change other files...
        // I hope we can soon find something standard instead of the following hand-coding.
        function () {
            this.plugin("done", function (stats) {
                var replaceInFile = function (filePath, toReplace, replacement) {
                    var replacer = function (match) {
                        console.log('Replacing in %s: %s => %s', filePath, match, replacement);
                        return replacement
                    };
                    var str = fs.readFileSync(filePath, 'utf8');
                    var out = str.replace(new RegExp(toReplace, 'g'), replacer);
                    fs.writeFileSync(filePath, out);
                };

                var hash = stats.hash; // Build's hash, found in `stats` since build lifecycle is done.

                replaceInFile(path.join(config.output.path, 'index.html'),
                    'bundle.js',
                    'bundle-' + hash + '.js'
                );
            });
        }
    );
}

[alex88]

Any updated way to achieve this?

[jhnns]

I think the recommended way is to use the plugin.

[neverfox]

because it didn't work with HMR dev mode...

FWIW, react-redux-starter-kit uses html-webpack-plugin with HMR just fine.

[mezzario]

Came up with this inline plugin:

var Path = require("path");
var FileSystem = require("fs");

var webpackConfig = {
    ...
    plugins: [
        ...
        function() {
            this.plugin("done", function(statsData) {
                var stats = statsData.toJson();

                if (!stats.errors.length) {
                    var htmlFileName = "index.html";
                    var html = FileSystem.readFileSync(Path.join(__dirname, htmlFileName), "utf8");

                    var htmlOutput = html.replace(
                        /<script\s+src=(["'])(.+?)bundle\.js\1/i,
                        "<script src=$1$2" + stats.assetsByChunkName.main[0] + "$1");

                    FileSystem.writeFileSync(
                        Path.join(__dirname, "%your build path%", htmlFileName),
                        htmlOutput);
                }
            });
        }
    ]
};

It will replace bundle.js in html with minified version and copy html to build/dist folder.

Enable only on production.

[jhnns]

@mezzario thanks for sharing 👍

[kospiotr]

The best practice I find is a noncached microloader that serves all required (forever cached) chunks.
So I have very simple entry point:

<!DOCTYPE html>
<html>
<head>
    <title>App</title>
</head>
<body>
    <script src="app.loader.js?cached=false"></script>
</body>
</html>

When app.loader is never cached it can serve frequently changing and no changing dependencies, for example: app, common, vendor, polyfills, styles, buildinfo etc.
Inspired by GWT but didn't find easy solution to implement this yet with webpack.

[sokra]

A noncached microloader would require an additional roundtrip, while script tags don't require it.

Multiple script tags is the better option. You can use the HtmlWebpackPlugin to generate the HTML file.

[joshunger]

@sokra in some scenarios people are publishing a library pointing to latest. You still need a micro loader. Sometimes a customer is using the bundle so you can't change the html as easily. I switched our code to use a 302 redirect. I was hoping aws s3 supports this but I ended up with nginx. How would you solve this scenario?

[arliber]

I used the same approach to concat a "prefetch" link to the head, this is what I did:
`
var path = require("path");
var fs = require("fs");

var webpackConfig = {
...
plugins: [
...
this.plugin('done', function(statsData) {
let stats = statsData.toJson();
let htmlPath = path.join(__dirname, 'public', 'assets', 'index.html');
if (!stats.errors.length) {
let prefetchLinks = '';
stats.assets
.filter(asset => !/.html$/.test(asset.name)) //Filter out the html file
.forEach((asset) => { //Add files to final string
console.log(Wepback: Added ${asset.name} to prefetch);
prefetchLinks += <link rel="prefetch" href="/public/asset/${asset.name}">;
});
var html = fs.readFileSync(htmlPath, 'utf8');
var htmlOutput = html.replace(//, <head> + prefetchLinks); //Append the html after

            fs.writeFileSync(htmlPath, htmlOutput);
        }
    });
]

};
`

[kbukum]

Hello ,

You can use a plugin which created for production bundle. The plugin provides copy and after change the file by the given regex pattern.
Plugin link : https://www.npmjs.com/package/webpack-file-changer

Example Usage:

const fileChanger = new FileChanger({
    move: [{
        from: settings.paths.assets,
        to: settings.paths.www
    }, {
        from: settings.paths.node_modules + "/bootstrap/dist",
        to: settings.paths.www + "/vendor/bootstrap"
    }
    ],
    change: [{
        file: "index.html"
        parameters: {
            "bundle\\.js": "bundle.[hash].js",
            "\\$VERSION": package.version,
            "\\$BUILD_TIME": new Date()
        }
    }
    ]
});

settings.webpack.plugins.push(fileChanger);

[mauleyzaola]

@mezzario solution didn't work for me but gave me great clues. This is what I've ended up with, in case it helps someone.

function() {
        this.plugin("done", function(statsData) {
            const stats = statsData.toJson();

            if (!stats.errors.length) {
                const htmlFileName = "index.html";
                const html = fs.readFileSync(path.join(__dirname, "src", "www", htmlFileName), "utf8");

                // need to read the final js bundle file to replace in the distributed index.html file
                const asset = stats.assets[0];
                let htmlOutput = html.replace("bundle.js", asset.name);

                fs.writeFileSync(
                    path.join(buildPath, htmlFileName),
                    htmlOutput);
            }
        });
    }

[Venryx]

Just wanted to add that if you already have the build's hash being included for the "main" bundles, using HtmlWebpackPlugin, and you want to add that hash to other, manually-added scripts in your html file (e.g. for ones compiled separately using webpack.DllPlugin), you can do the following: (just add it as a plugin after the HtmlWebpackPlugin)

function() {
		this.plugin("compilation", function(compilation) {
			compilation.plugin("html-webpack-plugin-after-html-processing", function(htmlPluginData, callback) {
				// this couldn't find the "manifest.json" asset
				/*var chunk0_filename = compilation.assets["manifest.json"][0];
				var hash = chunk0_filename.match(/?(.+)$/)[1];*/

				// this worked, except it used the "app.js"-specific content-hash, rather than the build's hash which we want
				//var hash = compilation.chunks[0].hash;

				// this gets the build's hash like we want
				var hash = htmlPluginData.html.match(/\.js\?([0-9a-f]+)["']/)[1];
				htmlPluginData.html = htmlPluginData.html.replace("/dll.vendor.js", "/dll.vendor.js?" + hash);
				callback(null, htmlPluginData);
			});
		});
	},

[IAMtheIAM]

If you control your server that hosts the app, such as a node.js app, then you can easily get the hashed filename using the webpack-manifest.json emitted by a webpack plugin.

The NodeJS server side code:

import * as path from 'path';
// import * as fs from 'fs';
const NODE_ENV = process.env.NODE_ENV || 'production',
    isDev          = NODE_ENV === "development" || NODE_ENV === "localhost",
    isQA           = NODE_ENV === "qa" || NODE_ENV === "debug",
    isStaging      = NODE_ENV === "staging",
    isProd         = NODE_ENV === "production",
    localIP        = require('my-local-ip')(),
    root           = path.normalize(__dirname + '/..'),
    devServerPort           = process.env.DEVSERVERPORT || 4000,
    webServerPort           = process.env.WEBSERVERPORT || 5000;

let webpackManifestDev;
if (isDev) {

    // This is a mock of webpack-manifest.json specifically for webpack-dev-server, so the pug template can run without waiting for WDS to generate the manifest and read from it. This needs to be managed manually and be synced with the actual output of webpack. It should not change often though.
    webpackManifestDev = {
        "app"            : {
            "js" : `http://${localIP}:${devServerPort}/js/app.bundle.js`,
            "css": `http://${localIP}:${devServerPort}/js/app.style.css`
        },
        "commons"        : {
            "js": `http://${localIP}:${devServerPort}/js/commons.bundle.js`
        },
        "polyfills"      : {
            "js": `http://${localIP}:${devServerPort}/js/polyfills.bundle.js`
        },
        "vendors"        : {
            "js": `http://${localIP}:${devServerPort}/js/vendors.bundle.js`
        },
        "webpack-runtime": {
            "js": `http://${localIP}:${devServerPort}/js/webpack-runtime.js`
        },
        "assets"         : {
            "loading-animation.css": `http://${localIP}:${devServerPort}/css/loading-animation.css`,
        }
    };
}

/**
 * Get the filename of the hashed bundle from the webpack manifest. This changes any time code within that
 * specific bundle changes.
 */
const webpackManifest = isDev ? webpackManifestDev : require(root + '/wwwroot/webpack-manifest.json');
// or this:
// const webpackManifest = isDev ? null : JSON.parse(fs.readFileSync(path.join(root, '/wwwroot/webpack-manifest.json'), 'utf-8'));

/** This loads the chunk-manifest from webpack ChunkManifestPlugin, which extracts the manifest from
 * the webpack-runtime.js script. Since the webpack-runtime is being inlined anyway, it doesn't affect
 * caching of that file anyway, but its good to separate out the chunk manifest (for lazy loaded chunks)
 * into its own file for use as needed.
 */
const webpackChunkManifest = isDev ? null : require(root + '/wwwroot/' + webpackManifest['webpack-runtime']['json']);

export const config: any = {
    NODE_ENV            : NODE_ENV || 'production',
    isDev               : isDev,
    isQA                : isQA,
    isStaging           : isStaging,
    isProd              : isProd,
    localIP             : localIP,
    root                : root,
    DEVSERVERPORT       : devServerPort,
    WEBSERVERPORT       : webServerPort,
    webpackManifest     : webpackManifest,
    webpackChunkManifest: webpackChunkManifest
    // webpackRuntime : webpackRuntime
};

Then in your index template, you'll need some sort of templating engine. I'm using Pug/Jade

doctype html
html(lang='')
    head
        title MyApp
        meta(charset='utf-8')
        meta(http-equiv='X-UA-Compatible', content='IE=edge')
        meta(name='viewport', content='width=device-width, initial-scale=1, shrink-to-fit=no' id="viewport-meta")
        meta(name='description', content='')
        link(href='https://fonts.googleapis.com/icon?family=Material+Icons', rel='stylesheet')
        link(href = config.webpackManifest['assets']['loading-animation.css'], rel = 'stylesheet' type = 'text/css')
        // base url
        base(href='/')

        script(type='text/javascript').
            window.webpackChunkManifest = !{JSON.stringify(config.webpackChunkManifest)};

        if config.NODE_ENV === 'development'

            // Development mode - use webpack-dev-server

        else if config.NODE_ENV === 'debug' || config.NODE_ENV === 'qa'

            // Development / Debug QA mode - no gzip
            link(href= config.webpackManifest['app']['css'], rel = 'stylesheet' type = 'text/css')

        else if config.NODE_ENV === 'staging' || config.NODE_ENV === 'production'

            // Production mode - gzipped (currently disabled)
            link(href= config.webpackManifest['app']['css'], rel = 'stylesheet' type = 'text/css')

        else

            // No config.NODE_ENV Set - no gzip
            link(href= config.webpackManifest['app']['css'], rel = 'stylesheet' type = 'text/css')


    body(class=config.NODE_ENV)

    div(id="loader-wrapper")
        div(id="loader")
    app

    script(type='text/javascript', src = config.webpackManifest['webpack-runtime']['js'])
    script(type='text/javascript', src = config.webpackManifest['commons']['js'])
    script(type='text/javascript', src = config.webpackManifest['polyfills']['js'])
    script(type='text/javascript', src = config.webpackManifest['vendors']['js'])
    script(type='text/javascript', src = config.webpackManifest['app']['js'])
        
    if config.NODE_ENV === 'development'

        // Webpack mode
        // Local development stuff here


    else if config.NODE_ENV === 'debug' || config.NODE_ENV === 'qa'

        // Debug / QA mode - no gzip
        // Debug / QA development stuff here


    else if config.NODE_ENV === 'staging' || config.NODE_ENV === 'production'

        // Production mode - gzipped (currently disabled)
        // Production development stuff here

    else

        // No config.NODE_ENV Set

In order to generate the webpack manifest, you'll need a webpack plugin to do that. I use assets-manifest-plugin since I like the style of the output manifest.json file. I forked it an added some additional options here, such as creating a property for ALL assets that webpack emits, instead of just assetsByChunkName which is limited to entry chunks and chunks from CommonsChunkPlugin.

You'll also want to enable CommonsChunkPlugin and ChunkManifestPlugin so you can enable long-term caching of all assets. If the asset ever changes, webpack will generate a new hash as you know, so this manifest will update with the hash and then bust the cache.

If the asset doesn't change, the hash stays the same. Make sure you use [chunkhash] in your asset names and not [hash], since the former only changes when the file contents change, and the later changes every build. Lastly you will need MD5 plugin for deterministic hashes.

You can use [chunkhash] in loaders like file-loader and you can use [hash] in CopyWebpackPlugin which strangly enough, acts like [chunkhash] since it only changes when the files change, not on every build.

webpackConfig: {

  plugins [
   /**
    * Webpack plugin that emits a json file with assets paths.
    * This is a fork of the original plugin with new options and functionality added
    *
    * See: https://github.com/IAMtheIAM/assets-webpack-plugin
    */

   new AssetsPlugin({
      path       : Helpers.root(`./${outputDir}`),
      filename   : webpackManifestName,
      prettyPrint: true,
      allAssets  : true, // If true, lists all assets emitted from webpack under property `assets` and `chunks` in the emitted manifest
      forceAbsolutePath: true // If true, prepends a leading / to the allAssets and chunks property values to make them site-root relative. If the value already starts with /, it will not be changed.
   }), 

   /** Plugin: ChunkManifestPlugin
    * Description: Extract chunk mapping into separate JSON file.
    * Allows exporting a JSON file that maps chunk ids to their resulting asset files.
    * Webpack can then read this mapping, assuming it is provided somehow on the client, instead
    * of storing a mapping (with chunk asset hashes) in the bootstrap script, which allows to
    * actually leverage long-term * caching.
    *
    * See: https://github.com/diurnalist/chunk-manifest-webpack-plugin */
   new ChunkManifestPlugin({
      filename        : 'chunk-manifest.json',
      manifestVariable: 'webpackChunkManifest'
   }),

      /**
       * CommonsChunkPlugin
       *
       * See: https://webpack.js.org/plugins/commons-chunk-plugin/
       * See: https://medium.com/webpack/webpack-bits-getting-the-most-out-of-the-commonschunkplugin-ab389e5f318
       */


      /**
       * This scans all entry points for common code, but not async lazy bundles,
       * and puts it in commons.bundle.js. An array of names is the same as running
       * the plugin multiple times for each entry point (name)
       */
      new webpack.optimize.CommonsChunkPlugin({
         name     : 'commons',
         minChunks: 2        
      }),
      /**
       * This only scans lazy bundles, then puts all common code in lazy bundles
       * into one chunk called "commons.lazy-vendor.chunk.js", if it isn't already
       * in commons.bundle.js. If it is already in there, it removes the common code
       * in the lazy bundles and references the modules in commons.bundle.js
       */
      new webpack.optimize.CommonsChunkPlugin({
         async    : 'commons-lazy',
         minChunks: 2 // if it appears in 2 or more chunks, move to commons-lazy bundle
      }),

      /**
       * NOTE: Any name given that isn't in the entry config will be where the
       * webpack runtime code is extracted into, which is what we want. It needs to
       * be the LAST one. Order matters.
       */
      new webpack.optimize.CommonsChunkPlugin({
         name    : 'webpack-runtime',
         filename: isDevServer
            ? 'js/webpack-runtime.js'  // in dev mode, it can't have a [hash] or webpack throws an error, for some reason
            : 'js/webpack-runtime.[hash].js'
      }),

   /**
    * Plugin: WebpackMd5Hash
    * Description: Plugin to replace a standard webpack chunkhash with md5.
    *
    * See: https://www.npmjs.com/package/webpack-md5-hash
    */
   new WebpackMd5Hash(),

   /**
    * Plugin: ExtractTextPlugin
    * Description: Extract SCSS/CSS from bundle into external .css file
    *
    * See: https://github.com/webpack/extract-text-webpack-plugin
    */
   new ExtractTextPlugin({
      filename : 'css/[name].style.[chunkhash].css',
      disable  : false,
      allChunks: true
   }),

      /**
       * Copy Webpack Plugin
       * Copies individual files or entire directories to the build directory.
       *
       *  For more info about available [tokens]
       *  See: https://github.com/webpack/loader-utils
       */

      // This brings over all assets except styles and *.txt files
      new CopyWebpackPlugin([{
         from : 'src/assets',
         to   : '[path][name].[hash].[ext]',
         force: true
      }], {
         copyUnmodified: false,
         ignore        : ['robots.txt', 'humans.txt'] // these shouldn't get hashed or they won't be read properly by bots
      })
  ]
}

Now, every asset is hashed using a deterministic hash based on the file contents, only changes when the contents change, and this allows you to use long term caching successfully. It also works with webpack-dev-server the way I configured above, with the psudo-manifest inlined into the node.js app.

Of course you'll have to configure your server to tell the browser to cache the files long term. In node, that's like this:

app.use(express.static(path.resolve(__dirname, '../wwwroot'), {
    /**
     * Configure Cache Control Headers
     */
    etag      : false,
   // maxAge: 31536000 // 365 days in seconds - use this or use setHeaders function
    setHeaders: function(res, path, stat) {
        res.header('Cache-Control', 'public, max-age=31536000000'); // 1 year in milliseconds
    }
}));

I hope this helps.

[jeffreyducharme]

I use this php to parse the files and load them via wordpress.

if ( !defined('ABSPATH') )
    define('ABSPATH', dirname(__FILE__) . '/');

loadHashedAssets();

function loadHashedAssets()
{

	$dirJS = new DirectoryIterator(ABSPATH.'dist');

	foreach ($dirJS as $file) {

		if (pathinfo($file, PATHINFO_EXTENSION) === 'js') {
			$fullName = basename($file);
			$name = substr(basename($fullName), 0, strpos(basename($fullName), '.'));

			switch($name) {

			    case 'bundle':
			        $deps = array('jQuery');
			        break;

			    default:
                    $deps = array('jQuery');
			        break;

			}

			wp_enqueue_script( $name, '/dist/'. $fullName, $deps, null, true );

		}

		if (pathinfo($file, PATHINFO_EXTENSION) === 'css') {
			$fullName = basename($file);
			$name = substr(basename($fullName), 0, strpos(basename($fullName), '.'));

			wp_enqueue_style( $name,  '/dist/'. $fullName,false, null, 'all' );
		}

	}
};