Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for customizing compress encodings #657

Closed
wants to merge 9 commits into from
Closed
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -519,6 +519,42 @@ Here's an example `.babelrc` file:
}
```

### Customizing compression encodings


<p><details>
<summary><b>Examples</b></summary>
<ul><li><a href="./examples/with-custom-compress-encodings">With custom compress encodings</a></li></ul>
</details></p>

By default Next.js will compress your core JavaScript assets for `gzip` encoding while it builds(with `next build`) the app.
If you need to add support for more encoding, you could simply add it via `next.config.js`.

Here's an example config which adds supports for both `br` and `gzip` encodings.

```js
var fs = require('fs')
var zlib = require('zlib')
var iltorb = require('iltorb')

module.exports = {
// Return a stream of compressed file for each of the encodings you want
//
// The first listed encoding has the higher priority over others.
// In this case, it'll try to serve the `br` version if the browser supports it.
// Otherwise, it'll server gzipped version.
compress: {
br: function(filePath) {
return fs.createReadStream(filePath).pipe(iltorb.compressStream())
Copy link
Contributor

@nkzawa nkzawa Jan 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should accept Promise instead of ReadableStream since errors might occur in the middle of compression process.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we capture it inside our code. We convert this into a promise.

Why I like this approach is, this is how most of the compression API works. So, it's pretty simple for the user. We handle the rest.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we capture it inside our code.

Actually, it doesn't capture all errors since errors are emitted on each stream. http://stackoverflow.com/questions/21771220/error-handling-with-node-js-streams

Additionally, we should consider the case you do something async inside the callback.

br (path) {
  something(path).then(() => {
    ..
  }).catch((err) => {
    // handle err
  })
}

Copy link
Contributor Author

@arunoda arunoda Jan 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nkzawa so what what do get from the promise?

  • Stream
  • Nothing: Let them to write to the disk.

We can do something like this too:

{
  br: function(inputPath, outputPath) {
    return new Promise()
  }
}

Copy link
Contributor

@nkzawa nkzawa Jan 5, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 That seems the simplest way.

Another idea is to accept only a transform stream.

    gzip: function() {
      return zlib.createGzip()
    }
  • We can catch all errors since other streams are prepared by us.
  • Maybe you can still do any async operations by wrapping them inside a stream.

I'm not so sure if this would cover all cases though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nkzawa I like that approach. And we can also also accept a promise from that function. (Which returns a stream)

},
gzip: function(filePath) {
return fs.createReadStream(filePath).pipe(zlib.createGzip())
}
}
}

```

## Production deployment

To deploy, instead of running `next`, you probably want to build ahead of time. Therefore, building and starting are separate commands:
Expand Down
29 changes: 29 additions & 0 deletions examples/with-custom-compress-encodings/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# Example app with custom compress encodings

## How to use

Download the example (or clone the repo)[https://github.com/zeit/next.js.git]:

```bash
curl https://codeload.github.com/zeit/next.js/tar.gz/master | tar -xz --strip=2 next.js-master/examples/with-custom-compress-encodings
cd with-custom-compress-encodings
```

Install it and run:

```bash
npm install
npm run dev
```

Deploy it to the cloud with [now](https://zeit.co/now) ([download](https://zeit.co/download))

```bash
now
```

## The idea behind the example

By default, Next.js compile your assets with `gzip` compression. But if you want to add support more encodings like `br` or `deflate` you can do it very easily.

This example shows how to add support for `br` and `gzip` compress encodings. For that it uses a config in `next.config.js`.
19 changes: 19 additions & 0 deletions examples/with-custom-compress-encodings/components/Counter.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import React from 'react'

let count = 0

export default class Counter extends React.Component {
add () {
count += 1
this.forceUpdate()
}

render () {
return (
<div>
<p>Count is: {count}</p>
<button onClick={() => this.add()}>Add</button>
</div>
)
}
}
19 changes: 19 additions & 0 deletions examples/with-custom-compress-encodings/components/Header.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import Link from 'next/link'

export default () => (
<div>
<Link href='/'>
<a style={styles.a} >Home</a>
</Link>

<Link href='/about'>
<a style={styles.a} >About</a>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be cool to use styled-jsx

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just wanna talk about this feature in this example.
Anyway, I don't mind using that too.

</Link>
</div>
)

const styles = {
a: {
marginRight: 10
}
}
19 changes: 19 additions & 0 deletions examples/with-custom-compress-encodings/next.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
var fs = require('fs')
var zlib = require('zlib')
var iltorb = require('iltorb')

module.exports = {
// Return a stream of compressed file for each of the encodings you want
//
// The first listed encoding has the higher priority over others.
// In this case, it'll try to serve the `br` version if the browser supports it.
// Otherwise, it'll server gzipped version.
compress: {
br: function (filePath) {
return fs.createReadStream(filePath).pipe(iltorb.compressStream())
},
gzip: function (filePath) {
return fs.createReadStream(filePath).pipe(zlib.createGzip())
}
}
}
17 changes: 17 additions & 0 deletions examples/with-custom-compress-encodings/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"name": "shared-modules",
"version": "1.0.0",
"description": "This example features:",
"main": "index.js",
"scripts": {
"dev": "next",
"build": "next build",
"start": "next start"
},
"dependencies": {
"iltorb": "^1.0.13",
"next": "^2.0.0-beta"
},
"author": "",
"license": "ISC"
}
10 changes: 10 additions & 0 deletions examples/with-custom-compress-encodings/pages/about.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import Header from '../components/Header'
import Counter from '../components/Counter'

export default () => (
<div>
<Header />
<p>This is the about page.</p>
<Counter />
</div>
)
10 changes: 10 additions & 0 deletions examples/with-custom-compress-encodings/pages/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import Header from '../components/Header'
import Counter from '../components/Counter'

export default () => (
<div>
<Header />
<p>HOME PAGE is here!</p>
<Counter />
</div>
)
24 changes: 16 additions & 8 deletions server/build/gzip.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@ import fs from 'fs'
import path from 'path'
import zlib from 'zlib'
import glob from 'glob-promise'
import getConfig from '../config'

export default async function gzipAssets (dir) {
const nextDir = path.resolve(dir, '.next')
const config = getConfig(dir)

const coreAssets = [
path.join(nextDir, 'commons.js'),
Expand All @@ -22,17 +24,23 @@ export default async function gzipAssets (dir) {
const currentChunk = allAssets.splice(0, 10)
if (currentChunk.length === 0) break

await Promise.all(currentChunk.map(gzip))
await Promise.all(currentChunk.map((f) => compress(config, f)))
}
}

export function gzip (filePath) {
const input = fs.createReadStream(filePath)
const output = fs.createWriteStream(`${filePath}.gz`)
export function compress (config, filePath) {
const compressionMap = config.compress || {
gzip: (f) => fs.createReadStream(f).pipe(zlib.createGzip())
}

return new Promise((resolve, reject) => {
const stream = input.pipe(zlib.createGzip()).pipe(output)
stream.on('error', reject)
stream.on('finish', resolve)
const promises = Object.keys(compressionMap).map((type) => {
return new Promise((resolve, reject) => {
const output = fs.createWriteStream(`${filePath}.${type}`)
const stream = compressionMap[type](filePath).pipe(output)
stream.on('error', reject)
stream.on('finish', resolve)
})
})

return Promise.all(promises)
}
18 changes: 12 additions & 6 deletions server/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import {
renderErrorJSON,
sendHTML,
serveStatic,
serveStaticWithGzip
serveStaticWithCompression
} from './render'
import Router from './router'
import HotReloader from './hot-reloader'
Expand All @@ -22,11 +22,17 @@ export default class Server {
this.dir = resolve(dir)
this.dev = dev
this.quiet = quiet
this.renderOpts = { dir: this.dir, dev, staticMarkup }
this.router = new Router()
this.hotReloader = dev ? new HotReloader(this.dir, { quiet }) : null
this.http = null
this.config = getConfig(this.dir)
this.supportedEncodings = Object.keys(this.config.compress || { gzip: true })
this.renderOpts = {
dir: this.dir,
supportedEncodings: this.supportedEncodings,
dev,
staticMarkup
}

this.defineRoutes()
}
Expand Down Expand Up @@ -62,12 +68,12 @@ export default class Server {

this.router.get('/_next/main.js', async (req, res, params) => {
const p = join(this.dir, '.next/main.js')
await serveStaticWithGzip(req, res, p)
await serveStaticWithCompression(req, res, p, this.supportedEncodings)
})

this.router.get('/_next/commons.js', async (req, res, params) => {
const p = join(this.dir, '.next/commons.js')
await serveStaticWithGzip(req, res, p)
await serveStaticWithCompression(req, res, p, this.supportedEncodings)
})

this.router.get('/_next/pages/:path*', async (req, res, params) => {
Expand Down Expand Up @@ -213,9 +219,9 @@ export default class Server {
return renderErrorJSON(err, req, res, this.renderOpts)
}

async serveStaticWithGzip (req, res, path) {
async serveStaticWithCompression (req, res, path) {
this._serveStatic(req, res, () => {
return serveStaticWithGzip(req, res, path)
return serveStaticWithCompression(req, res, path, this.supportedEncodings)
})
}

Expand Down
16 changes: 9 additions & 7 deletions server/render.js
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,9 @@ async function doRender (req, res, pathname, query, {
return '<!DOCTYPE html>' + renderToStaticMarkup(doc)
}

export async function renderJSON (req, res, page, { dir = process.cwd() } = {}) {
export async function renderJSON (req, res, page, { dir = process.cwd(), supportedEncodings } = {}) {
const pagePath = await resolvePath(join(dir, '.next', 'bundles', 'pages', page))
return serveStaticWithGzip(req, res, pagePath)
return serveStaticWithCompression(req, res, pagePath, supportedEncodings)
}

export async function renderErrorJSON (err, req, res, { dir = process.cwd(), dev = false } = {}) {
Expand Down Expand Up @@ -141,15 +141,17 @@ function errorToJSON (err) {
return json
}

export async function serveStaticWithGzip (req, res, path) {
const encoding = accepts(req).encodings(['gzip'])
if (encoding !== 'gzip') {
export async function serveStaticWithCompression (req, res, path, supportedEncodings) {
const acceptingEncodings = accepts(req).encodings()
const encoding = supportedEncodings.find((e) => acceptingEncodings.indexOf(e) >= 0)

if (!encoding) {
return serveStatic(req, res, path)
}

try {
const gzipPath = `${path}.gz`
res.setHeader('Content-Encoding', 'gzip')
const gzipPath = `${path}.${encoding}`
res.setHeader('Content-Encoding', encoding)
await serveStatic(req, res, gzipPath)
} catch (ex) {
if (ex.code === 'ENOENT') {
Expand Down
Loading