2.2.7

[brianlovin]

Status

• WIP
• Ready for review
• Needs testing

Deploy after merge (delete what needn't be deployed)

• api
• hyperion (frontend)
• athena

[spectrum-bot]

	Warnings
⚠️	These modified files do not have Flow enabled: mobile/components/Avatar/style.js shared/stripe/index.js src/components/toasts/index.js src/components/upsell/newUserUpsell.js

New dependencies added: rethinkdb-changefeed-reconnect and toobusy-js.

New dependencies added: helmet and hpp.

toobusy-js

Author: Unknown

Description: Don't fall over when your Node.JS server is too busy. Now without native dependencies!

Homepage: https://github.com/STRML/node-toobusy

Created	over 4 years ago
Last Updated	6 months ago
License	WTFPL
Maintainers	1
Releases	8
Direct Dependencies

README

Is Your Node Process Too Busy?

toobusy-js is a fork of lloyd's node-toobusy that removes native dependencies
in favor of using the unref introduced in node 0.9.1.

This package is a simpler install without native dependencies, but requires node >= 0.9.1.

Node-Toobusy

What happens when your service is overwhelmed with traffic?
Your server can do one of two things:

• Stop working, or...
• Keep serving as many requests as possible

This library helps you do the latter.

How it works

toobusy polls the node.js event loop and keeps track of "lag",
which is long requests wait in node's event queue to be processed.
When lag crosses a threshold, toobusy tells you that you're too busy.
At this point you can stop request processing early
(before you spend too much time on them and compound the problem),
and return a "Server Too Busy" response.
This allows your server to stay responsive under extreme load,
and continue serving as many requests as possible.

installation

npm install toobusy-js

usage

var toobusy = require('toobusy-js'),
    express = require('express');

var app = express();

// middleware which blocks requests when we're too busy
app.use(function(req, res, next) {
  if (toobusy()) {
    res.send(503, "I'm busy right now, sorry.");
  } else {
    next();
  }
});

app.get('/', function(req, res) {
  // processing the request requires some work!
  var i = 0;
  while (i < 1e5) i++;
  res.send("I counted to " + i);
});

var server = app.listen(3000);

process.on('SIGINT', function() {
  server.close();
  // calling .shutdown allows your process to exit normally
  toobusy.shutdown();
  process.exit();
});

tunable parameters

The library exposes a few knobs:

maxLag - This number represents the maximum amount of time in milliseconds that the event queue is behind,
before we consider the process too busy.
interval - The check interval for measuring event loop lag, in ms.

var toobusy = require('toobusy-js');

// Set maximum lag to an aggressive value.
toobusy.maxLag(10);

// Set check interval to a faster value. This will catch more latency spikes
// but may cause the check to be too sensitive.
toobusy.interval(250);

// Get current maxLag or interval setting by calling without parameters.
var currentMaxLag = toobusy.maxLag(), interval = toobusy.interval();

toobusy.onLag(function(currentLag) {
  console.log("Event loop lag detected! Latency: " + currentLag + "ms");
});

The default maxLag value is 70ms, and the default check interval is 500ms.
This allows an "average" server to run at 90-100% CPU
and keeps request latency at around 200ms.
For comparison, a maxLag value of 10ms results in 60-70% CPU usage,
while latency for "average" requests stays at about 40ms.

These numbers are only examples,
and the specifics of your hardware and application can change them drastically,
so experiment!
The default of 70 should get you started.

Events

As of 0.5.0, toobusy-js exposes an onLag method. Pass it a callback to be notified when
a slow event loop tick has been detected.

references

There is nothing new under the sun. (Ecclesiastes 1:9)

Though applying "event loop latency" to node.js was not directly inspired by anyone else's work,
this concept is not new. Here are references to others who apply the same technique:

• Provos, Lever, and Tweedie 2000 - "notes that dropping incoming connections when the server is overloaded improved the shape of the performance curve."

license

WTFPL

now-env

Author: Sergio Daniel Xalambrí

Description: Use `now.json` environment variables while developing

Homepage: https://github.com/zeit/now-env#readme

Created	11 months ago
Last Updated	about 2 months ago
License	MIT
Maintainers	3
Releases	16
Keywords	now, env, zeit, environment, variables and secrets

README

now-env

With the help of this package, you can easily set environment variables for the use in development.

If you're already using a now.json file or the now key in package.json, the env sub property will be assigned to process.env automatically.

In addition, you can store secrets locally, that are specific to the development environment.

IMPORTANT: Do not use this package in production. It should only be used locally, while developing. This is because the secrets that you're using locally shouldn't match the ones you use in production (more secure). In production, you should be adding secrets like this.

Usage

Firstly, install the package from npm:

yarn add now-env

As the last step, register the helper by loading it:

require('now-env')

That's all, you can now check process.env for the environment variables!

If your application is running inside Now cloud then this module is not going to do anything and let Now set your environment variables.

Secrets

Most probably you will want to use secret keys in your now.json file. This module allow you to use them too without worries in development.

Create a now.json with some secret defined as @secret-name, similar to:

{
  "env": {
    "SECRET": "@my-secret-key",
    "ANOTHER_SECRET": "@my-other-secret-key",
    "SECRET_FAIL": "@this-is-not-defined"
  }
}

Then create a now-secrets.json with the secrets names and values.

{
  "@my-secret-key": "keep-it-secret",
  "@my-other-secret-key": "keep-it-secret-too"
}

This file must be ignored to actually keep them secret.

Then when starting your application now-env will read the now.json and get the values from now-secrets.json. If a environment key can't be found in now-secrets.json (or the file doesn't exists) then is going to use the secret name as value, that means if DB_PASS is @db_pass and you don't define it inside now-secrets.json then the value will be @db_pass.

Required Variables

Now supports using the env key as an array of required values you'll need to provide when deploying. This module also allow you to use them in development.

Create a now.json with the array, similar to:

{
  "env": [
    "REQUIRED_KEY",
    "REQUIRED_SECRET"
  ]
}

Then create a now-required.json with the environment keys and values.

{
  "REQUIRED_KEY": "required-value",
  "REQUIRED_SECRET": "@required-secret"
}

You can also use secrets, for that you will need to create a now-secrets.json too.

Then when starting your application now-env will read the now.json and get the values from now-required.json (and now-secrets.json). If a environment key can't be found in now-required.json then is going to throw a reference error.

Migrating from dotenv

If you're already using the dotenv module you can switch to now-env easily:

1. Create a now.json file with the env key with every environment variable
2. Install now-env
3. Change require('dotenv').config() with require('now-env')
4. That's it!

New dependencies added: draft-js and toobusy-js.

rethinkdb-changefeed-reconnect

Author: Jeffrey Wescott

Description: automatically reconnect changefeed processors when connection is lost

Homepage: https://github.com/LearnersGuild/rethinkdb-changefeed-reconnect#readme

Created	over 1 year ago
Last Updated	about 1 month ago
License	ISC
Maintainers	2
Releases	5
Direct Dependencies	babel-runtime
Keywords	rethinkdb, rethinkdbdash, changefeed and connection

README

rethinkdb-changefeed-reconnect

Simple helper module for having RethinkDB changefeed listeners automatically reconnect if they lose their connection (i.e., if the database server goes down or restarts).

Right now, there is a single (default) function exported: processChangeFeedWithAutoReconnect.

Getting Started

1. Install the module in your project

    $ npm install --save rethinkdb-changefeed-reconnect
   

How to Use

Basically, you need to implement 3 functions and configure your options:

1. getFeed(): this should return a RethinkDB .changes() query Promise
2. onFeedItem(item): this is the function that processes each new data item
3. onError(err): this function is where errors will be sent

The available options are:

• changefeedName: used for logging purposes so you can differentiate your different feeds
• attemptDelay: how long (in milliseconds) to wait between reconnect attempts
• maxAttempts: how many reconnect attempts to try before giving up
• silent: whether or not you want the logger to be silenced (default = false)
• logger: must be an object with 4 functions: log, info, warn, and error (default = global.console)

Example

import rethinkdbdash from 'rethinkdbdash'
import processChangefeed from 'rethinkdb-changefeed-reconnect'

const r = rethinkdbdash({servers: {host: 'localhost', port: 28015, db: 'todo'}, silent: true})

processChangefeed(newTasksFeed, handleNewTask, handleError, {
  changefeedName: 'my changefeed',
  attemptDelay: 3000,
  maxAttempts: 3,
  silent: false,
  logger: global.console,
})

function newTasksFeed() {
  r.table('tasks')
    .changes()
    .filter(r.row('old_val').eq(null))
}

function handleNewTask(task) {
  console.log({task})
}

function handleError(err) {
  console.error(err.stack)
}

See also the provided example.

You can run it like so: node example

Then, after you run it, stop your RethinkDB server and watch what gets logged (assuming you passed silent: false in your options).

Notes

This module was developed on top of rethinkdbdash.

Contributing

Read the instructions for contributing.

1. Clone the repository.

2. Run the setup tasks:

    $ npm install
    $ npm test
   

License

See the LICENSE file.

draft-js

Author: Unknown

Description: A React framework for building text editors.

Homepage: http://draftjs.org/

Created	about 2 years ago
Last Updated	2 months ago
License	BSD-3-Clause
Maintainers	4
Releases	21
Direct Dependencies	fbjs, immutable and object-assign
Keywords	draftjs, editor, react and richtext

README

Draft.js

Draft.js is a JavaScript rich text editor framework, built for React and
backed by an immutable model.

• Extensible and Customizable: We provide the building blocks to enable
  the creation of a broad variety of rich text composition experiences, from
  simple text styles to embedded media.
• Declarative Rich Text: Draft.js fits seamlessly into
  React applications,
  abstracting away the details of rendering, selection, and input behavior with a
  familiar declarative API.
• Immutable Editor State: The Draft.js model is built
  with immutable-js, offering
  an API with functional state updates and aggressively leveraging data persistence
  for scalable memory usage.

Learn how to use Draft.js in your own project.

API Notice

Before getting started, please be aware that we recently changed the API of
Entity storage in Draft. The latest version, v0.10.0, supports both the old
and new API. Following that up will be v0.11.0 which will remove the old API.
If you are interested in helping out, or tracking the progress, please follow
issue 839.

Getting Started

Currently Draft.js is distributed via npm. It depends on React and React DOM which must also be installed.

npm install --save draft-js react react-dom

or

yarn add draft-js react react-dom

Using Draft.js

import React from 'react';
import ReactDOM from 'react-dom';
import {Editor, EditorState} from 'draft-js';

class MyEditor extends React.Component {
  constructor(props) {
    super(props);
    this.state = {editorState: EditorState.createEmpty()};
    this.onChange = (editorState) => this.setState({editorState});
  }
  render() {
    return (
      <Editor editorState={this.state.editorState} onChange={this.onChange} />
    );
  }
}

ReactDOM.render(
  <MyEditor />,
  document.getElementById('container')
);

Because Draft.js supports unicode, you must have the following meta tag in the <head> </head> block of your HTML file:

<meta charset="utf-8" />

Further examples of how Draft.js can be used are provided below.

Examples

Visit http://draftjs.org/ to try out a simple rich editor example.

The repository includes a variety of different editor examples to demonstrate
some of the features offered by the framework.

To run the examples, first build Draft.js locally:

git clone https://github.com/facebook/draft-js.git
cd draft-js
npm install
npm run build

then open the example HTML files in your browser.

Draft.js is used in production on Facebook, including status and
comment inputs, Notes, and
messenger.com.

Browser Support

IE / Edge	Firefox	Chrome	Safari	iOS Safari	Chrome for Android
IE11, Edge [1, 2]	last 2 versions	last 2 versions	last 2 versions	not fully supported [3]	not fully supported [3]

[1] May need a shim or a polyfill for some syntax used in Draft.js (docs).

[2] IME inputs have known issues in these browsers, especially Korean (docs).

[3] There are known issues with mobile browsers, especially on Android (docs).

Resources and Ecosystem

Check out this curated list of articles and open-sourced projects/utilities: Awesome Draft-JS.

Discussion and Support

Join our Slack team!

Contribute

We actively welcome pull requests. Learn how to
contribute.

License

Draft.js is BSD Licensed.
We also provide an additional patent grant.

Examples provided in this repository and in the documentation are separately
licensed.

helmet

Author: Adam Baldwin

Description: help secure Express/Connect apps with various HTTP headers

Homepage: https://helmetjs.github.io/

Created	about 6 years ago
Last Updated	about 1 month ago
License	MIT
Maintainers	2
Releases	75
Direct Dependencies	dns-prefetch-control, dont-sniff-mimetype, expect-ct, frameguard, helmet-csp, hide-powered-by, hpkp, hsts, ienoopen, nocache, referrer-policy and x-xss-protection
Keywords	security, headers, express, connect, x-frame-options, x-powered-by, csp, hsts and clickjack

README

Helmet

Helmet helps you secure your Express apps by setting various HTTP headers. It's not a silver bullet, but it can help!

Looking for a version of Helmet that supports the Koa framework?

Quick start

First, run npm install helmet --save for your app. Then, in an Express (or Connect) app:

var express = require('express')
var helmet = require('helmet')

var app = express()

app.use(helmet())

// ...

It's best to use Helmet early in your middleware stack so that its headers are sure to be set.

You can also use its pieces individually:

app.use(helmet.noCache())
app.use(helmet.frameguard())

You can disable a middleware that's normally enabled by default. This will disable frameguard but include the other defaults.

app.use(helmet({
  frameguard: false
}))

You can also set options for a middleware. Setting options like this will always include the middleware, whether or not it's a default.

app.use(helmet({
  frameguard: {
    action: 'deny'
  }
}))

If you're using Express 3, make sure these middlewares are listed before app.router.

How it works

Helmet is a collection of 12 smaller middleware functions that set HTTP headers. Running app.use(helmet()) will not include all of these middleware functions by default.

Module	Default?
contentSecurityPolicy for setting Content Security Policy
expectCt for handling Certificate Transparency
dnsPrefetchControl controls browser DNS prefetching	✓
frameguard to prevent clickjacking	✓
hidePoweredBy to remove the X-Powered-By header	✓
hpkp for HTTP Public Key Pinning
hsts for HTTP Strict Transport Security	✓
ieNoOpen sets X-Download-Options for IE8+	✓
noCache to disable client-side caching
noSniff to keep clients from sniffing the MIME type	✓
referrerPolicy to hide the Referer header
xssFilter adds some small XSS protections	✓

You can see more in the documentation.

toobusy-js

Author: Unknown

Description: Don't fall over when your Node.JS server is too busy. Now without native dependencies!

Homepage: https://github.com/STRML/node-toobusy

Created	over 4 years ago
Last Updated	6 months ago
License	WTFPL
Maintainers	1
Releases	8
Direct Dependencies

README

Is Your Node Process Too Busy?

toobusy-js is a fork of lloyd's node-toobusy that removes native dependencies
in favor of using the unref introduced in node 0.9.1.

This package is a simpler install without native dependencies, but requires node >= 0.9.1.

Node-Toobusy

What happens when your service is overwhelmed with traffic?
Your server can do one of two things:

• Stop working, or...
• Keep serving as many requests as possible

This library helps you do the latter.

How it works

toobusy polls the node.js event loop and keeps track of "lag",
which is long requests wait in node's event queue to be processed.
When lag crosses a threshold, toobusy tells you that you're too busy.
At this point you can stop request processing early
(before you spend too much time on them and compound the problem),
and return a "Server Too Busy" response.
This allows your server to stay responsive under extreme load,
and continue serving as many requests as possible.

installation

npm install toobusy-js

usage

var toobusy = require('toobusy-js'),
    express = require('express');

var app = express();

// middleware which blocks requests when we're too busy
app.use(function(req, res, next) {
  if (toobusy()) {
    res.send(503, "I'm busy right now, sorry.");
  } else {
    next();
  }
});

app.get('/', function(req, res) {
  // processing the request requires some work!
  var i = 0;
  while (i < 1e5) i++;
  res.send("I counted to " + i);
});

var server = app.listen(3000);

process.on('SIGINT', function() {
  server.close();
  // calling .shutdown allows your process to exit normally
  toobusy.shutdown();
  process.exit();
});

tunable parameters

The library exposes a few knobs:

maxLag - This number represents the maximum amount of time in milliseconds that the event queue is behind,
before we consider the process too busy.
interval - The check interval for measuring event loop lag, in ms.

var toobusy = require('toobusy-js');

// Set maximum lag to an aggressive value.
toobusy.maxLag(10);

// Set check interval to a faster value. This will catch more latency spikes
// but may cause the check to be too sensitive.
toobusy.interval(250);

// Get current maxLag or interval setting by calling without parameters.
var currentMaxLag = toobusy.maxLag(), interval = toobusy.interval();

toobusy.onLag(function(currentLag) {
  console.log("Event loop lag detected! Latency: " + currentLag + "ms");
});

The default maxLag value is 70ms, and the default check interval is 500ms.
This allows an "average" server to run at 90-100% CPU
and keeps request latency at around 200ms.
For comparison, a maxLag value of 10ms results in 60-70% CPU usage,
while latency for "average" requests stays at about 40ms.

These numbers are only examples,
and the specifics of your hardware and application can change them drastically,
so experiment!
The default of 70 should get you started.

Events

As of 0.5.0, toobusy-js exposes an onLag method. Pass it a callback to be notified when
a slow event loop tick has been detected.

references

There is nothing new under the sun. (Ecclesiastes 1:9)

Though applying "event loop latency" to node.js was not directly inspired by anyone else's work,
this concept is not new. Here are references to others who apply the same technique:

• Provos, Lever, and Tweedie 2000 - "notes that dropping incoming connections when the server is overloaded improved the shape of the performance curve."

license

WTFPL

toobusy-js

Author: Unknown

Description: Don't fall over when your Node.JS server is too busy. Now without native dependencies!

Homepage: https://github.com/STRML/node-toobusy

Created	over 4 years ago
Last Updated	6 months ago
License	WTFPL
Maintainers	1
Releases	8
Direct Dependencies

README

Is Your Node Process Too Busy?

toobusy-js is a fork of lloyd's node-toobusy that removes native dependencies
in favor of using the unref introduced in node 0.9.1.

This package is a simpler install without native dependencies, but requires node >= 0.9.1.

Node-Toobusy

What happens when your service is overwhelmed with traffic?
Your server can do one of two things:

• Stop working, or...
• Keep serving as many requests as possible

This library helps you do the latter.

How it works

toobusy polls the node.js event loop and keeps track of "lag",
which is long requests wait in node's event queue to be processed.
When lag crosses a threshold, toobusy tells you that you're too busy.
At this point you can stop request processing early
(before you spend too much time on them and compound the problem),
and return a "Server Too Busy" response.
This allows your server to stay responsive under extreme load,
and continue serving as many requests as possible.

installation

npm install toobusy-js

usage

var toobusy = require('toobusy-js'),
    express = require('express');

var app = express();

// middleware which blocks requests when we're too busy
app.use(function(req, res, next) {
  if (toobusy()) {
    res.send(503, "I'm busy right now, sorry.");
  } else {
    next();
  }
});

app.get('/', function(req, res) {
  // processing the request requires some work!
  var i = 0;
  while (i < 1e5) i++;
  res.send("I counted to " + i);
});

var server = app.listen(3000);

process.on('SIGINT', function() {
  server.close();
  // calling .shutdown allows your process to exit normally
  toobusy.shutdown();
  process.exit();
});

tunable parameters

The library exposes a few knobs:

maxLag - This number represents the maximum amount of time in milliseconds that the event queue is behind,
before we consider the process too busy.
interval - The check interval for measuring event loop lag, in ms.

var toobusy = require('toobusy-js');

// Set maximum lag to an aggressive value.
toobusy.maxLag(10);

// Set check interval to a faster value. This will catch more latency spikes
// but may cause the check to be too sensitive.
toobusy.interval(250);

// Get current maxLag or interval setting by calling without parameters.
var currentMaxLag = toobusy.maxLag(), interval = toobusy.interval();

toobusy.onLag(function(currentLag) {
  console.log("Event loop lag detected! Latency: " + currentLag + "ms");
});

The default maxLag value is 70ms, and the default check interval is 500ms.
This allows an "average" server to run at 90-100% CPU
and keeps request latency at around 200ms.
For comparison, a maxLag value of 10ms results in 60-70% CPU usage,
while latency for "average" requests stays at about 40ms.

These numbers are only examples,
and the specifics of your hardware and application can change them drastically,
so experiment!
The default of 70 should get you started.

Events

As of 0.5.0, toobusy-js exposes an onLag method. Pass it a callback to be notified when
a slow event loop tick has been detected.

references

There is nothing new under the sun. (Ecclesiastes 1:9)

Though applying "event loop latency" to node.js was not directly inspired by anyone else's work,
this concept is not new. Here are references to others who apply the same technique:

• Provos, Lever, and Tweedie 2000 - "notes that dropping incoming connections when the server is overloaded improved the shape of the performance curve."

license

WTFPL

New dependencies added: now-env and toobusy-js.

toobusy-js

Author: Unknown

Description: Don't fall over when your Node.JS server is too busy. Now without native dependencies!

Homepage: https://github.com/STRML/node-toobusy

Created	over 4 years ago
Last Updated	6 months ago
License	WTFPL
Maintainers	1
Releases	8
Direct Dependencies

README

Is Your Node Process Too Busy?

toobusy-js is a fork of lloyd's node-toobusy that removes native dependencies
in favor of using the unref introduced in node 0.9.1.

This package is a simpler install without native dependencies, but requires node >= 0.9.1.

Node-Toobusy

What happens when your service is overwhelmed with traffic?
Your server can do one of two things:

• Stop working, or...
• Keep serving as many requests as possible

This library helps you do the latter.

How it works

toobusy polls the node.js event loop and keeps track of "lag",
which is long requests wait in node's event queue to be processed.
When lag crosses a threshold, toobusy tells you that you're too busy.
At this point you can stop request processing early
(before you spend too much time on them and compound the problem),
and return a "Server Too Busy" response.
This allows your server to stay responsive under extreme load,
and continue serving as many requests as possible.

installation

npm install toobusy-js

usage

var toobusy = require('toobusy-js'),
    express = require('express');

var app = express();

// middleware which blocks requests when we're too busy
app.use(function(req, res, next) {
  if (toobusy()) {
    res.send(503, "I'm busy right now, sorry.");
  } else {
    next();
  }
});

app.get('/', function(req, res) {
  // processing the request requires some work!
  var i = 0;
  while (i < 1e5) i++;
  res.send("I counted to " + i);
});

var server = app.listen(3000);

process.on('SIGINT', function() {
  server.close();
  // calling .shutdown allows your process to exit normally
  toobusy.shutdown();
  process.exit();
});

tunable parameters

The library exposes a few knobs:

maxLag - This number represents the maximum amount of time in milliseconds that the event queue is behind,
before we consider the process too busy.
interval - The check interval for measuring event loop lag, in ms.

var toobusy = require('toobusy-js');

// Set maximum lag to an aggressive value.
toobusy.maxLag(10);

// Set check interval to a faster value. This will catch more latency spikes
// but may cause the check to be too sensitive.
toobusy.interval(250);

// Get current maxLag or interval setting by calling without parameters.
var currentMaxLag = toobusy.maxLag(), interval = toobusy.interval();

toobusy.onLag(function(currentLag) {
  console.log("Event loop lag detected! Latency: " + currentLag + "ms");
});

The default maxLag value is 70ms, and the default check interval is 500ms.
This allows an "average" server to run at 90-100% CPU
and keeps request latency at around 200ms.
For comparison, a maxLag value of 10ms results in 60-70% CPU usage,
while latency for "average" requests stays at about 40ms.

These numbers are only examples,
and the specifics of your hardware and application can change them drastically,
so experiment!
The default of 70 should get you started.

Events

As of 0.5.0, toobusy-js exposes an onLag method. Pass it a callback to be notified when
a slow event loop tick has been detected.

references

There is nothing new under the sun. (Ecclesiastes 1:9)

Though applying "event loop latency" to node.js was not directly inspired by anyone else's work,
this concept is not new. Here are references to others who apply the same technique:

• Provos, Lever, and Tweedie 2000 - "notes that dropping incoming connections when the server is overloaded improved the shape of the performance curve."

license

WTFPL

New dependencies added: toobusy-js.

toobusy-js

Author: Unknown

Description: Don't fall over when your Node.JS server is too busy. Now without native dependencies!

Homepage: https://github.com/STRML/node-toobusy

Created	over 4 years ago
Last Updated	6 months ago
License	WTFPL
Maintainers	1
Releases	8
Direct Dependencies

README

Is Your Node Process Too Busy?

toobusy-js is a fork of lloyd's node-toobusy that removes native dependencies
in favor of using the unref introduced in node 0.9.1.

This package is a simpler install without native dependencies, but requires node >= 0.9.1.

Node-Toobusy

What happens when your service is overwhelmed with traffic?
Your server can do one of two things:

• Stop working, or...
• Keep serving as many requests as possible

This library helps you do the latter.

How it works

toobusy polls the node.js event loop and keeps track of "lag",
which is long requests wait in node's event queue to be processed.
When lag crosses a threshold, toobusy tells you that you're too busy.
At this point you can stop request processing early
(before you spend too much time on them and compound the problem),
and return a "Server Too Busy" response.
This allows your server to stay responsive under extreme load,
and continue serving as many requests as possible.

installation

npm install toobusy-js

usage

var toobusy = require('toobusy-js'),
    express = require('express');

var app = express();

// middleware which blocks requests when we're too busy
app.use(function(req, res, next) {
  if (toobusy()) {
    res.send(503, "I'm busy right now, sorry.");
  } else {
    next();
  }
});

app.get('/', function(req, res) {
  // processing the request requires some work!
  var i = 0;
  while (i < 1e5) i++;
  res.send("I counted to " + i);
});

var server = app.listen(3000);

process.on('SIGINT', function() {
  server.close();
  // calling .shutdown allows your process to exit normally
  toobusy.shutdown();
  process.exit();
});

tunable parameters

The library exposes a few knobs:

maxLag - This number represents the maximum amount of time in milliseconds that the event queue is behind,
before we consider the process too busy.
interval - The check interval for measuring event loop lag, in ms.

var toobusy = require('toobusy-js');

// Set maximum lag to an aggressive value.
toobusy.maxLag(10);

// Set check interval to a faster value. This will catch more latency spikes
// but may cause the check to be too sensitive.
toobusy.interval(250);

// Get current maxLag or interval setting by calling without parameters.
var currentMaxLag = toobusy.maxLag(), interval = toobusy.interval();

toobusy.onLag(function(currentLag) {
  console.log("Event loop lag detected! Latency: " + currentLag + "ms");
});

The default maxLag value is 70ms, and the default check interval is 500ms.
This allows an "average" server to run at 90-100% CPU
and keeps request latency at around 200ms.
For comparison, a maxLag value of 10ms results in 60-70% CPU usage,
while latency for "average" requests stays at about 40ms.

These numbers are only examples,
and the specifics of your hardware and application can change them drastically,
so experiment!
The default of 70 should get you started.

Events

As of 0.5.0, toobusy-js exposes an onLag method. Pass it a callback to be notified when
a slow event loop tick has been detected.

references

There is nothing new under the sun. (Ecclesiastes 1:9)

Though applying "event loop latency" to node.js was not directly inspired by anyone else's work,
this concept is not new. Here are references to others who apply the same technique:

• Provos, Lever, and Tweedie 2000 - "notes that dropping incoming connections when the server is overloaded improved the shape of the performance curve."

license

WTFPL

New dependencies added: now-env and toobusy-js.

hpp

Author: Nicolai Kamenzky

Description: Express middleware to protect against HTTP Parameter Pollution attacks

Homepage: https://github.com/analog-nico/hpp

Created	about 3 years ago
Last Updated	4 months ago
License	ISC
Maintainers	1
Releases	7
Direct Dependencies	lodash and type-is
Keywords	hpp, http, parameter, pollution, attack and security

README

HPP

Express middleware to protect against HTTP Parameter Pollution attacks

Why?

Let Chetan Karande's slides do the explaining:

...and exploits may allow bypassing the input validation or even result in denial of service.

And HPP solves this how exactly?

HPP puts array parameters in req.query and/or req.body aside and just selects the last parameter value. You add the middleware and you are done.

Installation

This is a module for node.js and io.js and is installed via npm:

npm install hpp --save

Getting Started

Add the HPP middleware like this:

// ...
var hpp = require('hpp');

// ...
app.use(bodyParser.urlencoded()); // Make sure the body is parsed beforehand.

app.use(hpp()); // <- THIS IS THE NEW LINE

// Add your own middlewares afterwards, e.g.:
app.get('/search', function (req, res, next) { /* ... */ });
// They are safe from HTTP Parameter Pollution now.

Details about req.query

By default all top-level parameters in req.query are checked for being an array. If a parameter is an array the array is moved to req.queryPolluted and req.query is assigned the last value of the array:

GET /search?firstname=John&firstname=Alice&lastname=Doe

=>

req: {
    query: {
        firstname: 'Alice',
        lastname: 'Doe',
    },
    queryPolluted: {
        firstname: [ 'John', 'Alice' ]
    }
}

Checking req.query may be turned off by using app.use(hpp({ checkQuery: false })).

Details about req.body

Checking req.body is only done for requests with an urlencoded body. Not for json nor multipart bodies.

By default all top-level parameters in req.body are checked for being an array. If a parameter is an array the array is moved to req.bodyPolluted and req.body is assigned the last value of the array:

POST firstname=John&firstname=Alice&lastname=Doe

=>

req: {
    body: {
        firstname: 'Alice',
        lastname: 'Doe',
    },
    bodyPolluted: {
        firstname: [ 'John', 'Alice' ]
    }
}

Checking req.body may be turned off by using app.use(hpp({ checkBody: false })).

Whitelisting Specific Parameters

The whitelist option allows to specify parameters that shall not be touched by HPP. Usually specific parameters of a certain route are intentionally used as arrays. For that use the following approach that involves multiple HPP middlewares:

// Secure all routes at first.
// You could add separate HPP middlewares to each route individually but the day will come when you forget to secure a new route.
app.use(hpp());

// Add a second HPP middleware to apply the whitelist only to this route.
app.use('/search', hpp({ whitelist: [ 'filter' ] }));

GET /search?package=Helmet&package=HPP&filter=nodejs&filter=iojs

=>

req: {
    query: {
        package: 'HPP',
        filter:  [ 'nodejs', 'iojs' ], // Still an array
    },
    queryPolluted: {
        package: [ 'Helmet', 'HPP' ]
    }
}

The whitelist works for both req.query and req.body.

Performance

HPP was written with performance in mind since it eats CPU cycles for each request.

A performance test that includes two HPP middlewares plus a whitelist simulates an already demanding use case. On my Mac Book Air it measures 0.002ms to process a single request.

Contributing

To set up your development environment for HPP:

1. Clone this repo to your desktop,
2. in the shell cd to the main folder,
3. hit npm install,
4. hit npm install gulp -g if you haven't installed gulp globally yet, and
5. run gulp dev. (Or run node ./node_modules/.bin/gulp dev if you don't want to install gulp globally.)

gulp dev watches all source files and if you save some changes it will lint the code and execute all tests. The test coverage report can be viewed from ./coverage/lcov-report/index.html.

If you want to debug a test you should use gulp test-without-coverage to run all tests without obscuring the code by the test coverage instrumentation.

Change History

• v0.2.2 (2017-04-11)
  • Requiring individual lodash functions for faster boot time and lower memory footprint
    (Thanks to @mschipperheyn for suggesting this in issue #6)
• v0.2.1 (2016-04-03)
  • Added node v4 and v5 to CI build
  • Removed node v0.11 from CI build
  • Updated dependencies
• v0.2.0 (2015-05-25)
  • Bumped version to 0.2 to properly follow semver since the whitelist was added in v0.1.2
  • For better intuitiveness the last instead of the first value of an array is selected
  • Refactoring to improve readability and performance
    (Thanks to @le0nik for pull request #2)
  • Updated dependencies
    (Thanks to @maxrimue for pull request #3)
• v0.1.2 (2015-05-18)
  • Added whitelist feature
    (Thanks to @avaly for suggesting this in issue #1)
  • Updated dependencies
• v0.1.1 (2015-04-16)
  • Removed two closures
  • Updated lodash
• v0.1.0 (2015-04-12)
  • Updated dependencies
  • Use in production satisfactory
• v0.0.1 (2015-03-05)
  • Initial version

License (ISC)

In case you never heard about the ISC license it is functionally equivalent to the MIT license.

See the LICENSE file for details.

now-env

Author: Sergio Daniel Xalambrí

Description: Use `now.json` environment variables while developing

Homepage: https://github.com/zeit/now-env#readme

Created	11 months ago
Last Updated	about 2 months ago
License	MIT
Maintainers	3
Releases	16
Keywords	now, env, zeit, environment, variables and secrets

README

now-env

With the help of this package, you can easily set environment variables for the use in development.

If you're already using a now.json file or the now key in package.json, the env sub property will be assigned to process.env automatically.

In addition, you can store secrets locally, that are specific to the development environment.

IMPORTANT: Do not use this package in production. It should only be used locally, while developing. This is because the secrets that you're using locally shouldn't match the ones you use in production (more secure). In production, you should be adding secrets like this.

Usage

Firstly, install the package from npm:

yarn add now-env

As the last step, register the helper by loading it:

require('now-env')

That's all, you can now check process.env for the environment variables!

If your application is running inside Now cloud then this module is not going to do anything and let Now set your environment variables.

Secrets

Most probably you will want to use secret keys in your now.json file. This module allow you to use them too without worries in development.

Create a now.json with some secret defined as @secret-name, similar to:

{
  "env": {
    "SECRET": "@my-secret-key",
    "ANOTHER_SECRET": "@my-other-secret-key",
    "SECRET_FAIL": "@this-is-not-defined"
  }
}

Then create a now-secrets.json with the secrets names and values.

{
  "@my-secret-key": "keep-it-secret",
  "@my-other-secret-key": "keep-it-secret-too"
}

This file must be ignored to actually keep them secret.

Then when starting your application now-env will read the now.json and get the values from now-secrets.json. If a environment key can't be found in now-secrets.json (or the file doesn't exists) then is going to use the secret name as value, that means if DB_PASS is @db_pass and you don't define it inside now-secrets.json then the value will be @db_pass.

Required Variables

Now supports using the env key as an array of required values you'll need to provide when deploying. This module also allow you to use them in development.

Create a now.json with the array, similar to:

{
  "env": [
    "REQUIRED_KEY",
    "REQUIRED_SECRET"
  ]
}

Then create a now-required.json with the environment keys and values.

{
  "REQUIRED_KEY": "required-value",
  "REQUIRED_SECRET": "@required-secret"
}

You can also use secrets, for that you will need to create a now-secrets.json too.

Then when starting your application now-env will read the now.json and get the values from now-required.json (and now-secrets.json). If a environment key can't be found in now-required.json then is going to throw a reference error.

Migrating from dotenv

If you're already using the dotenv module you can switch to now-env easily:

1. Create a now.json file with the env key with every environment variable
2. Install now-env
3. Change require('dotenv').config() with require('now-env')
4. That's it!

Generated by 🚫 dangerJS

[maxrimue]

Might be offtopic but I get emails about this thread because I'm mentioned in hpp's readme above, which is not so nice tbh 😅

[brianlovin]

@maxrimue whoops, sorry for the annoying notification there! cc @mxstbr