-
Notifications
You must be signed in to change notification settings - Fork 31
/
readme.md
280 lines (217 loc) · 10.9 KB
/
readme.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
# fast-redact
very fast object redaction
[![Build Status](https://travis-ci.org/davidmarkclements/fast-redact.svg?branch=master)](https://travis-ci.org/davidmarkclements/fast-redact)
## Default Usage
By default, `fast-redact` serializes an object with `JSON.stringify`, censoring any
data at paths specified:
```js
const fastRedact = require('fast-redact')
const fauxRequest = {
headers: {
host: 'http://example.com',
cookie: `oh oh we don't want this exposed in logs in etc.`,
referer: `if we're cool maybe we'll even redact this`
}
}
const redact = fastRedact({
paths: ['headers.cookie', 'headers.referer']
})
console.log(redact(fauxRequest))
// {"headers":{"host":"http://example.com","cookie":"[REDACTED]","referer":"[REDACTED]"}}
```
## API
### `require('fast-redact')({paths, censor, serialize}) => Function`
When called without any options, or with a zero length `paths` array,
`fast-redact` will return `JSON.stringify` or the `serialize` option, if set.
#### `paths` – `Array`
An array of strings describing the nested location of a key in an object.
The syntax follows that of the EcmaScript specification, that is any JavaScript
path is accepted – both bracket and dot notation is supported. For instance in
each of the following cases, the `c` property will be redacted: `a.b.c`,`a['b'].c`,
`a["b"].c`, `a[``b``].c`. Since bracket notation is supported, array indices are also
supported `a[0].b` would redact the `b` key in the first object of the `a` array.
Leading brackets are also allowed, for instance `["a"].b.c` will work.
##### Wildcards
In addition to static paths, asterisk wildcards are also supported.
When an asterisk is place in the final position it will redact all keys within the
parent object. For instance `a.b.*` will redact all keys in the `b` object. Similarly
for arrays `a.b[*]` will redact all elements of an array (in truth it actually doesn't matter
whether `b` is in an object or array in either case, both notation styles will work).
When an asterisk is in an intermediate or first position, the paths following the asterisk will
be redacted for every object within the parent.
For example:
```js
const fastRedact = require('fast-redact')
const redact = fastRedact({paths: ['*.c.d']})
const obj = {
x: {c: {d: 'hide me', e: 'leave me be'}},
y: {c: {d: 'and me', f: 'I want to live'}},
z: {c: {d: 'and also I', g: 'I want to run in a stream'}}
}
console.log(redact(obj))
// {"x":{"c":{"d":"[REDACTED]","e":"leave me be"}},"y":{"c":{"d":"[REDACTED]","f":"I want to live"}},"z":{"c":{"d":"[REDACTED]","g":"I want to run in a stream"}}}
```
Another example with a nested array:
```js
const fastRedact = require('..')
const redact = fastRedact({paths: ['a[*].c.d']})
const obj = {
a: [
{c: {d: 'hide me', e: 'leave me be'}},
{c: {d: 'and me', f: 'I want to live'}},
{c: {d: 'and also I', g: 'I want to run in a stream'}}
]
}
console.log(redact(obj))
// {"a":[{"c":{"d":"[REDACTED]","e":"leave me be"}},{"c":{"d":"[REDACTED]","f":"I want to live"}},{"c":{"d":"[REDACTED]","g":"I want to run in a stream"}}]}
```
#### `remove` - `Boolean` - `[false]`
The `remove` option, when set to `true` will cause keys to be removed from the
serialized output.
Since the implementation exploits the fact that `undefined` keys are ignored
by `JSON.stringify` the `remove` option may *only* be used when `JSON.stringify`
is the serializer (this is the default) – otherwise `fast-redact` will throw.
If supplying a custom serializer that has the same behavior (removing keys
with `undefined` values), this restriction can be bypassed by explicitly setting
the `censor` to `undefined`.
#### `censor` – `<Any type>` – `('[REDACTED]')`
This is the value which overwrites redacted properties.
Setting `censor` to `undefined` will cause properties to removed as long as this is
the behavior of the `serializer` – which defaults to `JSON.stringify`, which does
remove `undefined` properties.
Setting `censor` to a function will cause `fast-redact` to invoke it with the original
value. The output of the `censor` function sets the redacted value.
Please note that asynchronous functions are not supported.
#### `serialize` – `Function | Boolean` – `(JSON.stringify)`
The `serialize` option may either be a function or a boolean. If a function is supplied, this
will be used to `serialize` the redacted object. It's important to understand that for
performance reasons `fast-redact` *mutates* the original object, then serializes, then
restores the original values. So the object passed to the serializer is the exact same
object passed to the redacting function.
The `serialize` option as a function example:
```js
const fastRedact = require('fast-redact')
const redact = fastRedact({
paths: ['a'],
serialize: (o) => JSON.stringify(o, 0, 2)
})
console.log(redact({a: 1, b: 2}))
// {
// "a": "[REDACTED]",
// "b": 2
// }
```
For advanced usage the `serialize` option can be set to `false`. When `serialize` is set to `false`,
instead of the serialized object, the output of the redactor function will be the mutated object
itself (this is the exact same as the object passed in). In addition a `restore` method is supplied
on the redactor function allowing the redacted keys to be restored with the original data.
```js
const fastRedact = require('fast-redact')
const redact = fastRedact({
paths: ['a'],
serialize: false
})
const o = {a: 1, b: 2}
console.log(redact(o) === o) // true
console.log(o) // { a: '[REDACTED]', b: 2 }
console.log(redact.restore(o) === o) // true
console.log(o) // { a: 1, b: 2 }
```
#### `strict` – `Boolean` - `[true]`
The `strict` option, when set to `true`, will cause the redactor function to throw if instead
of an object it finds a primitive. When `strict` is set to `false`, the redactor function
will treat the primitive value as having already been redacted, and return it serialized (with
`JSON.stringify` or the user's custom `serialize` function), or as-is if the `serialize` option
was set to false.
## Approach
In order to achieve lowest cost/highest performance redaction `fast-redact`
creates and compiles a function (using the `Function` constructor) on initialization.
It's important to distinguish this from the dangers of a runtime eval, no user input
is involved in creating the string that compiles into the function. This is as safe
as writing code normally and having it compiled by V8 in the usual way.
Thanks to changes in V8 in recent years, state can be injected into compiled functions
using `bind` at very low cost (whereas `bind` used to be expensive, and getting state
into a compiled function by any means was difficult without a performance penalty).
For static paths, this function simply checks that the path exists and then overwrites
with the censor. Wildcard paths are processed with normal functions that iterate over
the object redacting values as necessary.
It's important to note, that the original object is mutated – for performance reasons
a copy is not made. See [rfdc](https://github.com/davidmarkclements/rfdc) (Really Fast
Deep Clone) for the fastest known way to clone – it's not nearly close enough in speed
to editing the original object, serializing and then restoring values.
A `restore` function is also created and compiled to put the original state back on
to the object after redaction. This means that in the default usage case, the operation
is essentially atomic - the object is mutated, serialized and restored internally which
avoids any state management issues.
## Caveat
As mentioned in approach, the `paths` array input is dynamically compiled into a function
at initialization time. While the `paths` array is vigourously tested for any developer
errors, it's strongly recommended against allowing user input to directly supply any
paths to redact. It can't be guaranteed that allowing user input for `paths` couldn't
feasibly expose an attack vector.
## Benchmarks
The fastest known predecessor to `fast-redact` is the non-generic [`pino-noir`](http://npm.im/pino-noir)
library (which was also written by myself).
In the direct calling case, `fast-redact` is ~30x faster than `pino-noir`, however a more realistic
comparison is overhead on `JSON.stringify`.
For a static redaction case (no wildcards) `pino-noir` adds ~25% overhead on top of `JSON.stringify`
whereas `fast-redact` adds ~1% overhead.
In the basic last-position wildcard case,`fast-redact` is ~12% faster than `pino-noir`.
The `pino-noir` module does not support intermediate wildcards, but `fast-redact` does,
the cost of an intermediate wildcard that results in two keys over two nested objects
being redacted is about 25% overhead on `JSON.stringify`. The cost of an intermediate
wildcard that results in four keys across two objects being redacted is about 55% overhead
on `JSON.stringify` and ~50% more expensive that explicitly declaring the keys.
```sh
npm run bench
```
```
benchNoirV2*500: 59.108ms
benchFastRedact*500: 2.483ms
benchFastRedactRestore*500: 10.904ms
benchNoirV2Wild*500: 91.399ms
benchFastRedactWild*500: 21.200ms
benchFastRedactWildRestore*500: 27.304ms
benchFastRedactIntermediateWild*500: 92.304ms
benchFastRedactIntermediateWildRestore*500: 107.047ms
benchJSONStringify*500: 210.573ms
benchNoirV2Serialize*500: 281.148ms
benchFastRedactSerialize*500: 215.845ms
benchNoirV2WildSerialize*500: 281.168ms
benchFastRedactWildSerialize*500: 247.140ms
benchFastRedactIntermediateWildSerialize*500: 333.722ms
benchFastRedactIntermediateWildMatchWildOutcomeSerialize*500: 463.667ms
benchFastRedactStaticMatchWildOutcomeSerialize*500: 239.293ms
```
## Tests
```
npm test
```
```
224 passing (499.544ms)
```
### Coverage
```
npm run cov
```
```
-----------------|----------|----------|----------|----------|-------------------|
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s |
-----------------|----------|----------|----------|----------|-------------------|
All files | 100 | 100 | 100 | 100 | |
fast-redact | 100 | 100 | 100 | 100 | |
index.js | 100 | 100 | 100 | 100 | |
fast-redact/lib | 100 | 100 | 100 | 100 | |
modifiers.js | 100 | 100 | 100 | 100 | |
parse.js | 100 | 100 | 100 | 100 | |
redactor.js | 100 | 100 | 100 | 100 | |
restorer.js | 100 | 100 | 100 | 100 | |
rx.js | 100 | 100 | 100 | 100 | |
state.js | 100 | 100 | 100 | 100 | |
validator.js | 100 | 100 | 100 | 100 | |
-----------------|----------|----------|----------|----------|-------------------|
```
## License
MIT
## Acknowledgements
Sponsored by [nearForm](http://www.nearform.com)