Error decoding object with large buffer

[BryceCicada]

The following snippet fails with error at decode:

    let buffer = fs.readFileSync('some_large_file');
    buffer = buffer.slice(0, 16377);

    let object = {
        a: buffer,
        b: 1
    };

    let encoded = cbor.encode(object);
    let decoded = cbor.decode(encoded);

Error: Insufficient data
    at Object.decodeFirstSync (/*****/node_modules/cbor/lib/decoder.js:131:15)
    at /******/index.js:24:24
    at Object.<anonymous> (/*****/index.js:181:1)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)
    at Function.Module.runMain (module.js:575:10)
    at startup (node.js:159:18)

The size of buffer for which the error occurs depends on the name of the object properties a and b. That is, if the names are longer then shorter buffers produce the error.

With property names a and b, the error does not occur if I reduce the size of buffer to 16376 bytes, like:

    buffer = buffer.slice(0, 16376);

The error does not occur if I switch the order of a and b in object, like:

    let object = {
        b: 1,
        a: buffer
    };

Reproduced with 2.0.0 and 3.0.0

[BryceCicada]

Issue is not present in forked project at https://www.npmjs.com/package/borc

[hildjj]

My guess is that there is an anomaly in the buffering code. looking into it.

[hildjj]

@BryceCicada, this is the point at which we blow past the highWaterMark in the output NoFilter. If you stream the output, rather than using the sync mode, it works correctly:

let o = {
  a: new Buffer(16377),
  b: 1
}
let e = new cbor.Encoder()
e.pipe(process.stdout)
e.write(o)

Still working on a fix.

[hildjj]

@linuxwolf thought he might be able to look at this. Note: I think the actual issue is roughly at https://github.com/hildjj/nofilter/blob/509192e5ccf71208b6cef8734933ddc5a1d8ffce/src/index.coffee#L252, where I don't check the return value of push. In this case, it's returning false.

[linuxwolf]

@hildjj

In this case, I don't think checking the result of push and "doing something" is going to fix things. The highWaterMark is hit, and the NoFilter stream will be paused until more is consumed from the readable side, but no more can be consumed until a complete value type is found. Chicken, meet egg.

A quick fix is to set the highWaterMark for the underlying NoFilter to the size of input within this project. The better fix is probably to do that same thing, but in nofilter. Will start playing around with the quick fix, but likely will submit a PR against nofilter rather than here.

[linuxwolf]

Actually, nevermind all that (-:

It is a problem with the highWaterMark, but during cbor.encode(). When the Encoder is called as a one-shot operation, it's possible with a large amount of data to exceed the highWaterMark before any consuming of said data, which then leads to pausing the streams which invisibly stops the encoder from finishing.

The fix is probably to work around the highWaterMark during one-shot encoding, and/or trigger an error if cbor.encode() hits that highWaterMark.

[yoursunny]

I'm seeing the same issue. It seems that the problem occurs only if the large item is both encoded and decoded by node-cbor. It does not occur if either encoding or decoding is performed with another library.
In the following program, I use both node-cbor and cbor-sync to encode then decode a large array.

const cbor = require("cbor");
const CBOR = require("cbor-sync");
const isArray = require("util").isArray;

const length = 5000; // >=3500 triggers the bug
const input = [];
for (let i = 0; i < length; ++i) {
  input.push(i.toString());
}

function testEncodeDecode(title, encode, decode) {
  const wire = encode(input);
  let result;
  try {
    result = decode(wire);
  }
  catch (ex) {
    console.error(title, ex);
    return;
  }
  if (!isArray(result)) {
    console.error(title, "bad-type");
    return;
  }
  if (result.length != input.length) {
    console.error(title, "bad-length", result.length);
    return;
  }
  console.log(title, "OK");
}

testEncodeDecode("cbor => cbor", cbor.encode, cbor.decodeFirstSync);
testEncodeDecode("cbor => cbor-sync", cbor.encode, CBOR.decode);
testEncodeDecode("cbor-sync => cbor", CBOR.encode, cbor.decodeFirstSync);
testEncodeDecode("cbor-sync => cbor-sync", CBOR.encode, CBOR.decode);

The output is:

cbor => cbor Error: Insufficient data
    at decodeFirstSync (cbor-test/node_modules/cbor/lib/decoder.js:158:15)
    at testEncodeDecode (cbor-test/x.js:15:14)
    at Object.<anonymous> (cbor-test/x.js:32:1)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)
    at Function.Module.runMain (module.js:694:10)
    at startup (bootstrap_node.js:204:16)
cbor => cbor-sync OK
cbor-sync => cbor OK
cbor-sync => cbor-sync OK

[longngn]

I got into the same problem, solved with https://github.com/dignifiedquire/borc

[dancon]

I got the same problem, and borc work will indeed.

[leobel]

Same problem here, fixed with borc

[aheissenberger]

I got the same problem, and borc work will indeed.

[joepie91]

It's important to note that borc does not support all of the same data types that cbor does, and so it's not a drop-in replacement even for synchronous usage.