Buffer - feature request: Buffer.compare with offsets.

[rootslab]

I'm really impressed with the new iojs features, an awesome work!
Buffer.compare is a great iojs addition ( memcmp is very fast!), but, unfortunately, the current method implementation doesn't offer a way to compare 2 portions of Buffers: you are forced to slice one or both Buffers to compare desired portions of data.

Now, Buffer.compare accepts 2 buffers as arguments, instead, it could accept 4 args (+1 optional), or something like that:

function ( Buffer b1, Number start1, Buffer b2, Number start2 [, Number length ] ) {}
/*
 * For example, comparing 10 bytes:
 *  - b1 from index 0 to 9 
 *  - b2 from index 16 to 25
 */
Buffer.compare( b1, 0, b2, 16, 10 )
// or, to mantain current signature:
Buffer.compare( b1, b2, 0, 16, 10 )

@trevnorris It would be great if it was planned to add this functionality, it simplify things a lot!!

[feross]

What about just slicing the buffers before comparing? Slicing will
reference memory in the parent buffer so there's no copy. How much overhead
would this actually save?
On Mon, Jan 19, 2015 at 7:45 PM gu [email protected] wrote:

I'm really impressed with the new iojs features, an awesome work!
Buffer.compare is another great iojs addition ( memcmp is very
fast!), but, unfortunately, the current method implementation doesn't offer
a way to compare 2 portions of Buffers: you are forced to slice one or both
Buffers to compare desired portions of data.

Now, Buffer.compare accepts 2 buffers as arguments, instead, it could
accept 4 args (+1 optional), or something like that:

function ( Buffer b1, Number start1, Buffer b2, Number start2 [, Number length ] ) {}/* * For example, comparing 10 bytes: * - b1 from index 0 to 9 * - b2 from index 16 to 25 */
Buffer.compare( b1, 0, b2, 16, 10 )

@trevnorris https://github.com/trevnorris It would be great if it was
planned to add this functionality, it speeds up things a lot!!

—
Reply to this email directly or view it on GitHub
#521.

[rootslab]

Hi @feross,
"What about just slicing the buffers before comparing? "
I obviously know that :) ".. you are forced to slice one or both Buffers to compare desired portions of data.
For example, If you compare a fixed pattern against a data stream, you have to repeatedly slice, the same chunk of incoming data, to the current start offset, instead of simple incrementing an index and executing memcmp / Buffer.compare on a portion of it (anyway, native memcmp gets 3 arguments, 2 pointers and a number for specifying length).

[rootslab]

P.S
@feross, It is also possible to use Buffer.copy without indexes if you slice input buffers to desired offsets :)), but we prefer to use direct offset indexes; for an analogue reason, I think, we could get a Buffer.compare method with offset indexes.
Probably, as you says, it doesn't saves a lot in terms of average execution time, I don't know that, I really mean "simplify things". Thanks for your interest.

[jorangreef]

Whenever I compare buffers, it's also almost always using an offset and common length argument, exactly as @rootslab suggested, for example compare(buffer1, offset1, buffer2, offset2, comparisonLength), so this would really fit my typical use case. Slicing to compare within a binary search (usually where I need to compare) would make no sense.

[trevnorris]

I'm cool with the idea. Though having that many overloads would become a pain to check. How about we start with the simple case:

Buffer#compare(buf[, offset[, length]])

We can work on the syntax for Buffer.compare() afterwards.

[trevnorris]

Oh, and it's not an iojs addition. The feature is also in v0.11. :-)

[rvagg]

https://github.com/bnoordhuis/node-buffertools is my go-to for this kind of thing but I'm a big +1 on putting the most useful stuff in core because having to compile an addon just to do simple things often makes me resort to this kind of thing which I'm sure would make @trevnorris scream: buf1.toString('hex') == buf2.toString('hex').

[rootslab]

@jorangreef, then, I'm not alone ;))

@trevnorris ops.. I didn't see it in v0.11! ;)
Unfortunately (or fortunately?) my c++/v8 skills are very poor , but I have written some js code for checking index ranges before calling native compare. It's very "tight code", I know, but it works and it could be easy modified or probably used as a simple reference.

@rvagg thanks for suggestion, I know buffertools by @bnoordhuis, however, let me scream together with @trevnorris about toString('hex') ;)) it also converts every byte of raw data to 2 bytes (ASCII) chars, before comparing.

Thanks to all! ;)

[rootslab]

@trevnorris @rvagg, I hope it is useful for something :)

compare : function ( buf1, buf2, bpos1, bpos2, bytes ) {

    if ( ! ( buf1 instanceof Buffer &&
             buf2 instanceof Buffer ) )
        throw new TypeError( 'Arguments must be Buffers' );

    var abs = Math.abs
        , min = Math.min
        , blen1 = b1.length
        , blen2 = b2.length
        // normalize arg values
        , s1 = bpos1 >>> 0 ? abs( + bpos1 ) : 0
        , s2 = bpos2 >>> 0 ? abs( + bpos2 ) : 0
        , len = bytes >>> 0 ? abs( + bytes ) : min( blen1 - s1, blen2 - s2 )
        ;

    if ( ! blen1 || ! blen2 )
        throw new Error( '..0 length buffer..' );

    if ( ( len <= 0 ) ||
         ( s1 + len > blen1 ) ||
         ( s2 + len > blen2 ) )
        throw new RangeError( 'out of range index' );
  /*
    * Indexes range:
    *
    * - buf1 -> slice(s1, s1 + len)
    * - buf2 -> slice(s2, s2 + len)
    *
    * call native compare, with safe indexes:
    */
    return internal.compare( buf1, buf2, s1, s2, len );
}

[Trott]

@trevnorris et al.: Assuming this is still a desirable feature, how would the arguments for Buffer#compare(buf[, offset[, length]]) work? Specifically, would offset and length apply to both buffers or just to buf?

[trevnorris]

@Trott If the hope is to skip the .slice() operation then you'll need a signature of:

compare(buf[, bufOffset[, bufLength[, thisOffset]]])

It's a bit ugly, but don't see much way around that.

EDIT: Guess thisLength wouldn't be necessary.

[jasnell]

Fairly -1 on this. Using slice is a bit more verbose but it works.

[jorangreef]

I wanted to use buffer.compare for a fast substring search implementation the other day, but could not, because of all the garbage that .slice() generates. It was in a hot function and the C memcmp would have been useful but I had to use a while loop instead.

I think Trevor's API suggestion is right.

[trevnorris]

I can't see a technical reason why this API isn't a valid proposition. We already support similar with Buffer#copy() to also allow bypassing .slice() because the operation is expensive enough to hinder the hot path.

[jasnell]

Ok. That works for me then.
On Mar 22, 2016 11:18 PM, "Trevor Norris" [email protected] wrote:

I can't see a technical reason why this API isn't a valid proposition. We
already support similar with Buffer#copy() to also allow bypassing
.slice() because the operation is expensive enough to hinder the hot path.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#521 (comment)