Ruby hash broken for Messages with repeated fields.

[abscondment]

Expected

All Ruby Message objects return a hash.

Actual

Calling hash on a Message with a repeated field will occasionally raise the following error:

RangeError: bignum too big to convert into `long'

Test Case

I have written a gist with tests demonstrating the behavior.

Details

This is because of a bug in repeated_field.c, wherein the hash function occasionally returns a Bignum instead of a Fixnum. RepeatedField_hash can return a Bignum because of the algorithm it uses for combining hashes: bitwise shift left, then xor. If the original hash is large enough, the << will cause Ruby to automagically promote the value from a Fixnum to a Bignum.

Ruby indicates that hash should return a Fixnum, so this behavior is not correct.

Up the stack, storage.c attempts to convert this Bignum to a long, which produces the RangeError.

Proposal

I think the correct behavior is to use rb_hash_start -> rb_hash_uint -> rb_hash_end -> INT2FIX as storage.c does. I'd be happy to provide a patch for this if that sounds correct. I've provided a PR.