[Tests] SHA256 + official shuffling vectors

[mratsim]

This PR switches the hash function to SHA2-256 from Keccak256 and also uses the official shuffling test vectors.

Don't merge yet, I still have to fix a typedesc issue (https://github.com/status-im/nim-beacon-chain/compare/test-official-shuffling-vectors?expand=1#diff-b16dc57c1bf32e43578965533a7d7198R114) in nim-serialization.

Here are the accompanying changes:

• no crash when asking to shuffle an empty list
• old shuffling tests have been removed (based on keccak256)
• tree hashing tests have been commented out (based on keccak256 + SSZ is outdated)
• a small optimization by using toOpenArray, we can probably optimize further with in-place hashing into buffers using this interface: https://github.com/cheatfate/nimcrypto/blob/b90ba3abd/nimcrypto/sha2.nim#L570
  • Note that unfortunately the context for sha2 ciphers does not start empty so no optimization like https://github.com/status-im/nim-beacon-chain/compare/test-official-shuffling-vectors?expand=1#diff-9835c951698318fa2842b822ed8ac5c1L39
  • We are departing from the spec, we can reconcile with it and keep performance with a get_shuffled_index_impl proc that accepts the buffers and result as parameters (naively implementing the spec requires 90 alloc of 33-37 bytes buffers per validators)

[mratsim]

Typedesc issue opened: status-im/nim-serialization#4

We can go ahead with this and having parseTestsType(file) instead of parseTests(file, Type)

[tersec]

This appears to reintroduce calling clear and thus burnMem from https://github.com/cheatfate/nimcrypto/blob/b90ba3abd6db9461419f33db4cbae8206d3fde18/nimcrypto/keccak.nim which has consumed up to 25-30% of CPU. While it doesn't seem like we can use the same avoid-init shortcut as with keccak because SHA2 has actual context to init, just calling sha256.digest does unnecessary work.

[tersec]

LGTM with the caveats shown -- if you want, we can see how the perf around the hashing looks. I'm okay with merging it and fixing that later if it's still a problem. It's not a subtle perf drop/profiling signature if it's still one, and it might not be as relatively significant with the change of hash function. A lot of the rest is temporary around the 0.6 transition and would be medium-term inefficient to try to get perfect now.

[tersec]

Good time for this mass of semi-manual (albeit auto-code-generated) tests to disappear anyway, in favor of YAML or some other test harness format.

[tersec]

Yeah, good to mark the prevoius versoin as a kind of specific sort of TODO -- that is, when I scan for updates to perform, I (conceptually) grep for the previous version string(s). So I'd take this to mean that that set of constants should be specifically re-evaluated during a 0.6 transition.

[tersec]

Hopefully this is going to be a temporary special-case kludge. Kind of ugly to have to check (I write as the person who wrote the code around which you're working here) -- ideally, the 0.6 native solution just does this naturally.

[mratsim]

I think it's here to stay, I have to check how Python get away with a modulo 0 in

# https://github.com/ethereum/eth2.0-specs/blob/v0.6.0/specs/core/0_beacon-chain.md#get_permuted_index
def get_shuffled_index(index: ValidatorIndex, index_count: int, seed: Bytes32) -> ValidatorIndex:
    """
    Return the shuffled validator index corresponding to ``seed`` (and ``index_count``).
    """
    assert index < index_count
    assert index_count <= 2**40

    # Swap or not (https://link.springer.com/content/pdf/10.1007%2F978-3-642-32009-5_1.pdf)
    # See the 'generalized domain' algorithm on page 3
    for round in range(SHUFFLE_ROUND_COUNT):
        pivot = bytes_to_int(hash(seed + int_to_bytes(round, length=1))[0:8]) % index_count
        flip = (pivot - index) % index_count
        position = max(index, flip)
        source = hash(seed + int_to_bytes(round, length=1) + int_to_bytes(position // 256, length=4))
        byte = source[(position % 256) // 8]
        bit = (byte >> (position % 8)) % 2
        index = flip if bit else index

    return index