Accelerated api for processing n x m entities

[hardbyte]

At the moment in entitymatch.cffi_filter_similarity_k we pass in two lists of bloom filters, but we end up calling the native code n times. This issue is to move that iteration into C.

We also are dealing with "nice" python bitarrays which require some manipulation (1) before passing into native code. We might want to consider adding an accelerated interface that takes our custom bit packed data as plain python bytes.

1: [ffi.new("char[128]", bytes(f[0].tobytes())) for f in filters1]

I've started experimenting in

• Branch feature-chunked-speedup for a C implementation of many x many comparisons.
• Branch feature-direct-cffi builds ontop of that with a look at accessing bitarray data from C without a memcopy. Only does a bitarray popcount for now.

# Assume ba is a bitarray
addr = ba.buffer_info()[0]
pntr = ffi.cast("char *", addr)
lib.popcount(pntr)

[unzvfu]

This issue has been split into issues #66 and #67.