Querying datastore currently very expensive

[whyrusleeping]

Right now, querying the datastore, whether through the blockstore, or separately as is the case for provider records. Some of this is because the query returns the keys as strings, which we then convert to datastore keys (which isnt a free operation), and then do some extra (probably unncessary) processing per key. We should work on improving this soon.

[kevina]

I would be really interested in the overhead to convert from a binary key to the hex32 encoding and how much overhead in general the Datastore library causes as oppose to using the leveldb directly.

Related ipfs/go-datastore#40.

[kevina]

I have done some work on this in #3376 although they only relate to querying through the blockstore, in the blockstore case the extra key processing does not have a significant impact on a multi-core machine. On a single core machine it might, and reducing the extra processing will likely help in CPU usage.

[kevina]

@whyrusleeping now that all of my performance improvements are in, do you think we can close this?

[whyrusleeping]

I think this can probably be closed for now. Good job @kevina :)