Concurrent merges

[pascutto]

Fixes #13.
On top of #105 to ease debugging.
This is still very WIP.

[samoht]

What about adding hooks to ease testing -- maybe a lock that the testing framework can hold to block the merge while doing other operations, to check that there are progress going on.

More generally, I think we should try to redesign the library a bit to make it more easily testable -- I know @craigfe started to work a bit on this on the I/O side, but would be great if we could push this "design-to-ease-tests" at the core of the library, especially for the merges.

[icristescu]

Update on debugging:

• from the logs, it seems that the readonly_and_merge tests fail when a write occurs at the end of a merge. In this case it is expected to fail, a flush is needed between a write and a read of a RO instance. I've updated the tests, and cannot reproduce an error for this particular tests. A test to reproduce this failing behaviour is _write_after_merge.
• tests readonly_s and readonly still fail and it seems (again from reading the logs) that both fail on the same case: log_async is written during a merge and moved into log at the end of the merge. But between the copy of log_async to log and the Io.sync log the log becomes empty. The merge is concurrent with a find of a RO instance.

[icristescu]

• tests readonly_s and readonly still fail and it seems (again from reading the logs) that both fail on the same case: log_async is written during a merge and moved into log at the end of the merge. But between the copy of log_async to log and the Io.sync log the log becomes empty. The merge is concurrent with a find of a RO instance.

This is now fixed by the last commit. The issue was a buffer shared between an RO and an RW instance. But there is (at least) one bug remaining.

[icristescu]

But there is (at least) one bug remaining.

This bug was caused by something like* the following scenario:
RW - writes value in log, then calls merge, the value is moved into index, but the generation is not set
RO - calls sync_log and reads generation
RW - clears log, sets generation
RO - reads offset of log but does not see any changes in neither generation nor offset. it does not update its index and therefore cannot find the value.

I proposed a quick fix which is to force an update of the index when the value is not found. This is only to check that the bug is fixed, maybe we can improve it.

*something like this, because is hard to track what really happens as too many logs introduce delays and the bug does not reappear.

[craigfe]

After the most-recent commit (e76715c), I have gone through 1000 consecutive test runs without any failures; this seems to have fixed something at least 🎉

[craigfe]

Final round of nit-picks.

[pascutto]

Fixed ✔️