Add public API WriteWithCallback to support custom callbacks

[jowlyzhang]

This PR adds a DB::WriteWithCallback API that does the same things as DB::Write while takes an argument UserWriteCallback to execute custom callback functions during the write.

We currently support two types of callback functions: OnWriteEnqueued and OnWalWriteFinish. The former is invoked after the write is enqueued, and the later is invoked after WAL write finishes when applicable.

These callback functions are intended for users to use to improve synchronization between concurrent writes, their execution is on the write's critical path so it will impact the write's latency if not used properly. The documentation for the callback interface mentioned this and suggest user to keep these callback functions' implementation minimum.

Although transaction interfaces' writes doesn't yet allow user to specify such a user write callback argument, the DBImpl::Write* type of APIs do not differentiate between regular DB writes or writes coming from the transaction layer when it comes to supporting this UserWriteCallback. These callbacks works for all the write modes including: default write mode, Options.two_write_queues, Options.unordered_write, Options.enable_pipelined_write

Test plan:
Added unit test in ./write_callback_test

[anand1976]

How does the user specify the callback? In WriteOptions perhaps?

[jowlyzhang]

How does the user specify the callback? In WriteOptions perhaps?

Yeah, that's a good question. WriteOptions would work, another idea is maybe we can make WriteWithCallback a public API too?

[anand1976]

How does the user specify the callback? In WriteOptions perhaps?

Yeah, that's a good question. WriteOptions would work, another idea is maybe we can make WriteWithCallback a public API too?

Yeah, WriteWithCallback should be fine too. We should have an overload that doesn't take WriteCallback as that seems specific to transaction DB, and maybe name PostWalWriteCallback to something more generic.

[anand1976]

LGTM! Thanks for the PR and quick turnaround

[jowlyzhang]

LGTM! Thanks for the PR and quick turnaround

Thank you for the quick review!

[anand1976]

LGTM. A couple of minor nits.

[facebook-github-bot]

@jowlyzhang has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@jowlyzhang merged this pull request in fc59d8f.