Building a Deno.Kv backend in LMDB

[alexgleason]

Hi all!

I have been working with @xyzshantaram (see his comment here) to add LMDB support directly into Deno, as an optional backend to Deno.KV.

We started writing denokv_lmdb in Rust: https://github.com/xyzshantaram/denokv_lmdb But we would like to know if this is something the Deno team would consider merging if we complete it?

It is a significant time investment (and we don't want to maintain a fork of Deno) so we are at a crossroads where it would help to know Deno team's opinion on this before we proceed. Would you be interested in having LMDB as an optional backend to Deno.Kv, built directly into Deno?

We were thinking users could supply a URI like await Deno.openKv('lmdb://mydb.db') to select the LMDB backend. It would work seamlessly with the existing Deno.Kv API and provide a high-peformance option. SQLite would continue working with a bare path.

We explored other options first, like just using lmdb-js (@kriszyp) directly. I even started building a lmdb-js to Deno.Kv adapter in TypeScript. But unfortunately lmdb-js on Deno is a nonstarter due to unimplemented Node APIs.

So, then we started exploring the Rust side, and realized Deno.Kv is already very modular in the Deno codebase, and we began exploring that with denokv_lmdb.

We have been struggling with 30GB+ SQLite files in our application. Meanwhile LMDB is very fast in our tests. Here's a very simple benchmark of the difference:

benchmark       time (avg)        iter/s             (min … max)       p75       p99      p995
---------------------------------------------------------------- -----------------------------
DenoKv.get      29.41 µs/iter      34,004.4   (21.44 µs … 979.2 µs) 31.83 µs 55.47 µs 64.11 µs
DenoKv.set     419.32 µs/iter       2,384.8   (253.62 µs … 1.45 ms) 615.63 µs 783.8 µs 863.51 µs
lmdb.get       298.42 ns/iter   3,350,926.1  (278.8 ns … 517.18 ns) 301.35 ns 373.32 ns 517.18 ns
lmdb.set         1.67 ms/iter         600.3   (567.74 µs … 8.36 ms) 1.66 ms 1.87 ms 5.83 ms

It seems to me like Deno.KV and LMDB are a perfect match for running self-hosted instances. The author of lmdb-js even described Deno.Kv as "straightforward" with "pretty direct mappings". It wants to exist.

But the question is - if we build you a perfect LMDB backend for Deno.Kv, up to your code standards, do you want to merge it? Depending on this answer, we will either keep chipping away or switch to some sort of FFI implementation.

Thank you for your consideration!

[BlowaterNostr]

Having no FFI and a faster self hosted Deno.KV will be great.

I am working on https://github.com/BlowaterNostr/relayed which will benefit a lot from this

[lucacasonato]

Hey, this is a cool project!

For now, we'd be pretty unlikely to merge this, as we already have SQLite in the Deno binary, and adding LMDB just for this would quite significantly increase our binary size.

[alexgleason]

We ended up abandoning the KV idea and built something on Postgres with postgresjs, because it worked better than anything else we tried on Deno.