Long pauses using LMDB on Windows

[ChrisTrenkamp]

Environment

• LWJGL version: 3.2.2
• LWJGL build #: ?
• Java version: 8
• Platform: Windows
• Module: core|lmdb

Description

I'm investigating lwjgl3's LMDB module. The program starts with a massive bulk insert. For some reason, there is roughly a 10 second pause every couple thousand inserts (it needs to make a couple hundred-thousand inserts). I ran the same program on a Linux box and it does not have this problem. I can also confirm that lmdbjni and lmdbjava does not have this problem, not even on Windows.

I turned on all the memory debugging options (-Dorg.lwjgl.util.Debug=true -Dorg.lwjgl.util.DebugAllocator=true -Dorg.lwjgl.util.DebugStack=true) and there haven't been any reports while the program is running. I also tried turning on MDB_NOSYNC and MDB_NOMETASYNC and it is still pausing.

Typically, what I would expect to see in the Task Manager is the CPU usage to spike near 100% and the memory usage would slowly climb over the duration of the program. What I'm seeing with lwjgl3's LMDB module is the CPU only slightly goes up and the memory usage remains relatively flat.

I'm not sure what else to look for. Any idea's on what else to look for?

[Spasi]

Hey @ChrisTrenkamp,

The LMDB build included in LWJGL supports sparse file mappings on Windows, which has not made it to the official LMDB release (and probably never will - search for ITS#8324 for details). The incremental file growth was initially very expensive, that's why we have applied this patch. It was much faster in tests, but it sounds like there's something else that interacts badly with this approach and causes the pauses in your case.

I could revert LMDB to the official release. This issue will be fixed but we lose sparseness, i.e. database files will grow immediately to their maximum size. I think you're the first user that does anything serious with LMDB on Windows, so please let me know if you're OK with this.

[ChrisTrenkamp]

ITS#8324 does mention a performance reduction. If there's a build with the patch reverted, I can test it to see if that's the culprit.

[ChrisTrenkamp]

I should mention I'm currently using a non-patched LMDB release on Windows, so I'm okay with the files immediately growing to their max size.

[Spasi]

Hey @ChrisTrenkamp, sorry for the delay, a build you may test is available here. Compiled from current head of https://github.com/LMDB/lmdb/commits/mdb.RE/0.9, the only difference is that malloc/free calls have been replaced by LWJGL's configurable allocator.

Pass -Dorg.lwjgl.util.Debug=true -Dorg.lwjgl.util.DebugLoader=true to the JVM to verify that the new build is being used. You'll need to remove lwjgl-lmdb-natives-windows.jar from the class/module-path and also pass -Djava.library.path or -Dorg.lwjgl.librarypath to the JVM (pointing to the folder that contains the new lwjgl_lmdb.dll).

[ChrisTrenkamp]

Yup, the patch is the cause of the slowdown. When using the test lmdb build, it's back to normal.

[Spasi]

@ChrisTrenkamp Thank you!