SharedLibraryLoader - process cannnot access the file because in use by another process

[kappaOne]

I can consistently reproduce the following error when using the SharedLibraryLoader on Windows 8.1:

On 32bit JVM

java -jar myapp.jar
Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\Users\username\AppData
\Local\Temp\lwjglusername\131b1409\lwjgl.dll: The process cannot access the file 
because it is being used by another process
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
        at java.lang.Runtime.load0(Runtime.java:809)
        at java.lang.System.load(System.java:1086)
        at org.lwjgl.system.Library.loadSystem(Library.java:115)
        at org.lwjgl.system.Library.loadSystem(Library.java:105)
        at org.lwjgl.system.Library.loadSystem(Library.java:85)
        at org.lwjgl.stb.LibSTB.<clinit>(LibSTB.java:17)
        at org.lwjgl.stb.STBTTPackContext.<clinit>(STBTTPackContext.java:25)
        at <init>(Unknown Source)

On 64bit JVM

java -jar myapp.jar
Exception in thread "main" java.lang.UnsatisfiedLinkError: C:\Users\username\AppData
\Local\Temp\lwjglusername\3e9533cf\lwjgl_stb.dll: The process cannot access the file
 because it is being used by another process
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
        at java.lang.Runtime.load0(Runtime.java:809)
        at java.lang.System.load(System.java:1086)
        at org.lwjgl.system.Library.loadSystem(Library.java:115)
        at org.lwjgl.system.Library.loadSystem(Library.java:105)
        at org.lwjgl.system.Library.loadSystem(Library.java:85)
        at org.lwjgl.stb.LibSTB.<clinit>(LibSTB.java:17)
        at org.lwjgl.stb.STBTTPackContext.<clinit>(STBTTPackContext.java:25)
        at org.Font.<init>(Unknown Source)

These errors only happen the first time the jar is run and the natives are extracted, the second time everything works fine. It can be reproduced by visiting %temp% (from run) and deleting the lwjglusername folder and re-running the jar.

It appears to be that an extracted native remains in use by the SharedLibraryLoader and then when LWJGL attempts to load the native it fails. It might be related to something like a FileOutputStream not being closed after the native is extracted from the jar and would explain why on next run it works, as native is not extracted by SharedLibraryLoader on second run, (possibly this one requires a .close() after the write).

[Spasi]

It might be related to something like a FileOutputStream not being closed after the native is extracted from the jar and would explain why on next run it works, as native is not extracted by SharedLibraryLoader on second run, (possibly this one requires a .close() after the write).

That's a try-with-resources block. Afaict all streams in the SharedLibraryLoader are being properly closed.

These errors only happen the first time the jar is run and the natives are extracted, the second time everything works fine. It can be reproduced by visiting %temp% (from run) and deleting the lwjglusername folder and re-running the jar.

I'm afraid I cannot reproduce it on Windows 10. Could you please try again with Windows Defender and other antivirus/anti-malware programs disabled?

[kappaOne]

Ah, you are correct, disabling Panda Free Antivirus, causes the issue not to happen. So the antivirus is temporarily locking the files when extracted. Annoying problem to workaround for end users.

[Spasi]

I don't think the SharedLibraryLoader is a solution that can be relied upon for deploying LWJGL applications to end user systems. A better option is native installers for each target operating system. The SLL is more of a convenience for developers.

With that said, we can maybe detect this case, sleep a bit, then try again. I tried installing Panda Free Antivirus, but again, couldn't reproduce it. Could you maybe do a few tests to see if something like that works?

[kappaOne]

Sleeping and retrying should work, however another possible solution on Windows might be to use Java's FileChannel/FileLock classes.

So we could extract a native and get a file lock on it (or even lock the FileOutputStream while extracting), then load the natives using LWJGL after which the native file can be unlocked.

If this works, it might be a good solution, since in theory would prevent other applications from getting an exclusive lock on the native file and/or deleting it before its loaded/unlocked.

A shared file lock should even allow other application like a virus scanner to scan the file while its in use.

Guess this will need a bit of experimenting to see if it works.

[Spasi]

Did some quick tests, looks like for it to work we need to write+close the extracted file, then acquire a read lock. This means that the antivirus may grab a lock on the file before us, so we can't avoid a sleep+retry loop.

I have attached a possible fix, could you please do some tests with it?

sll_lock_fix.zip

[kappaOne]

Thanks, I'll give it a test and report back.

We might be able to avoid the sleep look by using FileChannel.lock() instead of FileChannel.tryLock(), as that method waits for any existing locks to unlock before returning.

[kappaOne]

Just finished testing the sll_lock_fix code and can confirm that it works and there is no in use by another process error.

Everything is fine when the antivirus is disabled, however when enabled there is a 1.5 second delay for each native that is extracted (which adds up due to number of natives). We could leave code as is and leave the delay as its the scanner which is interfering but is pretty annoying and may not be clear to end users where delay is coming from.

I've done some debugging to see where in the code the delay happens and it appears to be to kicking in straight after the line:

while ( (n = input.read(buffer)) > 0 ) output.write(buffer, 0, n);

when the extractFile method returns.

It appears the scanner is getting in before the lock and after the native has been written.

One idea could be to get the lock in the extractFile method. This could be done just before or with the FileOutputStream by getting a FileChannel and storing it in a static global variable. Then after the native has been loaded by LWJGL the FileChannel/Lock can be released in the sll.

[Spasi]

Updated SharedLibraryLoader.java to lock in extractFile (before the stream is closed).

[kappaOne]

Tested updated code however no luck. The first code seems cleaner.

After lots of testing I'm not sure locking files is going to help much, best we can do is before loading a native, test if its locked, if it is, then do a sleep+retry loop.

[Spasi]

Tested updated code however no luck.

Still slow? Or does it fail?

After lots of testing I'm not sure locking files is going to help much

Have you tried taking an exclusive lock instead? (will need to use StandardOpenOption.WRITE and lock(0L, Long.MAX_VALUE, false))

best we can do is before loading a native, test if its locked, if it is, then do a sleep+retry loop.

Use the original fix and suffer the delays?

[kappaOne]

Same behaviour as before, i.e. still slow when antivirus is running

Yeh, tested exclusive lock too, same issue.

1. The plan was to lock the file, get Java's System.load to load the locked native file and then unlock the native file, however this doesn't work, in my testing System.load fails if the native file is locked, even if it's by the same process.

2. Your initial code, did the following:
   a) extracted the natives and
   b) then tries using a sleep+retry until it gets a file lock.
   c) Thereafter it unlocks native file and
   d) System.load loads the unlocked native file.

The problem was the antivirus kicked in immediately after a) and blocks the process for 1.5 seconds. Maybe there is some sort of notification by the OS to antivirus as soon as a file is created.

3. Your second code did the following
   a) extracts the native file and
   b) obtains the lock immediately on the file before the stream closes.
   c) then unlocks the native file on the method extractFiles returning and cleaning up
   d) System.load loads the unlocked native file.

The problem was the antivirus kicked in immediately after c) when the file is unlocked and blocks the process for 1.5 seconds. Maybe there is some sort of notification by the OS to antivirus as soon as a file is created.

I've tried maintaining a lock on the native file for as long as the jvm runs, however running into the problem at 1).

Therefore the easiest solution atm seems to be 2) above i.e. extract the natives, let the antivirus get the file lock, wait until we can get a lock then continue.

[Spasi]

Unless I missed something important, your description is wrong. In both attempts the file is still locked when System.load runs. Note that I'm locking the FileChannel but never close/release the FileLock. The lock is released when the channel is closed, which happens in Library.java after the call to System.load (the try-with-resources block does it).

[kappaOne]

very likely I've got it wrong then, which would mean the antivirus is interfering even with the lock on. Searching for solutions online seem to point to the the catch file in use error, sleep and try again method.

Another possible way around this is we could extract the natives with the SharedLibraryLoader as early as possible (all of them) but lazy load them only on demand, this gap might be enough for an antivirus to scan the files but not lock them when they are needed by System.load.

[Spasi]

Could you please try the next nightly build? It's a combination of the first fix and some improvements to diagnostic messages. On the first run with -Dorg.lwjgl.util.Debug=true -Dorg.lwjgl.util.DebugLoader=true you should see the message: File is locked by another process, waiting...

Another possible way around this is we could extract the natives with the SharedLibraryLoader as early as possible (all of them) but lazy load them only on demand

This is technically possible. I'll think about it, not sure if it's worth the trouble. What would the API look like?