[Bug]: Cannot open realm files on vfat / exFAT filesystems on linux

[peppy]

What happened?

realm-dotnet and realm studio will not open realm files when placed on an ExFAT or similar filesystem.

Reported here.

Note that I have tested the same scenario on macOS and it works as expected.

Repro steps

• Move an existing realm file to an ExFAT volume on linux
• Attempt to open via Realm.GetInstance or Realm Studio

Version

.net 5.0

What SDK flavour are you using?

Local Database only

What type of application is this?

Other

Client OS and version

Linux (tested multiple variants)

Code snippets

No response

Stacktrace of the exception/crash you're getting

Unhandled exception. Realms.Exceptions.RealmFileAccessErrorException: No such file or directory
   at Realms.NativeException.ThrowIfNecessary(Func`2 overrider)
   at Realms.SharedRealmHandle.Open(Configuration configuration, RealmSchema schema, Byte[] encryptionKey)
   at Realms.RealmConfiguration.CreateRealm()
   at Realms.Realm.GetInstance(RealmConfigurationBase config)
   at osu.Game.Database.RealmContextFactory.CreateContext() in /var/lib/buildkite-agent/builds/debian-gnu-linux-vm-1/ppy/osu/osu.Game/Database/RealmContextFactory.cs:line 177

Relevant log output

No response

[papafe]

Hi @peppy, thanks for reporting this. So far I did not manage to reproduce it.
Would it be possible for you to share the realm file you're using? Maybe there is something specific that is triggering this issue.
Also, can you be more specific with the Linux variants you tested?

[bdach]

Hi, I also can reproduce this on my machine, Ubuntu 18.04. uname -a says:

Linux dachb-7559 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I'm reproducing with three separate realm files:

• one from my local install of osu! - the project which I am a collaborator on and which spurred the thread mentioned in the OP;
• one created via realm studio from a CSV;
• and the sample realm file which is linked on the splash screen of realm studio.

So this being specific to the realm looks unlikely.

I don't normally have a FAT partition, I'm reproducing by copying and pasting the realm file(s) onto a random nearby USB stick with vfat on it.

I'm not a huge filesystem person but I have a strong suspicion that this will be related to the client.realm.note named pipe that realm seemingly creates for IPC purposes. I'm saying this because when I run mkfifo when the working directory is one on the mounted USB stick, the response from the program is 'Operation not permitted'. Therefore the 'No such file or directory' error would start to make sense if realm wrappers were looking for said client.realm.note pipe. That said, I've ran strace on realm studio twice now and I haven't been able to find much of a trace that the application did try to access that file.

Curiously this isn't an issue on NTFS partitions, although that I suspect will be because to the best of my knowledge NTFS on Linux is done via FUSE, so the userspace emulation layer may be what permits named files to be created there.

If my suspicions above are correct then I'm not sure there's much left to be done here, if the .note file is central to realm operation.

[bdach]

I believe I've managed to produce the "smoking gun" (so to speak) that seems to confirm my suspicions via strace. My mistake was not specifying -f to follow forks. In the baseline reproduction scenario I get:

> [pid   318] lstat("/media/dachb/KINGSTON/default2.realm.note",  <unfinished ...>
  [pid   317] stat("/media/dachb/KINGSTON",  <unfinished ...>
  [pid   316] lstat("/media/dachb/Dokumenty1/test/default.realm",  <unfinished ...>
  [pid   317] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
  [pid 32502] futex(0x50870738990, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid   318] <... lstat resumed> 0x7f765dc38850) = -1 ENOENT (No such file or directory)

If I touch default2.realm.note as a normal file just for kicks (else known as confirmation purposes that the browser gets any farther than before), I get this:

[pid  2111] openat(AT_FDCWD, "/media/dachb/KINGSTON/default2.realm.note", O_RDWR) = 49
[pid  2111] fcntl(49, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid  2111] epoll_create(1)             = 50
[pid  2111] pipe([51, 52])              = 0
[pid  2111] fcntl(51, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid  2111] fcntl(52, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid  2111] epoll_ctl(50, EPOLL_CTL_ADD, 51, {EPOLLIN, {u32=51, u64=51}}) = 0
[pid  2111] mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f126b95e000
[pid  2111] mprotect(0x7f126b95f000, 8388608, PROT_READ|PROT_WRITE) = 0
[pid  2111] clone(child_stack=0x7f126c15db70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f126c15e9d0, tls=0x7f126c15e700, child_tidptr=0x7f126c15e9d0) = 2145
strace: Process 2145 attached
[pid  2111] epoll_ctl(50, EPOLL_CTL_ADD, 49, {EPOLLIN|EPOLLET, {u32=49, u64=49}} <unfinished ...>
[pid  2111] <... epoll_ctl resumed> )   = -1 EPERM (Operation not permitted)

Which looks pretty conclusive to me.

[jedelbo]

Yes, it is not possible to create a pipe on VFAT filesystem. You can assign the name of a directory in which it would be possible to create the pipe to fifo_files_fallback_path on the configuration object that is passed down to the storage layer. I am not sure how/if this is exposed in the .NET SDK.

[papafe]

Unfortunately this is not something we expose through our configuration, but probably should.

[bdach]

I'd say it would be pretty nice if the fifo_files_fallback_path was indeed configurable from the .NET SDK - it'd potentially be an alternative escape hatch for several other minor but slightly annoying issues we've had with the named pipe file already.

For one instance, our project supports storage migration to another location, and we've already had to manually exclude the .realm.note file from this migration process because it did not play nice. Another case is that windows VMs that share a filesystem with a linux host tend to blow up when navigating to a directory that contains such a named pipe.

If we could set that path to something like somewhere in /tmp then that'd resolve those woes pretty definitively.

[papafe]

I definitely understand your concerns, we can work to expose that through the configuration of the realm.