How to simulate File System on other platform

[markusschaber]

Hi,
As far as I can see, your mocked file system implementation supports the semantics of Windows, Linux and MacOS.

However, I could not find how to tell which semantics I want when creating an instance of the mock.

Our use case would be to test our code against a file system with Windows and MacOS semantics in the CI pipeline which currently only runs on Linux.

[vbreuss]

Hello @markusschaber,

That's a very interesting use case, I never thought about. Currently we mostly distinguish between file systems via the Execute methods, which would (at least in theory) be able to be changed, so that the behaviour could be overwritten. However there are also some other edge cases to consider (probably there are even more, but these just came to my mind immediately):

• All methods that rely on Path methods would need to be changed manually to use the correct directory separator depending on such a setting
• Some methods only work on the corresponding OS and rely on .NET features that are not supported on another OS, e.g. File.GetUnixFileMode has an UnsupportedOSPlatform("windows") attribute and would throw when running on Windows. This attribute could not be changed at runtime with a setting.

The primary goal of this library is to behave (as much as possible) the same as the real file system on the corresponding platform, so removing the attributes or relaxing some requirements, would not really be an option.

Do you have any ideas, how to overcome these challenges?
I would be happy to elaborate further on it!

[markusschaber]

The platform identification would neet to be instance members of the MockFileSystem, instead of statically accessing the Execute class. The default values would still reflect the current platform, so everything is strictly opt-in.

I had been falsely assuming that the mock implementation actually does not depend on any underlying file system code. You're right, some methods like Path would need reimplementation, so they can switch between behaviours at runtime. I guess the actual platform specific implementations could be gathered together from .NET itself (with appropriate attribution), as both projects use MIT license.

As far as I can see, the SupportedOSPlatformAttribute and UnsupportedOSPlatformAttribute are just for compile time analysis - I think their diagnostics could be suppressed for the test projects. The exceptions at runtime are actually thrown by the implementation, which could emulate the platform specific behaviour instead.

[vbreuss]

Let's talk about a possible approach:

1. Make the Execute class not static and adding it to the MockFileSystem with a constructor that can receive an OSPlatform enum and react accordingly.

2. Then we could add a method to replace it on the MockFileSystem, e.g.

   public MockFileSystem SimulatingOperatingSystem(OSPlatform osPlatform)
   {
     Execute = new Execute(osPlatform);
     return this;
   }

   and use this property instead of the static calls to Execute.

3. Update all Path-related calls to use a custom implementation instead.

4. Update tests, so that they also cover other OS platforms. Here I am still unsure, how to make sure, that the result matches the real file system.
   Any ideas?

5. At this point, we would probably run in some additional necessary changes 😄

Would you be willing to help with such a large change?

[markusschaber]

We should have the Tests run in 4 modes: Windows, MacOS, Linux and "native" - which is the OS we're actually running on.

I'm not exactly sure what's the best way to achieve this, as xUnit does not support parametrizing on the assembly/project level.

The approach I use is to write the tests in an abstract class, and then create a derived class per platform, where I just inject the platform in the base class via constructor.

I saw that there's quite some generated code, so maybe the multiplication could be wired into the code generator.

I'll check with my employer whether I get some permission to contribute some work.

[markusschaber]

I don't know the exact relationship between TestableIO and Testably projects. Which project would be the better place for this kind of contribution?

I've been under the impression that the Testably implementation might be the more advanced and "closer" to native FS behaviour, but we now have some unit tests which pass with native Windows and Linux FS, as well as the MockFileSystem from TestableIO 20.0.15, while they fail with Testably.Abstractions.Testing 2.6.1. (I did not have time to investigate any details yet.)

[vbreuss]

I wrote the Testably file system, because I had some requirements, that were not really feasible in the TestableIO testing helper. My main goal was to be as close as possible to the real file system, which is why all unit tests in Testably.Abstractions.Tests run both against the real and the mock file system (this is why I used source generators).
In order to simplify the transition, I worked in TestableIO to separate the interfaces from the wrappers and used the same interfaces.

The change you are proposing, is probably easier to implement here, even if it is still a huge change!

[vbreuss]

but we now have some unit tests which pass with native Windows and Linux FS, as well as the MockFileSystem from TestableIO 20.0.15, while they fail with Testably.Abstractions.Testing 2.6.1.

I would be really interested in these failing tests. Could you create issues for them, so that I can help investigating?

[vbreuss]

@markusschaber
I implemented some preliminary changes, so that the Execute and Test classes that encapsulate the OS-specific behaviour in the MockFileSystem and the tests respectively, are no longer static. The main obstacle now is, to implement the Path-specific behaviour in the PathMock class.

[markusschaber]

Thanks a lot. I still got no reply to my inquiry, but I intend to contribute.

[vbreuss]

Thanks a lot. I still got no reply to my inquiry, but I intend to contribute.

As a starting point, you can look at this commit, which adapts the tests to run against all platforms. The test run, but currently fail (probably due to the incorrect Path handling):

[markusschaber]

Sorry for keeping you hanging for so long. I now have the official permission to contribute. I'm a bit under water right now, but I'll be able to work on this soon. thanks for your patience.

[vbreuss]

Thanks for the information. No worries 😄

[vbreuss]

I finished the implementation of this feature with #576.
All tests now run successfully against

• The RealFileSystem and the native MockFileSystem (as before)
• The simulated MockFileSystem for Linux, MacOS and Windows

I added a short explanation to the README.md

Limitations:

• The simulation does not work on .NET Framework
• The simulation does not work for the Testably.Abstractions.AccessControl extensions, because the ACL-related classes cannot be used (and therefore also not simulated) on Linux or MacOS.

@markusschaber: Could you please have a look, if this solves your use-case?

[github-actions]

This is addressed in release v3.2.0.

[markusschaber]

@vbreuss Sorry for being silent so long, I've been quite busy and distracted with other problems. I'll definitely come back to this issue, but it might take one or two more weeks.

[markusschaber]

I finally got around to test the code, and the platform abstraction part seems to work fine. I could successfully adapt our tests to run on the simulated platforms after adapting some code in our tests to consider MockFS.SimulationMode when trying to determine platform dependent behavior.

I also could implement central workarounds for our failing unit tests, so they're not a blocker anymore. This allowed me to run all platform independent test cases on all simulation targets. Apparently, the MockFS still has some problems when directories are created (or renamed to) a name ending in slash or backslash, and it also doesn't throw exceptions when calling EnumerateFileSystemInfos() on a file. I stripped down our tests for some minimal repro tests below:

FsTests.cs

If you're interested, I could try to prepare a fix and pull request for those problems.

But so far, it seems to be a success, allowing us to test (most of) our Windows specific file handling code in the Linux based CI/CD pipeline, and the Linux specific code on the Windows develomer machines. Thank you very much!

There are some other minor differences, e. G. when trying to read a directory as file, where the platforms throw different exceptions, and it seems that, sometimes, the Mock throws a different exception. Investigation pending, and low priority.

I did not yet investigate our explicit platform dependent tests, which rely on implementation details like concurrent access behaviour - Windows tends to throw Exceptions when someone tries to write, move, replace or delete a file which has open file handles (or folders containing such files), while Linux doesn't. Some of our tests verify the handling of those exceptions in our code, and I did not check yet whether they can be simulated on Linux with MockFS.

[vbreuss]

@markusschaber: Great to hear, that the solution works for you!

If you're interested, I could try to prepare a fix and pull request for those problems.

Yes, a pull request would be welcome.
Maybe create a separate issue for the failing tests, so that the comments can stay on topic?

There are some other minor differences, e. G. when trying to read a directory as file, where the platforms throw different exceptions, and it seems that, sometimes, the Mock throws a different exception. Investigation pending, and low priority.
I did not yet investigate our explicit platform dependent tests, which rely on implementation details like concurrent access behaviour - Windows tends to throw Exceptions when someone tries to write, move, replace or delete a file which has open file handles (or folders containing such files), while Linux doesn't. Some of our tests verify the handling of those exceptions in our code, and I did not check yet whether they can be simulated on Linux with MockFS.

Also here I would be interested in more details, if you can share them. There is quite some OS-dependent logic for concurrent file access in InMemoryStorage and in the InMemoryContainer.RequestAccess method, but probably some edge cases were missed?