[FEATURE REQUEST]: Update docs regarding tests

[JavierLight]

Description
First of all, sorry if this is something that has been already replied to. I searched "tests" in the issues (open and old) and couldn't find any useful information.

I've been unable to understand how to deal with tests involving a spark session with a specific configuration. My code instances a SparkSession with a configuration that I would like to test in my CI pipeline (check that the remote HDFS warehouse is accessible and so on). I'm using local as the resource manager for the sake of simplicity.

If I write a test using xUnit to test this service, I'm getting an error from the JVM Bridge when building the spark session:

System.Exception : JVM method execution failed: Nonstatic method 'getOrCreate' failed for class '1' when called with no arguments

I've seen you have E2E tests with a fixture that I tried to replace with a docker container listening on the port 5567. The output in the container logs show nothing when running the test (but if I don't set up the container the test fails because the port 5567 is not listening).

However, if I run the same test inside the container (sharing a volume with my Test.dll) it works perfectly fine. I'm not sure if what I'm trying to do in my local environment should work or if I am missing something.

Describe the solution you'd like
I think it would be nice to provide a brief explanation in the docs to be able to deal with tests that need to test the integration for a specific implementation of this library.

[Niharikadutta]

@JavierLight Thanks for your question! If I understand your setup correctly, you are running the Dotnet backend process in your docker container listening on port 5567, and your application in your local box? If that is correct, this is currently not supported, but is being worked on in #537 . As of today the dotnet backend only listens on loopback address, which is why when you run the test inside the container it works.

[Niharikadutta]

As for documentation, we take that as important feedback, and will work on making the test docs more comprehensive and explanatory for your and similar scenarios. Please let us know if that helps, thanks!

[JavierLight]

Thanks for the explanation. I hope #537 will be available soon :) in the meanwhile we will run these tests inside a docker container.