Does anybody use spark .net in production yet?

[apostarn]

Hi,

Does anybody use spark .net in production yet and if yes, what are the use cases?
I'm thinking of using it in financial sector to parallelize the calculation of various risks for tradeable assets.
Would like to get an idea if there is anybody out there who already done approximately the same in production and how successful the setup is.

[MikeRys]

Hi Andrey We have several Microsoft teams and other customers using .NET for Spark. We have some customer testimonials in our v1.0 announcement at https://techcommunity.microsoft.com/t5/azure-synapse-analytics/microsoft-and-the-net-foundation-announce-the-release-of-version/ba-p/1820908. Please note that a lot of production customers are probably not here on Github, since many of them are using it through their HDInsight or Azure Synapse Spark experiences. If you have any further questions on how to use it, feel free to reach out. What Spark engine/platform are you looking at using? Cheers Michael From: Andrey Postarnakov <[email protected]> Sent: Wednesday, January 13, 2021 12:38 AM To: dotnet/spark <[email protected]> Cc: Subscribed <[email protected]> Subject: [dotnet/spark] Does anybody use spark .net in production yet? (#809) Hi, Does anybody use spark .net in production yet and if yes, what are the use cases? I'm thinking of using it in financial sector to parallelize the calculation of various risks for tradeable assets. Would like to get an idea if there is anybody out there who already done approximately the same in production and how successful the setup is. - You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fspark%2Fdiscussions%2F809&data=04%7C01%7Cmrys%40microsoft.com%7Caefc0e68092a469b73a408d8b79e9988%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637461239101361378%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rG72fQ3VmYwEXhWk0EAyhMd4qQhLKHuoLOUvaTYnJHE%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACZXGJAXAK577MQVEH42VITSZVLYHANCNFSM4WAOBW5Q&data=04%7C01%7Cmrys%40microsoft.com%7Caefc0e68092a469b73a408d8b79e9988%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637461239101361378%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=OXQjgQUr9wI1%2F9zeJd3VCit3nngv%2FcE8Awc9E%2B5CT7g%3D&reserved=0>.

[dbeavon]

Please note that a lot of production customers are probably not here on Github, since many of them are using it through their HDInsight or Azure Synapse Spark experiences.

Can you tell me if you are aware of anyone using this in production within Azure Databricks? Prior to the 1.0 GA release, there were warnings in the docs about the fact that .Net for Spark were unsupported in production. Now it appears the warnings are all removed , see:
https://docs.microsoft.com/en-us/dotnet/spark/tutorials/databricks-deployment

... So I'm assuming that the folks at Azure Databricks are now ready to support this? (... of course I would probably come ask about my problems in the github community as well ... but sometimes managers like to know that there was a Microsoft support ticket that was created, for whatever it's worth).