Potential issue scaling out Oqtane

[knight1219]

I've been reading through some of the code and I have a interest in NoSQL solutions (MongoDB is my preferred NoSQL of choice). As I was looking around how the code interacts with the SQL database as it sits now I see potential areas of improvement:

1. Create a IDatabase interface that the modules can then use for CRUD operations. This would allow the creation of different database providers that can be switched out at startup. I understand this would be a quiet large task to undertake, but feel it would allow a much more flexible system to be built.

2. If Oqtane was deployed to a system that auto-scales (like Kubernetes in my instance and I'm sure something like Azure) doing a replace on the app settings file would only change the one pod instance that did the install and all other pods wouldn't get this change. There are ways around this with persistent volume claims but its not a very straight forward process. Could thought be given to how this could be handled in a scalable environment?

But other then that this seems like an amazing framework and should get people up and running with Blazor and a website very easily. I cant wait to get my hands into creating some modules and themes for this.

[sbwalker]

In regards to 1, I can understand the theoretical benefit of supporting multiple databases. However from a practical perspective there is obviously the initial implementation effort - but the far greater challenge is maintaining this capability through subsequent versions as you need to deal with migrations for each supported database, regression testing, etc... In addition, third party developers creating modules would also be expected to support multiple databases which based on my experience with DNN is not feasible. My personal opinion is that the benefit of supporting multiple databases in a framework like Oqtane is far outweighed by the long term costs.

In regards to 2, I am very keen on allowing Oqtane to support modern infrastructure options such as containers and Kubernetes. For the most part the goal is to maintain the app settings in the database rather than in a static settings file. Further research will be required in this area.

[knight1219]

I think #184 will a very good first pass at creating a database agnostic platform. Once that gets merged I would love to see if I could take that work and create a drop-in replacement with something like MongoDb. I can see supporting multiple database could be difficult based on database schema migrations (they all have their own way of doing things) but if the migrations are always just scripts then it's up to the InstallationManager to know how to handle the different databases and find the fight set of scripts. Module developers could support multiple databases if they so choose, or state that they only support one. I mean personally, as much as I don't want to spin up a $5 a month Azure SQL Server database just to test this I will. You now automatically have a dollar amount in just testing this product out in a real world situation, which is a bit hard to swallow. (I will because after seeing how this framework works I think it could jumpstart Blazor developer 1000-fold).

As for the app settings (and for install-able modules/themes for that matter), I've been thinking about this and it's a bit of a catch-22: Each container needs do know about the the appsettings file and the downloaded modules going forward. The only thing I can think of is having a set of tables that allows you to know how many instances of Oqtane are connected to your database. Then when you change a setting/install a module, there would be a HostedService that could check these tables and compare its installed modules with the database and download the packages (not sure where from or even how). Then all instances report the package is downloaded and once all containers have them installed they are available to be used on a page. NOW, this would also have to happen at startup as well because kubernetes pods are stateless.. they can be torn down and spun up at will so the hosted service would need to run and do all of this setup work for modules before it could ever signal to k8s it is ready to accept traffic.

[mikecasas]

@knight1219 Why not use a local SQL instance to do any testing? I run SQL Server on my laptop using Docker.

[knight1219]

I can.. and for POC that would work, but past that I'm not sure it would. I am desperately trying to get my company on the Docker/container bandwagon. Everything here is very locked down so I cant even install docker.

[knight1219]

Not sure I want to issue a pull request quite yet, but this is my WIP for moving some the classes around to better support module/theme creation. Added in some of the work from #184's PR as well, just haven't been able to fully implement yet. There is also a test module using the new separated libraries that can be uploaded, installed and added as well. Still has some work to do before I would be ready to issue a PR tho.

https://github.com/knight1219/oqtane.framework

[sbwalker]

Did you look at the existing sample Blogs module which is included in the Oqtane repository: https://github.com/oqtane/oqtane.module.blogs? I am asking because the Blogs sample module is a real world implementation developed as an independent project with multiple components making up the user interface, back-end services, and installation with provisioning via database scripts.

[knight1219]

I did, and that's where my push to do some refactoring came from. In order to create a module you would have to reference everything in Oqtane, which seems a little extreme. I've refactored out everything I've been able to see that would be the minimum to create an Module without making the developer need to package the whole framework. They only need to know about interfaces and now they do. I still need to see how packaging up server components (api controllers and services) would be but I think I'm on the right track.

[sbwalker]

I looked at your fork and you have definitely done a lot of refactoring. Based on the magnitude of changes it’s actually quite difficult to determine why some of the changes were made. I would definitely not be able to accept this as a single pull request. Pull requests need to be more granular so that it is clear about the the specific problem they are solving. Perhaps once you get further along it will be possible to cherry pick some of the refactoring into a new branch which will make it easier to submit PRs for review and integration.

[sbwalker]

Since the approach is that a module would ultimately be deployed into an Oqtane framework installation I am not sure why it’s a problem to have to reference the framework projects during development? Once you build a more real world module you will notice that there are a lot of Oqtane core services you will want to leverage.

[knight1219]

My mindset is only give the developers the needed features to build a module. No real need to ship anything other in interfaces if I can create a Razor Class Library and create a module (which I've been able to do). Which is why I split out things into the Oqtane.Core.* libraries, just the bare bones to create modules. Do they really need access to the core modules in 3rd party development? Implementations of services or repositories?

Maybe my refactor is coming to soon in Oqtane's development, but I'm all for doing legwork upfront so i don't have to do it later. Or maybe my thinking is just to far off from where yours is.

[mikecasas]

I'm a bit in the middle, just create three Nuget packages for the Framework: Client, Server, and Shared, and then as you build your module, just reference those.

[sbwalker]

Currently the Blog sample module has 3 projects - Client, Server, and Shared ( exactly the same as the core framework ). These projects have dependencies on the corresponding Oqtane.Client, Oqtane.Server, Oqtane.Shared DLLs - so it’s quite simple to set up. The Blog module utilizes quite a number of framework features including services, generic UI components, helpers, base classes, etc... It is true that a simple Hello World module would not utilize many framework features - but I don’t expect people to build only Hello World modules. This is the reason why I created the Blog module - so that it demonstrates a real world scenario.

[mikecasas]

No, I know, I am working through my own module based on that. Did you see my question about multi-tenancy in that repo? Thanks for answering all these questions. I plan on helping out on the project as much as possible!

https://github.com/oqtane/oqtane.module.blogs/issues/23#issue-550788839

[PizzaConsole]

Isn't this thread title a little vague? It seems like this thread is adressing multiple issues. Shouldn't this be changed to adress the first issue that was addressed about supporting multiple Database infrastructures? It is a lot easier to link specific issues to commits that way.

[knight1219]

Your right. I was digging into one thing and hit a rabbit hole of things. This can be closed for now i guess.