Orchard Core in a Web Farm

[randaratceridian]

Is there any documentation that lists what is required to run Orchard Core in a web farm, running in Azure? We were running with multiple servers and saw some odd behavior with saving, so we quickly fell back to a single web server until we could figure it out. But we need to bring up another web server soon.

At one point, we were concerned it was a caching issue. We thought we might need to add some external caching e.g. Redis. But it wasn't based on any concrete evidence.

I also wanted to confirm we don't need sticky load balancing.

Are there any docs we can refer to?

[Skrypt]

The Distributed Cache feature is currently developped by @jtkech which should solve most issues with scaling up OC. There is no documentation yet as it is WIP.

[randaratceridian]

Thanks @Skrypt. Reading between the lines, does that mean that OC will not work in a web farm right now?

[Skrypt]

It works, but the data cached on each instance will be different unless you do an app restart periodically to refresh the local cache of each instance. So you will experience unsynced data from one instance to the other which might not be so great for production on some use cases.

[randaratceridian]

@Skrypt or @jtkech. Any estimate on when distributed caching will be available in dev branch or released?

[Skrypt]

There's a PR for Redis and DistributedCache. Feel free to try it and provide feedback to @jtkech.

#5249
#5815

I can't say for the estimate.

[jtkech]

Not before rc2 that will come very soon, then maybe one or two months after

#5249 uses new caching / states patterns using distributed services and #5815 defines concrete implementations of these distributed services, so we will need both PRs. That said #5815 can be tested alone as it is compatible with the previous caching and signal token rules.

Note: Not completely finished but most of the work has been done to keep things in sync

[randaratceridian]

@Skrypt or @jtkech. I know this is a change of topic, but "rc2 that will come very soon" confused me. I thought rc2 was already out based on this https://docs.orchardcore.net/en/ag-releasenotes/docs/releases/1.0.0-rc2/ ? We were just about to start upgrading to rc2. If it's not out, when will it be out?

[Piedone]

Not yet, the docs page you linked is in the in-progress ag-releasenotes branch. The latest current docs you can see under https://docs.orchardcore.net/.

[randaratceridian]

One thing to clarify. Is all the information in Redis transient? i.e. is it only used as a cache? I saw a reference to Redis Message Bus in the PR. What is it used for? We just want to know how much we care if we lose all the data in Redis.

[jtkech]

If you make a service needing persistent data in redis, it's up to you to configure your redis instance, right now we only use it for caching, not as a persistent store by itself, so all data can be flushed, they will be retrieved (or rebuilt from other data) from a shared store. Also used for shared states that are transient by nature and doesn't need any persistent data.

In fact it is a multi level cache composed of a scoped cache, a local memory cache and a distributed cache. We will have a generic service to keep in sync any single document between a persistent store and this multi level cache.

Based on the same service you can also have a volatile document without any persistent storage, in that case we only keep in sync the local memory cache with the distributed cache. Note: Here the document is first build with a provided factory that may use (or not) other data from a persistent store or whatever you need if you make your own service.

About the message bus, before it was used to invalidate local memory caches, so without using a distributed cache. Now we don't use the same pattern as our multi level cache include a distributed cache. It was also planed to be used e.g. to keep in sync tenant states when there are enabled / disabled / released / reloaded, but here also maybe we will do it without a message bus, i'm working on. So if we can do all we want without using it, maybe we will remove it.

[olegbogomaz]

@Skrypt and @jtkech, I was just wondering if there are any updates on Redis and Distributed Cache feature implementation? Is there a timeline when it may be available?

I also wanted to ask a couple of questions to clarify a few things.

1. How much load can we expect on the Redis servers? How powerful of an instance would a mid sized Orchard Core site require?
2. How will the application behave when Redis is down?
3. To confirm one more time, can all data be flushed safely? Is there storing of any session information?

Thank you!

[jtkech]

@olegbogomaz

Yes there are 2 pending PRs, #5249 whose main goal is to keep in sync documents between a distributed cache and a document store, and #5815 for e.g. a concrete implementation of a distributed cache based on redis.

I merged recently these 2 PRs in a new branch, so i'm reworking on it

How much load can we expect on the Redis servers? How powerful of an instance would a mid sized Orchard Core site require?

So, it's a bit early to have precise measurements

That said, we will cache shared documents as site settings, content definitions, layers, queries, templates, roles and so on, you can already see these documents in the database. In fact we will use a multilevel cache meaning that each running instance will also use a local memory cache, and for a given document we will only get from the distributed cache an identifier to see if the local cache need to be refreshed, and this only when you request this document.

So, once the distributed cache and all local caches have been set up, if you don't update the shared documents only their identifiers are retrieved from the distributed cache and only when requesting the related documents. Note: The multilevel cache also has a scoped cache, so a given identifier is requested once per request.

So here you can have an idea of the amount of data that will be cached, and it will depend on the rps of your site and how many shared documents you are using per request, and if you are updating frequently or not these documents.

Also, currently i'm working to keep in sync tenant states, e.g. when you enable / disable, release / reload a tenant e.g. after adding a feature, changed some settings. Because we don't want to use a message bus, here we will check shell identifiers periodically through an IHostedService , so here it will depend on the number of tenants.

How will the application behave when Redis is down?

Good question, we have to think about it ;) Currently it fails saying that it can't connect to redis, but you can remove the redis config e.g. from the appsettings.json and then restart the app. By default you don't loose any needed data because by default they are persisted in a shared database.

Note: We will also have volatile documents that are kept in sync between the distributed cache and local caches but that are not persistent, but it's up to you to use these kind of documents, we use some volatile documents but only when they can be rebuilt from other persistent data.

To confirm one more time, can all data be flushed safely?

Yes

Is there storing of any session information?

To preserve data between requests we don't use Session state by default, we will have a redis module with a redis cache feature that will register (among other things) an IDistributedCache implementation based on redis, then it will be up to you to do e.g. services.AddSession(options => ...), but in a tenant context meaning through the ConfigureServices() of a module startup, or from the app through our helpers e.g. .AddOrchardCms.ConfigureServices(tenantServices => ...)

[randaratceridian]

@jtkech and @Skrypt. I wanted to check in on this thread. We are going to need Web Farm support very soon. We are currently running RC1 and it wouldn't be trivial to upgrade at this point.

So our thought was:

1. Merge in the two PRs DistributedCache #5249 and Redis Services #5815, into our RC1 based codebase
2. Work through any conflicts
3. Test with local Redis instances
4. Test with our QA environment, which will be a web farm

How challenging do you think 1. and 2. will be at this point?

[jtkech]

For infos #5815 has been merged in the dev branch and i'm working on #5249, so better to wait a little more

About #5249, recently i added the ability to keep in sync tenants accross instances when you enable / disable / edit / create / setup a tenant (need a stateless configuration), i did a little demo recently, be indulgent for my english in this demo ;)

But the #5249 is still marked as not ready because i'm doing some last important tweaks

[Piedone]

I'd really recommend you upgrade to the latest dev containing #5815 (and wait a bit for #5249) using the preview feed instead of trying to cherry-pick the changes. You'll need to do the upgrade at one point anyway and back-porting such changes is really-really painful and risky. At that point, upgrading is a lot more efficient IMO.

[Piedone]

FYI a small rundown of what you need: #15149 (comment).