AlchemyCMS can become extremly unresponsive with thousands of pages.

[nicolai86]

Hey,

this is more a (re-)collection of problems we're facing, preparing our installation of alchemy (= 3.2.0) for production, and our mitigations to these problems. Hopefully they might be useful for others when evaluating alchemy as a CMS.

background:
We decided to use alchemy as a replacement for a medium sized CMS setup with ~100k pages. Our setup is completely hosted in AWS, using postgreSQL as database.

problems:

• having more than ~1k pages increases the backend load time to a point where it becomes unusable (> 60s load times). This is because some backend functionality requires the entire page hierarchy to be in DOM (display, search, reorder). Fetching & rendering pages becomes a bottleneck, which we solved by rewriting the alchemy admin pages backend.
• replacing the backend also meant that search & reorder needed to be implemented again, in a manner that supported useable autocompletion (e.g. search for pages by hierarchy fragments…). We did this by introducing a SQL view with a recursive CTE (this removes MySQL & sqlite support).
• having more than ~1k pages also meant re-writing alchemys internal link editor, which also worked by rendering all pages to the DOM, at least for internal links.
• having more than some thousand pages made managing the backend slow mostly because of some important indices missing on alchemy_pages table. Easy to catch /w the appropriate monitoring in place.
• having more than some tens of thousand of pages brings down the database because of SQL queries generated by awesome_nested_set on page insertion, deletion & reordering. We did not look into improving awesome_nested_set but are actively looking into removing it from alchemy all together. This means binding yourself to a specific version of alchemy, with high costs of migrating to newer, maybe better versions. closure_tree looks like an interesting contender to solve this issue.

To summarize: alchemy_cms seems to only work well with some hundred pages. If you have more pages you need to fallback to monkey patching, or even worse, a private fork.
However, YMMV.

related:

• PR with improved link editor [feature] load pages asynchronously in link editor #889

[tvdeyen]

Hey, yes, this is already discussed and the solution would be to separate the navigation tree from the pages.

If we do that, then lots of these issues will be gone IMO. Especially the performance issues of the nested set. I presume, that, if you don't have all these thousand of pages in the navigation, right? And even then, we could change that to an eager loading tree.

Having the pages a normal resource, we can load them paginated and in chunks and we could add real search/autocomplete.

I published the branch, where the feature is almost finished (https://github.com/AlchemyCMS/alchemy_cms/tree/4.0/nodes). Some specs are failing and need to be rewritten for the new implementation. And we need some nice upgrade path.

We would be super happy to address these issues!

[tvdeyen]

And here is the Trello Board: https://trello.com/b/EDoGFAul/alchemy-4

[tvdeyen]

@nicolai86 please PM me, so I can add you to the Trello Board.

[robinboening]

Thanks for sharing your experiences @nicolai86.

Thomas already mentioned the node branch (best thing since sliced bread, at least for Alchemy) ;).
I am not sure, but I think the node branch is like 90% ready to work, so you can save a lot of time if you use this branch, catch up with the master and finish it. It will definitely solve your performance issues. The other great thing is we can merge that into master and you don't need to fork and cook your individual thing.

Thomas was working on it last year but there was no time to finish it. So, you could also hire us to finish it, I think it makes sense because Thomas knows all the details.

[nicolai86]

@tvdeyen we do not use the alchemy navigation at all - we just use the sitemap & many stand alone pages.

@robinboening at this point we've a "stablish" fork of alchemy 3.2 which contains fixes for all the above issues and more, so waiting for alchemy 4 is no option for us. Hopefully, given time, we'll be able to contribute some of the changes back.

[tvdeyen]

Just stumbled upon http://mceachen.github.io/closure_tree/
Looks very promising.

@nicolai86 thanks. Would be great if you could invest the time to contribute back.

[nicolai86]

@tvdeyen closure_tree is what I used as a replacement, and it worked great. You should probably investigate it as an alternative. Sadly I won't be able to contribute a patch for this change since I moved on since opening the issue.