Preliminary Implementation of Wiki Farm Support for Canasta #250

chl178 · 2023-06-05T02:37:51Z

Related Issue #57

This pull request outlines my progress on the project as part of the Google Summer of Code (GSoC) program. The objective of the project is to implement support for a wiki farm in Canasta, a Docker-based MediaWiki distribution.

Background

Currently, Canasta offers a streamlined way to set up a feature-rich MediaWiki instance on virtually any server. However, it does not currently support running multiple wikis, or a wiki farm, within the same container. This project aims to fill this gap and provide the ability to run different wikis in the same container. The wikis could vary by directory (e.g., example.com/a, example.com/b), subdomain (e.g., a.example.com, b.example.com), or completely different domains (e.g., example1.com, example2.com).

In addition, this project intends to extend Canasta's command-line interface (CLI) to support wiki farm configuration. This would allow administrators to effortlessly create, manage, and delete individual wikis.

Implementation Approach

To simplify management and facilitate use, I've implemented a shared common setting that applies to all wikis. Alongside this commonality, each wiki is distinguished by its unique ID and has its own customized settings. This allows for the definition of unique skins, extensions and configurations for each wiki. In this way, multiple wikis can run independently in a single container.

For wikis under directories, I have scripted an automatic generator generatewikihtaccess.sh for .htaccess files to manage access permissions.

Test

As you can see three different wiki a, k and local run independently in one container.

They are under different domain names and different directories.

Next Steps

While this initial implementation is a significant step towards our goal, there is still considerable work to do. The next phase of this project will focus on developing the Canasta CLI to make it even more user-friendly and efficient in managing wiki farms.
I'm looking forward to feedback and suggestions on how we can improve this implementation and successfully deliver the project's objectives.

chl178 · 2023-06-05T03:25:19Z

For people who want to test it locally

As this is an early version for wiki farm support for Canasta, it currently does not support CLI installation and can only be manually installed using docker-compose. You may refer to the official docker-compose manual installation documentation for guidance.
The test template for docker-compose.

Modify your hosts file to add an alias for 127.0.0.1 and restart the Apache service.
Download my canasta code and create an image by docker build -t canastafarm .
Replace the web image in the docker-compose.override.yml file with the image you've just built.
Modify the caddyfile file in the docker-compose's config, add an alias in the form of http://{corresponding alias}.

Note: Because this is local testing, if you use https, you might not be able to access it due to the web server lacking an SSL certificate. Therefore, you need to change https to http in order to disable Caddy's https.
Copy .env.example to .env, and modify MW_SITE_SERVER and MW_SITE_FQDN in the .env file, changing them both to start with http.
Create a wikis.yaml file and place it in the config directory.
Run the command docker-compose up -d
Navigate to the installation address and install MediaWiki.
After installation, restart the installation and choose a different database for another wiki (repeat as many times as the number of wikis you wanted to install).
Run the command docker-compose down
Place the LocalSettings.php into the config, and add the commonsettings.php and each wiki's customized LocalSettings_wikiID.php.
Run the command docker-compose up -d to complete the installation.

My docker-compose structure:

chl178 · 2023-06-05T17:25:47Z

@yaronkoren @jeffw16

_sources/canasta/CanastaDefaultSettings.php

_sources/canasta/FarmConfigLoader.php

_sources/canasta/CanastaDefaultSettings.php

_sources/canasta/FarmConfigLoader.php

vedmaka · 2023-07-16T14:56:42Z

_sources/configs/.htaccess

@@ -1,5 +1,9 @@
 RewriteEngine On

+# Capture original request URL
+RewriteCond %{THE_REQUEST} \s(.*?)\s
+RewriteRule ^ - [E=ORIGINAL_URL:%1]


Are there any benefits over using $_SERVER['REQUEST_URI'] ?

vedmaka · 2023-07-16T14:59:33Z

_sources/canasta/FarmConfigLoader.php

+$path = null;
+
+// Retrieve the server name and request path if available
+if (isset($_SERVER['SERVER_NAME'])) {


Afaik, we don't have UserCanonicalName and ServerName set in Canasta Apache configurations and thus this makes the SERVER_NAME spoofable, see https://shiflett.org/blog/2006/server-name-versus-http-host

What exactly is the problem with it being spoofable? Perhaps it is the intention of a legitimate user to access a specific wiki by setting it manually in a cURL request or something like that.

_sources/canasta/CanastaDefaultSettings.php

_sources/scripts/run-apache.sh

vedmaka · 2023-07-16T16:03:08Z

Some general comments:

It looks like some changes to the job runner scripts are also necessary, to make it run against all the farm wikis ( mwjobrunner.sh , mwsitemapgen.sh , mwtranscoder.sh )
The update.php run also need to be changed by adding support for multiple wikis
Images directories ($wgUploadDirectory) need to be also altered per wiki. Otherwise, files will be overwritten
Not quite sure about Canasta preference here, but maybe it'd be nice to have the farm mode switch controlled not by wikis.yaml or CommonSettings.php file presence but by some ENV variable instead

Please do not treat my comments as a call to action but as suggestions

_sources/canasta/CanastaDefaultSettings.php

_sources/canasta/FarmConfigLoader.php

_sources/scripts/generatewikihtaccess.sh

_sources/scripts/run-apache.sh

jeffw16 · 2023-07-17T18:59:44Z

_sources/scripts/run-apache.sh

@@ -180,9 +186,11 @@ cd "$MW_HOME" || exit

 ########## Run maintenance scripts ##########
 echo "Checking for LocalSettings..."
-if [ -e "$MW_VOLUME/config/LocalSettings.php"  ]; then
+if [ -e "$MW_VOLUME/config/LocalSettings.php" ] || [ -e "$MW_VOLUME/config/CommonSettings.php" ]; then


Again, should change this to support any .php file in that directory.

_sources/scripts/run-apache.sh

_sources/canasta/FarmConfigLoader.php

chl178 · 2023-07-18T22:45:37Z

Some general comments:

It looks like some changes to the job runner scripts are also necessary, to make it run against all the farm wikis ( mwjobrunner.sh , mwsitemapgen.sh , mwtranscoder.sh )

The update.php run also need to be changed by adding support for multiple wikis

Images directories ($wgUploadDirectory) need to be also altered per wiki. Otherwise, files will be overwritten

Not quite sure about Canasta preference here, but maybe it'd be nice to have the farm mode switch controlled not by wikis.yaml or CommonSettings.php file presence but by some ENV variable instead

Please do not treat my comments as a call to action but as suggestions

Thank @vedmaka for your comments and feedback:
We've already initiated the process of resolving the issues concerning the upload and cache directories. This should ensure that files are not overwritten and each wiki in the farm has its own distinct space.

Regarding the ENV variables, we've had discussions with Daniel, a respected MediaWiki community developer. His advice was not to define too many environment variables. As the current practice in the MediaWiki community is to use a YAML file for setting up farms, we've decided to follow this established practice.

We're taking into account your points on modifying the job runner scripts (mwjobrunner.sh, mwsitemapgen.sh, mwtranscoder.sh), and incorporating support for multiple wikis in update.php.

Thanks again for your suggestions. They're not direct calls to action, but we highly value your input.

jeffw16 · 2023-07-19T03:02:04Z

_sources/scripts/run-apache.sh

-}
+run_maintenance_scripts() {
+  # Iterate through all the .sh files in /maintenance-scripts/ directory
+  for maintenance_script in $(find /maintenance-scripts/ -maxdepth 1 -mindepth 1 -type f -name "*.sh"); do


Could we make this order predictable (i.e. load lexicographically) rather than nondeterministic?

jeffw16 · 2023-07-19T03:02:22Z

_sources/scripts/maintenance-scripts/mw_job_runner.sh

@@ -0,0 +1,44 @@
+#!/bin/bash


Rename file to 1_mw_job_runner.sh

Sorry, I only saw this now. I disagree with this - whether or not these three maintenance scripts should get renamed, the renaming should not happen in this patch.

jeffw16 · 2023-07-19T03:02:41Z

_sources/scripts/maintenance-scripts/mw_transcoder.sh

@@ -0,0 +1,33 @@
+#!/bin/bash


Rename file 2_mw_transcoder.sh

jeffw16 · 2023-07-19T03:03:10Z

_sources/scripts/maintenance-scripts/mw_sitemap_generator.sh

@@ -1,5 +1,13 @@
 #!/bin/bash


Rename file 3_mw_sitemap_generator.sh

_sources/canasta/FarmConfigLoader.php

jeffw16 · 2023-07-19T03:19:50Z

_sources/scripts/maintenance-scripts/mw_job_runner.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+RJ=$MW_HOME/maintenance/runJobs.php


Please don't abbreviate this variable. It was confusing to me what it referred to until I found its definition. Please rename it to RUN_JOBS_SCRIPT_PATH.

It might be more verbose, but it's more descriptive. Verbose but descriptive variable names are typically more helpful than concise but nondescriptive variable names.

jeffw16 · 2023-07-19T03:20:12Z

_sources/scripts/maintenance-scripts/mw_transcoder.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+
+RJ=$MW_HOME/maintenance/runJobs.php


Please see my comments on the other file about the naming of this variable.

_sources/canasta/FarmConfigLoader.php

jeffw16

There's simple code style issues that should be fixable with a linter. Let me know when you've fixed them all and then I can continue my code review.

_sources/canasta/CanastaDefaultSettings.php

_sources/canasta/FarmConfigLoader.php

added farm support

ac3ab50

chl178 mentioned this pull request Jun 5, 2023

Wiki farm installation template CanastaWiki/Canasta-DockerCompose#45

Merged

chl178 added 2 commits June 20, 2023 17:25

structure 0.1

4357d04

structure 0.2

3754e84

vedmaka reviewed Jul 16, 2023

View reviewed changes

vedmaka requested review from yaronkoren and jeffw16 July 16, 2023 16:03

jeffw16 requested changes Jul 17, 2023

View reviewed changes

chl178 added 3 commits July 18, 2023 09:27

cleaned my code under coding conventions

3a3db93

cleaned my code under coding conventions

a95a28a

cleaned my code under coding conventions

859b864

chl178 added 2 commits July 18, 2023 19:30

cleaned my code under coding conventions

cdfd6ea

cleaned my code under coding conventions

163dc25

jeffw16 requested changes Jul 19, 2023

View reviewed changes

Added error handling

2e0b746

jeffw16 requested changes Jul 21, 2023

View reviewed changes

chl178 added 7 commits July 25, 2023 20:08

Fixed the code convention by mediawiki composer

069573c

Added canasta_img.php

8d7ce2b

Updated the canasta_img.php with seperate storage

622404e

Add validation when wikis.yaml doesn't exist and clean the code

a9125ad

Merge branch 'master' into farm

2282a8c

Update to HTTPS

1900035

Canasta2.0 adding farm support feature

eaa1c24

chl178 closed this Sep 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preliminary Implementation of Wiki Farm Support for Canasta #250

Preliminary Implementation of Wiki Farm Support for Canasta #250

chl178 commented Jun 5, 2023 •

edited

Loading

chl178 commented Jun 5, 2023 •

edited

Loading

chl178 commented Jun 5, 2023

vedmaka Jul 16, 2023

vedmaka Jul 16, 2023

jeffw16 Jul 17, 2023

vedmaka commented Jul 16, 2023 •

edited

Loading

jeffw16 Jul 17, 2023

chl178 commented Jul 18, 2023 •

edited

Loading

jeffw16 Jul 19, 2023

jeffw16 Jul 19, 2023

yaronkoren Aug 9, 2023

jeffw16 Jul 19, 2023

jeffw16 Jul 19, 2023

jeffw16 Jul 19, 2023

jeffw16 Jul 19, 2023

jeffw16 left a comment

		@@ -0,0 +1,44 @@
		#!/bin/bash

		RJ=$MW_HOME/maintenance/runJobs.php

		@@ -0,0 +1,33 @@
		#!/bin/bash

		RJ=$MW_HOME/maintenance/runJobs.php

Preliminary Implementation of Wiki Farm Support for Canasta #250

Preliminary Implementation of Wiki Farm Support for Canasta #250

Conversation

chl178 commented Jun 5, 2023 • edited Loading

Related Issue #57

Background

Implementation Approach

Test

Next Steps

chl178 commented Jun 5, 2023 • edited Loading

For people who want to test it locally

chl178 commented Jun 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vedmaka commented Jul 16, 2023 • edited Loading

Choose a reason for hiding this comment

chl178 commented Jul 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeffw16 left a comment

Choose a reason for hiding this comment

chl178 commented Jun 5, 2023 •

edited

Loading

chl178 commented Jun 5, 2023 •

edited

Loading

vedmaka commented Jul 16, 2023 •

edited

Loading

chl178 commented Jul 18, 2023 •

edited

Loading