Skip to content

MinIO

Jump to bottom
ekatchko edited this page Jan 19, 2023 · 2 revisions

Setting up a MinIO Stratum 0 -> Stratum 1 client

Start an instance with Ubuntu, run sudo apt update and sudo apt upgrade and reboot the machine.
Install the MinIO client as described on their MinIO mc reference page and set the alias for the S3 Stratum 0 source bucket and the S3 Stratum 1 target bucket as described in Create an alias.
Create a log file for MinIO where the ubuntu user has read and write permissions, e.g. at /home/ubuntu/minio/minio.log.
Next, create a file mirror.sh (e.g. at /home/ubuntu/minio/mirror.sh), make it executable and add:

#! /bin/bash
source /home/ubuntu/.API_KEY
EXCLUDE_STRING=$(curl -s -H "X-AUTH-HEADER: ${API_KEY}" -X GET "FTS3-CLIENT-IP-PORT/transfers?unfinished_only=true&output_for_minio_exclude=true")
/home/ubuntu/minio/mc mirror --quiet --remove --overwrite ${EXCLUDE_STRING} ALIAS-SOURCE/STRATUM_0_BUCKET_NAME ALIAS_TARGET/STRATUM_1_BUCKET_NAME

Switch FTS3-CLIENT-IP-PORT with the ip and port where your FTS3-Client runs. The mirror.sh file now asks the FTS3-Client for unfinished jobs it should exclude when mirroring, which makes sure that only finished jobs are mirrored to the Stratum 1 bucket.

Next, create a crontab with crontab -e and add:

*/15 * * * * /usr/bin/flock -n /tmp/mc.lock -c "/usr/bin/date >> /home/ubuntu/minio/minio.log 2>&1; /home/ubuntu/minio/mirror.sh >> /home/ubuntu/minio/minio.log 2>&1"

Change the path accordingly if you stored the files at a different path.

Now MinIO tries every 15 minutes the mirroring from the Stratum 0 source bucket to the Stratum 1 target bucket.

Setting up a MinIO Stratum 1 -> X client

Start an instance with Ubuntu, run sudo apt update and sudo apt upgrade and reboot the machine.
Install the MinIO client as described on their MinIO mc reference page and set the alias for the S3 Stratum 1 source bucket and the S3 target bucket as described in Create an alias.
Create a log file for MinIO where the ubuntu user has read and write permissions, e.g. at /home/ubuntu/minio/minio.log. Next, create a crontab with crontab -e and add:

*/15 * * * * /usr/bin/flock -n /tmp/mc.lock -c "/usr/bin/date >> /PATH/TO/YOUR/minio.log 2>&1; /PATH/TO/CLIENT/EXECUTABLE/mc mirror --quiet --remove --overwrite ALIAS-SOURCE/SOURCE_BUCKET_NAME ALIAS_TARGET/TARGET_BUCKET_NAME >> /PATH/TO/YOUR/minio.log 2>&1"

Now MinIO tries every 15 minutes the mirroring from the Stratum 1 source bucket to the S3 target bucket.

Clone this wiki locally