Skip to content

Latest commit

 

History

History
312 lines (264 loc) · 8.86 KB

README.md

File metadata and controls

312 lines (264 loc) · 8.86 KB
Raw

FTP backup

Backup a data-oriented container and a MariaDB/MySQL container and upload them to a FTP/SFTP server using lftp.

This script will archive your /data (or custom) folder and use mysqldump to backup your database.

ENV variables

  • FTP_USER - FTP server username
  • FTP_PASS - FTP server user password
  • FTP_HOST - FTP server hostname
  • FTP_PORT - FTP server port
  • FTP_PROTO - Protocol to use (default: ftp)
  • LOCAL_PATH - Absolute path for folder to backup (default: /data)
  • REMOTE_PATH - Your FTP backup destination folder

Optional env vars to control archiving and FTP upload

  • COMPRESS - (Optional) Default: 1, compress files TAR archive
  • CHUNK_SIZE - (Optional) Default: 0, in mega Bytes, splits TAR archive into parts for better FTP upload with very large archives
  • PARALLEL_UPLOADS - (Optional) Default: 3, only for split archives, max parallel uploads at the same time

Optional env vars to dump MySQL or PostgreSQL databases

  • DB_USER - (Optional) MySQL user name
  • DB_HOST - (Optional) MySQL host name
  • DB_PASS - (Optional) MySQL user password
  • DB_NAME - (Optional) MySQL name
  • PGDATABASE - (Optional) PostgreSQL Database nam
  • PGHOST - (Optional) PostgreSQL host name
  • PGOPTIONS - (Optional) PostgreSQL options
  • PGPORT - (Optional) PostgreSQL port
  • PGUSER - (Optional) PostgreSQL user name
  • PGPASSWORD - (Optional) PostgreSQL user password

Your PostgreSQL server version must match pg_dump: version 12.x max

Usage

docker run --rm -t --name="backup1" -v my-data-volume:/data:ro \
           -e DB_USER="toto" \
           -e DB_HOST="mariadb" \
           -e DB_PASS="123abc" \
           -e DB_NAME="foo_db" \
           -e FTP_USER="username" \
           -e FTP_PASS="butterfly" \
           -e FTP_HOST="foobar.com" \
           -e FTP_PORT="21" \
           -e REMOTE_PATH="/backups/my-site" \
           -e COMPRESS="0" \
           -e CHUNK_SIZE="128" \
           --link my-mariadb:mariadb ambroisemaupate/ftp-backup

Do not compress when not necessary

If your folder to backup contains mostly JPG, PNG, WebP images (or any already compressed data), do not compress archive as it will use lots of CPU for a little less space: COMPRESS=0.

Split for weak FTP servers

Splitting archives before uploading can enhance stability and allow transfers to resume if you disconnected from FTP server : CHUNK_SIZE=128. split command will generate *.partaa to *.partzz files in order to keep them in order and easily allow cat to join them back.

If you need to recover archive from a split file, use:

cat 20210727_0904_files/20210727_0904_files.tar.part* >> 20210727_0904_files.tar

How to automatize backups

I use a simple bash script to automatize docker backups along a ftp-credential.sh script to store FTP access data once for all.

I variabilized docker name to generate automatically container names. I use the following naming policy:

  • Main worker container is called NAME
  • Database worker container is called after NAME + _DB suffix
  • Main data container is called after NAME + _DATA suffix
  • Database data container is called after NAME + _DBDATA suffix
  • Main back-up container is called after NAME + _BCK suffix
#!/usr/bin/env bash
# Author: Ambroise Maupate
# File: /root/scripts/bck-my-docker-container.sh

. `dirname $0`/ftp-credentials.sh || {
    echo "`dirname $0`/ftp-credentials.sh";
    echo 'Impossible to import your ftp config.';
    exit 1;
}

NAME="my-docker-container"

docker run --rm -t --name="${NAME}_BCK" -v ${NAME}_DATA:/data:ro \
           -e FTP_USER="${FTP_USER}"\
           -e FTP_PASS="${FTP_PASS}"\
           -e FTP_HOST="${FTP_HOST}"\
           -e FTP_PORT="${FTP_PORT}"\
           -e DB_USER="my_docker_container_dbuser"\
           -e DB_HOST="mariadb"\
           -e DB_PASS="my_docker_container_dbpass"\
           -e DB_NAME="my_docker_container_db"\
           -e REMOTE_PATH="/docker-bck/${NAME}"\
           --link ${NAME}_DB:mariadb ambroisemaupate/ftp-backup

Here is the central ftp access data script.

#!/usr/bin/env bash
# Author: Ambroise Maupate
# File: /root/scripts/ftp-credentials.sh

FTP_USER="myFtpUser"
FTP_PASS="myFtpPassword"
FTP_HOST="myFtp.host.com"
FTP_PORT="21"

Then all you need is to setup this in your root’s crontab:

0 2 * * * /bin/bash ~/scripts/bck-my-docker-container.sh >> bcklog-my-docker-container.log

And do not forget to set executable flag on your scripts:

chmod u+x ~/scripts/ftp-credentials.sh
chmod u+x ~/scripts/bck-my-docker-container.sh

Use SFTP protocol

Add FTP_PORT and FTP_PROTO environment vars.

docker run --rm -t --name="backup1" -v my-data-volume:/data:ro \
           -e DB_USER="toto" \
           -e DB_HOST="mariadb" \
           -e DB_PASS="123abc" \
           -e DB_NAME="foo_db" \
           -e FTP_USER="username" \
           -e FTP_PASS="butterfly" \
           -e FTP_HOST="foobar.com" \
           -e FTP_PORT="22" \
           -e FTP_PROTO="sftp" \
           -e CHUNK_SIZE="512" \
           -e REMOTE_PATH="/home/username/backups/my-site" \
           --link my-mariadb:mariadb ambroisemaupate/ftp-backup

Example usage in docker-compose

version: "3"
services:
  db:
    image: mysql:5.7
    volumes:
      - DBDATA:/var/lib/mysql
    environment:
      MYSQL_DATABASE: test
      MYSQL_USER: test
      MYSQL_PASSWORD: test
      MYSQL_RANDOM_ROOT_PASSWORD: "yes"
    restart: always

  backup:
    image: ambroisemaupate/ftp-backup
    depends_on:
      - db
    environment:
      LOCAL_PATH: /var/www/html
      DB_USER: test
      DB_HOST: db
      DB_PASS: test
      DB_NAME: test
      FTP_PROTO: ftp
      FTP_PORT: 21
      FTP_HOST: ftp.server.test
      FTP_USER: test
      FTP_PASS: test
      CHUNK_SIZE: 512
      REMOTE_PATH: /home/test/backups
    volumes:
      - public_files:/var/www/html/web/files:ro

volumes:
  public_files:
  DBDATA:

Rolling backups

You can add as much services as you want to create rolling backups: daily, weekly, monthly:

  # DAILY
  backup_daily:
    image: ambroisemaupate/ftp-backup
    depends_on:
      - db
    environment:
      LOCAL_PATH: /var/www/html
      DB_USER: test
      DB_HOST: db
      DB_PASS: test
      DB_NAME: test
      FTP_PROTO: ftp
      FTP_PORT: 21
      FTP_HOST: ftp.server.test
      FTP_USER: test
      FTP_PASS: test
      REMOTE_PATH: /home/test/backups/daily
    volumes:
      - public_files:/var/www/html/web/files:ro

  backup_cleanup_daily:
    image: ambroisemaupate/ftp-cleanup
    environment:
      FTP_PROTO: ftp
      FTP_PORT: 21
      FTP_HOST: ftp.server.test
      FTP_USER: test
      FTP_PASS: test
      STORE_DAYS: 7
      FTP_PATH: /home/test/backups/daily
  
  # WEEKLY
  backup_weekly:
    image: ambroisemaupate/ftp-backup
    depends_on:
      - db
    environment:
      LOCAL_PATH: /var/www/html
      DB_USER: test
      DB_HOST: db
      DB_PASS: test
      DB_NAME: test
      FTP_PROTO: ftp
      FTP_PORT: 21
      FTP_HOST: ftp.server.test
      FTP_USER: test
      FTP_PASS: test
      REMOTE_PATH: /home/test/backups/weekly
    volumes:
      - public_files:/var/www/html/web/files:ro

  backup_cleanup_weekly:
    image: ambroisemaupate/ftp-cleanup
    environment:
      FTP_PROTO: ftp
      FTP_PORT: 21
      FTP_HOST: ftp.server.test
      FTP_USER: test
      FTP_PASS: test
      STORE_DAYS: 30
      FTP_PATH: /home/test/backups/weekly
  
  # MONTHLY
  backup_monthly:
    image: ambroisemaupate/ftp-backup
    depends_on:
      - db
    environment:
      LOCAL_PATH: /var/www/html
      DB_USER: test
      DB_HOST: db
      DB_PASS: test
      DB_NAME: test
      FTP_PROTO: ftp
      FTP_PORT: 21
      FTP_HOST: ftp.server.test
      FTP_USER: test
      FTP_PASS: test
      REMOTE_PATH: /home/test/backups/monthly
    volumes:
      - public_files:/var/www/html/web/files:ro

  backup_cleanup_monthly:
    image: ambroisemaupate/ftp-cleanup
    environment:
      FTP_PROTO: ftp
      FTP_PORT: 21
      FTP_HOST: ftp.server.test
      FTP_USER: test
      FTP_PASS: test
      STORE_DAYS: 366
      FTP_PATH: /home/test/backups/monthly

then launch them once a day, once a week, once a month from your crontab:

# Rolling backups (do not use same hour of night to save CPU)
# Daily
00 2 * * * cd /mywebsite.com && /usr/local/bin/docker-compose run --rm --no-deps backup_daily
20 2 * * * cd /mywebsite.com && /usr/local/bin/docker-compose run --rm --no-deps backup_cleanup_daily
# Weekly
00 3 * * 1 cd /mywebsite.com && /usr/local/bin/docker-compose run --rm --no-deps backup_weekly
20 3 * * 1 cd /mywebsite.com && /usr/local/bin/docker-compose run --rm --no-deps backup_cleanup_weekly
# Monthly
00 4 1 * * cd /mywebsite.com && /usr/local/bin/docker-compose run --rm --no-deps backup_monthly
20 4 1 * * cd /mywebsite.com && /usr/local/bin/docker-compose run --rm --no-deps backup_cleanup_monthly