s3sync-service

The s3sync-service is a lightweight tool designed with k8s in mind and aimed to syncing data to S3 storage service for multiple sites (path + bucket combination). Each site can have its own set of credentials and be in different region, which makes the s3sync-service really flexible.

Check out the quickstart note or head over to the documentation where you will find more examples on running the application locally or on k8s.

Quickstart

1. Create directory with configuration file, eg. - /path/to/config/config.yml.
2. Run docker container with providing AWS credentials via environment variables (IAM role should also do the trick), alternatively credentials could be provided in the config file, mount directory containing the config file and all of the backup directories listed in the config file:

docker run --rm -ti \
  -e "AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE" \
  -e "AWS_SECRET_ACCESS_KEY=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY" \
  -e "AWS_DEFAULT_REGION=us-east-1" \
  -v "/path/to/config:/opt/s3sync-service" \
  -v "/backup/path:/backup" \
  ghcr.io/mazay/s3sync-service:master \
  ./s3sync-service -config /opt/s3sync-service/config.yml

Docker Compose can also be used:

version: '3.3'
services:
  s3sync-service:
    environment:
      - AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
      - AWS_SECRET_ACCESS_KEY=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
      - AWS_DEFAULT_REGION=us-east-1
    volumes:
      - '/backup/path:/backup'
      - '/path/to/config.yml:/app/config.yml'
    image: ghcr.io/mazay/s3sync-service:master

Configuration

Example configuration, check this for more details:

upload_workers: 10
sites:
- local_path: /local/path1
  bucket: backup-bucket-path1
  bucket_region: us-east-1
  storage_class: STANDARD_IA
  access_key: AKIAI44QH8DHBEXAMPLE
  secret_access_key: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
  exclusions:
    - .[Dd][Ss]_[Ss]tore
    - .[Aa]pple[Dd]ouble
- local_path: /local/path2
  bucket: backup-bucket-path2
  bucket_path: path2
  exclusions:
    - "[Tt]humbs.db"
- local_path: /local/path3
  bucket: backup-bucket-path3
  bucket_path: path3
  exclusions:
    - "[Tt]humbs.db"

Contributing

If you feel like contributing to the project - there are various ways of doing so.

Support

You can buy me a coffee if you feel like supporting my motivation in working on this project. :)