sftp-client-s3

SFTP client to exchange files between SFTP server and S3 bucket.

Available Commands

Help

$ docker run brainstation/sftp-client-s3 --help
sftp-client <command>

Commands:
  sftp-client server-to-s3  Download files from SFTP server and put them in S3
                            bucket
  sftp-client s3-to-server  Get a file from S3 and upload that on the SFTP
                            server

Options:
  --version  Show version number                                       [boolean]
  --help     Show help                                                 [boolean]

Server to S3

$ docker run brainstation/sftp-client-s3 server-to-s3 --help
sftp-client server-to-s3

Download files from SFTP server and put them in S3 bucket

Options:
      --version                        Show version number             [boolean]
      --help                           Show help                       [boolean]
  -h, --host, --sftp-host              SFTP host IP address or URL
                                                             [string] [required]
  -p, --port, --sftp-port              SFTP host port number
                                                          [number] [default: 22]
  -u, --user, --sftp-user, --username  SFTP username         [string] [required]
      --private-key-s3-uri             S3 URI for the private key to
                                       authenticate SFTP session
                                                             [string] [required]
  -l, --location, --remote-location    Path to the file location in SFTP server
                                                             [string] [required]
  -r, --remove, --rm, --delete         Delete remote files after successfull
                                       upload to S3   [boolean] [default: false]
  -f, --filename, --filename-pattern   A string that will be used as a regular
                                       expression to find a list of files in the
                                       server location                  [string]
  -b, --bucket, --bucket-name          S3 bucket name        [string] [required]
      --key-prefix                     S3 key prefix, original filename from the
                                       server will get appended
                                                             [string] [required]
      --gunzip, --uncompress           Uncompress file content if the file has a
                                       `.gz` extension[boolean] [default: false]

S3 to Server

$ docker run brainstation/sftp-client-s3 s3-to-server --help
sftp-client s3-to-server

Get a file from S3 and upload that on the SFTP server

Options:
      --version                        Show version number             [boolean]
      --help                           Show help                       [boolean]
  -h, --host, --sftp-host              SFTP host IP address or URL
                                                             [string] [required]
  -p, --port, --sftp-port              SFTP host port number
                                                          [number] [default: 22]
  -u, --user, --sftp-user, --username  SFTP username         [string] [required]
      --private-key-s3-uri             S3 URI for the private key to
                                       authenticate SFTP session
                                                             [string] [required]
  -l, --location, --remote-location    Path to the file location in SFTP server
                                                             [string] [required]
  -b, --bucket, --bucket-name          S3 bucket name        [string] [required]
      --s3-key                         S3 key for the file to upload
                                                             [string] [required]
  -r, --remove, --rm, --delete         Delete remote file (if exists) before
                                       upload from S3 [boolean] [default: false]
      --gzip, --compress               Compress file content if the filename
                                       does not have a `.gz` extension
                                                      [boolean] [default: false]

Environment variables

Name	Description
SFTP_HOST	SFTP host IP address or URL
SFTP_PORT	SFTP host port number
SFTP_USER	SFTP username
PRIVATE_KEY_S3_URI	S3 URI for the private key to authenticate SFTP session
REMOTE_LOCATION	Path to the file location in SFTP server
FILENAME	server-to-s3: A string that will be used as a regular expression to find a list of files in the server location
BUCKET_NAME	S3 bucket name
S3_KEY	S3 key for the file to upload
KEY_PREFIX	S3 key prefix, original filename from the server will get appended
DELETE_REMOTE	s3-to-server: Delete remote file (if exists) before upload from S3
DELETE_REMOTE	server-to-s3: Delete remote files after successfull upload to S3
COMPRESS	Compress file content if the filename does not have a .gz extension
UNCOMPRESS	Uncompress file content if the file has a .gz extension

When a combination of the above environment variables are present, corresponding command options become optional. But if you provide value for a command option, that has higher priority over environemnt variables.

Examples

SFTP server to s3

$ docker run brainstation/sftp-client-s3 server-to-s3 \
    -h example.com \
    -p 22 \
    -u test_user \
    --private-key-s3-uri s3://bucket-name/foo/bar/id_rsa \
    -l /outbox \
    -b my-bucket \
    --key-prefix my-project/section-1/year=2021/month=02/day=01/

S3 to SFTP server

$ docker run brainstation/sftp-client-s3 s3-to-server \
    -h example.com \
    -p 22 \
    -u test_user \
    --private-key-s3-uri s3://bucket-name/foo/bar/id_rsa \
    -l /inbox \
    -b my-bucket \
    -s3-key my-project/file-to-upload.txt

Parameters

Name	server-to-s3	s3-to-server
-h, --host, --sftp-host	✔️	✔️
-p, --port, --sftp-port	✔️	✔️
-u, --user, --sftp-user, --username	✔️	✔️
--private-key-s3-uri	✔️	✔️
-l, --location, --remote-location	✔️	✔️
-r, --remove, --rm, --delete	✔️	✔️
-f, --filename, --filename-pattern	✔️	✖️
-b, --bucket, --bucket-name	✔️	✔️
--s3-key	✖️	✔️
--key-prefix	✔️	✖️
--gunzip, --uncompress	✔️	✖️
--gzip, --compress	✖️	✔️