sup3

A standalone S3 tool

Warning

This is beta

Testing is not near exhaustive, bug reports and pull requests welcome.

Usage goal

• CI and other minimal environments where pulling in python or building dependencies might be undesirable
• Eventually, as a consistent, reliable, fast and fun S3 tool for interactive use

Design

• Act as much as possible like existing unix commands (e.g. cp, ls, cat)
• Progress reporting by default where it makes sense (e.g. cp, down, up)

Features

• Cross-platform
• Uses authorisation support from the Rust AWS SDK
• Streaming async transfers (files not read into memory)
• Upload resume on remote errors (provided by the SDK)
• No startup delay
• No runtime non-platform dependencies (e.g. libc, libm, libgcc_s)
• Concurrent transfers
• Optional progress reporting
• Recursive upload support
• Recursive download support
• Remote globbing (e.g. sup3 ls s3://bucket/media/**/highres*.png .)
• Binary size reduction
• Upload arguments, e.g. ACLs
• Config file support
• List only files or only directories
• List paging
• Server to server copy
• Custom endpoints for other S3-compatible hosts (--endpoint)
• A sync operation (as part of copy/up/down or separate)
• Shell completion

Commands

• Upload (upload|down) (local files to S3 remote)
• Download (download|down) (S3 URIs to local file/directory)
• Remove (rm) (S3 URIs)
• List (ls) (1..N S3 URIs)
• List Buckets (list-buckets|lb)
• Copy (cp)
• Cat (S3 URIs)
• Make Bucket (mb) (S3 URIs)

Speed

Meaurements performed on an i7-7700HQ, with an approximately 50ms RTT to the S3 server.

Download - 20MiB file

Command	Mean [s]	Min [s]	Max [s]	Relative	CPU User [s]	CPU System [s]	CPU Relative
sup3 cp s3://bucket/file dir/	1.306 ± 0.047	1.249	1.416	1.00	0.095	0.153	100%
aws s3 cp s3://bucket/file dir/	1.855 ± 0.106	1.724	2.044	142% ± 10%	0.565	0.122	277%

Download - 224byte file

Command	Mean [s]	Min [s]	Max [s]	Relative	CPU User [s]	CPU System [s]	CPU Relative
sup3 cp s3://bucket/file dir/	0.246 ± 0.006	0.234	0.255	100%	0.004	0.007	100%
aws s3 cp s3://bucket/file dir/	0.910 ± 0.013	0.892	0.937	370% ± 11%	0.394	0.050	4036%

ls - 63,186 files from bucket by substring

Command	Mean [s]	Min [s]	Max [s]	Relative	CPU User [s]	CPU System [s]	CPU Relative
sup3 ls --substring s3://bucket/substring >/dev/null	10.053 ± 0.694	9.253	10.487	100%	0.485	0.141	100%
aws s3 ls s3://bucket/substring >/dev/null	22.050 ± 3.809	19.643	26.441	219% ± 41%	11.772	0.143	1900%