S3 arguments

[kimpham54]

new arguments for you

./bin/docker-run.sh s3-file -h
./bin/docker-run.sh s3 -h

Testing steps:

1. install as usual,

python3 -m venv myenv
source myenv/bin/activate
export PYTHONPATH="${PYTHONPATH}:src"
pip install -r requirements.txt
python src/jp2_remediator/main.py -h

• move test files in repo
• mkdir logs
• uncomment in .bin/docker-run.sh the option to pass aws credentials if you want to use aws
• create aws credentials and place in .env file in bin directory:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_SESSION_TOKEN=

5. ./bin/docker-build.sh

6. docker entrypoint/docker exec

container has to be alive, e.g. docker run --rm --mount type=bind,source=${PWD},target=/data -it --entrypoint /bin/bash artifactory.huit.harvard.edu/lts/jp2_remediator $@

./bin/docker-run.sh file /data/[your_test_images_folder]/[testfile.jp2]
./bin/docker-run.sh directory /data/[your_test_images_folder]

INPUT BUCKET/PREFIX > OUTPUT BUCKET/PREFIX
only input bucket name is required, other arguments are optional and would refer to the default input bucket as output

./bin/docker-run.sh s3 \
[input-bucket-name] \
--input-prefix [input/paths] \
--output-bucket [output-bucket-name] \
--output-prefix [output/paths]

INPUT FILE > OUTPUT FILE

./bin/docker-run.sh s3-file \
lts-jp2-remediation-dev \
--input-key [input-path/to/file.jp2] \
--output-bucket lts-jp2-remediation-dev \
--output-key [output-path/to/file.jp2] 

INPUT FILE > OUTPUT PREFIX
specifying output prefix instead of output key, appends _modified_yyyymmdd.jp2 to single file:

./bin/docker-run.sh s3-file \
lts-jp2-remediation-dev \
--input-key [input-path/to/file.jp2] \
--output-bucket [output-bucket-name] \
--output-prefix [output/paths]

• if the "folder"/path doesn't exist in the specified --output-prefix for both s3 and s3-file, it will get created
• a custom filename can be used in --output-key, or it defaults to appending _modified_yyyymmdd.jp2

[awoods]

Please replace all print statements with self.logger.info(...)

[awoods]

Change from:

"input_bucket", help="Name of the AWS S3 bucket containing the JP2 file"

to:

"--input_bucket", help="Name of the AWS S3 bucket containing the JP2 file", required=True

[kimpham54]

should this still be required after adding the -- flag? and it will be --input-bucket instead of --input_bucket

[awoods]

Make required

[awoods]

We can remove this argument

[kimpham54]

do you want to leave it as an option or remove completely? if it is optional but not used, it defaults to the input bucket

[kimpham54]

keep this one, after discussion with andrew

[awoods]

Make required

[kimpham54]

right now it is optional, is there a situation where you wouldn't want to use the --output-prefix option, such as if you wanted to just put it in the same input directory, or same bucket?

[kimpham54]

• take out s3 (bucket only) option, not needed