chmods 777 for podman

[fabricebrito]

Rationale: Use podman as a replacement to docker

This PR adds a chmod 777 to the local folders mounted on the container enabling the replacement of docker with podman (as a symbolic link (e.g. /usr/bin/docker: symbolic link to /usr/bin/podman)

This issue is linked to:

• Document podman compatability #1344
• https://cwl.discourse.group/t/cwl-runner-and-podman/210/1

Tests have been successfully ran according to the contributing guidelines

[mr-c]

Thank you @fabricebrito ! As per #1344, do you think we should autodetect & use podman if there is no docker on the system? Or should there be a manual command line option?

[codecov]

Codecov Report

Merging #1516 (5e27d91) into main (ffb7513) will decrease coverage by 9.82%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1516      +/-   ##
==========================================
- Coverage   65.69%   55.86%   -9.83%     
==========================================
  Files          89       45      -44     
  Lines       15938     7998    -7940     
  Branches     4045     2028    -2017     
==========================================
- Hits        10470     4468    -6002     
+ Misses       4337     2985    -1352     
+ Partials     1131      545     -586     

Impacted Files	Coverage Δ
cwltool/resolver.py
cwltool/task_queue.py
cwltool/executors.py
cwltool/checker.py
cwltool/validate_js.py
cwltool/stdfsaccess.py
cwltool/utils.py
cwltool/argparser.py
cwltool/singularity.py
cwltool/provenance_profile.py
... and 35 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffb7513...5e27d91. Read the comment docs.

[fabricebrito]

Thank you @fabricebrito ! As per #1344, do you think we should autodetect & use podman if there is no docker on the system? Or should there be a manual command line option?

@mr-c , my two cents: udocker and singularity have a command line option, we could go for the same approach for podman

[cwl-bot]

This pull request has been mentioned on Common Workflow Language Discourse. There might be relevant details there:

https://cwl.discourse.group/t/cwl-runner-and-podman/210/8

[mr-c]

Does it have to be world writeable?

https://github.com/common-workflow-language/cwltool/pull/1516/checks?check_run_id=3568589880#step:9:24

[fabricebrito]

yes, this what we had to do.

While running, I get:

(base) [jovyan@jupyter-fbrito-dev workspace]$ ll -a /tmp/eatw95id
total 65932
drwxrwxrwx 2 jovyan users      4096 Sep 14 10:41 .
drwxrwxrwt 1 root   root       4096 Sep 14 10:39 ..
-rw-r--r-- 1 101000 100099 67505506 Sep 14 10:41 S2B_53HPA_20210723_0_L2A.tif

while running:

INFO [job band_math.cwl] /tmp/eatw95id$ docker \
    run \
    -i \
    --mount=type=bind,source=/tmp/eatw95id,target=/BixLsD \
    --mount=type=bind,source=/tmp/bnll6nbs,target=/tmp \
    --workdir=/BixLsD \
    --read-only=true \
    --user=1001:100 \
    --rm \
    --cidfile=/tmp/catzbo8t/20210914103914-031344.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/BixLsD \
    docker.io/terradue/otb-7.2.0 \
    otbcli_BandMathX \
    -out \
    S2B_53HPA_20210723_0_L2A.tif \
    -exp \
    '(im3b1 == 8 or im3b1 == 9 or im3b1 == 0 or im3b1 == 1 or im3b1 == 2 or im3b1 == 10 or im3b1 == 11) ? -2 : (im1b1 - im2b1) / (im1b1 + im2b1)' \
    -il \
    /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B8A.tif \
    /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B12.tif \
    /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/SCL.tif

with:

(base) [jovyan@jupyter-fbrito-dev workspace]$ id
uid=1001(jovyan) gid=100(users) groups=100(users)

So anything below 777, fails with a permission denied

[mr-c]

Thanks for this @fabricebrito ! I found another way without having to adjust permissions over in #1519

[fabricebrito]

Wow @mr-c ! That's great!

It works like a charm:

(base) [jovyan@jupyter-fbrito-dev workspace]$ cwltool --podman  band_math.cwl band_math.yml 
INFO /opt/anaconda/bin/cwltool 3.1.20210928171851
INFO Resolved 'band_math.cwl' to 'file:///workspace/band_math.cwl'
INFO [job band_math.cwl] /tmp/tu7n8uiq$ podman \
    run \
    -i \
    --userns=keep-id \
    --mount=type=bind,source=/tmp/tu7n8uiq,target=/CUcOTS \
    --mount=type=bind,source=/tmp/u59t_mm4,target=/tmp \
    --workdir=/CUcOTS \
    --read-only=true \
    --user=1001:100 \
    --rm \
    --cidfile=/tmp/upptxits/20210929110408-353063.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/CUcOTS \
    docker.io/terradue/otb-7.2.0 \
    otbcli_BandMathX \
    -out \
    S2B_53HPA_20210723_0_L2A.tif \
    -exp \
    '(im3b1 == 8 or im3b1 == 9 or im3b1 == 0 or im3b1 == 1 or im3b1 == 2 or im3b1 == 10 or im3b1 == 11) ? -2 : (im1b1 - im2b1) / (im1b1 + im2b1)' \
    -il \
    /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B8A.tif \
    /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/B12.tif \
    /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/53/H/PA/2021/7/S2B_53HPA_20210723_0_L2A/SCL.tif
2021-09-29 11:05:36 (INFO) BandMathX: Default RAM limit for OTB is 256 MB
2021-09-29 11:05:36 (INFO) BandMathX: GDAL maximum cache size is 204 MB
2021-09-29 11:05:36 (INFO) BandMathX: OTB will use at most 4 threads
2021-09-29 11:05:36 (INFO) BandMathX: Image #1 has 1 components
2021-09-29 11:05:36 (INFO) BandMathX: Image #2 has 1 components
2021-09-29 11:05:36 (INFO) BandMathX: Image #3 has 1 components
2021-09-29 11:05:36 (INFO) BandMathX: Using expression: (im3b1 == 8 or im3b1 == 9 or im3b1 == 0 or im3b1 == 1 or im3b1 == 2 or im3b1 == 10 or im3b1 == 11) ? -2 : (im1b1 - im2b1) / (im1b1 + im2b1)
2021-09-29 11:05:36 (INFO): Estimated memory for full processing: 574.839MB (avail.: 256 MB), optimal image partitioning: 3 blocks
2021-09-29 11:05:36 (INFO): File S2B_53HPA_20210723_0_L2A.tif will be written in 4 blocks of 3072x3072 pixels
Writing S2B_53HPA_20210723_0_L2A.tif...: 100% [**************************************************] (1m 28s)
INFO [job band_math.cwl] Max memory used: 0MiB
INFO [job band_math.cwl] completed success
{
    "nbr": {
        "location": "file:///workspace/S2B_53HPA_20210723_0_L2A.tif",
        "basename": "S2B_53HPA_20210723_0_L2A.tif",
        "class": "File",
        "checksum": "sha1$6f1b9a5230e53d9bf30ee1c1b09b8aa2e9d45d6b",
        "size": 120604786,
        "path": "/workspace/S2B_53HPA_20210723_0_L2A.tif"
    }
}
INFO Final process status is success