explicit support for podman #1519

mr-c · 2021-09-14T12:34:29Z

Builds off of #1516

@fabricebrito I'm getting some failures from running the CWL conformance tests and podman; those will need to be fixed before we can merge

codecov · 2021-09-14T12:45:07Z

Codecov Report

Merging #1519 (cec2fe5) into main (c014c36) will increase coverage by 0.06%.
The diff coverage is 83.13%.

@@            Coverage Diff             @@
##             main    #1519      +/-   ##
==========================================
+ Coverage   65.65%   65.71%   +0.06%     
==========================================
  Files          89       91       +2     
  Lines       15982    16072      +90     
  Branches     4055     4075      +20     
==========================================
+ Hits        10493    10562      +69     
- Misses       4358     4367       +9     
- Partials     1131     1143      +12

Impacted Files	Coverage Δ
cwltool/docker.py	`68.69% <0.00%> (-0.61%)`	⬇️
cwltool/expression.py	`89.71% <ø> (ø)`
cwltool/validate_js.py	`84.29% <ø> (ø)`
cwltool/singularity.py	`59.33% <50.00%> (-0.77%)`	⬇️
cwltool/workflow_job.py	`55.08% <60.00%> (-1.07%)`	⬇️
cwltool/factory.py	`87.23% <71.42%> (-1.14%)`	⬇️
cwltool/singularity_utils.py	`85.71% <85.71%> (ø)`
cwltool/sandboxjs.py	`74.59% <86.95%> (+8.91%)`	⬆️
cwltool/loghandler.py	`90.47% <87.50%> (-9.53%)`	⬇️
cwltool/argparser.py	`89.63% <100.00%> (+0.04%)`	⬆️
... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c014c36...cec2fe5. Read the comment docs.

mr-c · 2021-09-14T12:52:13Z

Test 56 failed: /home/michael/cwltool/env3.9/bin/cwltool --podman --outdir=/tmp/tmp9gm2vz34 --quiet 'tests/search.cwl#main' tests/search-job.json
Test InitialWorkDirRequirement linking input files and capturing secondaryFiles on input and output. Also tests the use of a variety of parameter references and expressions in the secondaryFiles field.

~/cwl-v1.2$ cwltool --debug --podman 'tests/search.cwl#main' tests/search-job.json
[…]
INFO [workflow ] completed success
DEBUG [workflow ] outputs {
    "indexedfile": {
        "location": "/home/michael/cwl-v1.2/tests/whale.txt",
        "basename": "input.txt",
        "nameroot": "input",
        "nameext": ".txt",
        "class": "File",
        "checksum": "sha1$327fc7aedf4f6b69a42a7c8b808dc5a7aff61376",
        "size": 1111,
        "secondaryFiles": [
            {
                "basename": "input.txt.idx1",
                "location": "file:///tmp/2g8czb4d/input.txt.idx1",
                "class": "File",
                "nameroot": "input.txt",
                "nameext": ".idx1",
                "checksum": "sha1$1f6fe811644355974cdd06d9eb695d6e859f3b44",
                "size": 1500,
                "http://commonwl.org/cwltool#generation": 0
            },
            {
                "basename": "input.idx2",
                "location": "file:///tmp/2g8czb4d/input.idx2",
                "class": "File",
                "nameroot": "input",
                "nameext": ".idx2",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "http://commonwl.org/cwltool#generation": 0
            },
            {
                "basename": "input.txt.idx3",
                "location": "file:///tmp/2g8czb4d/input.txt.idx3",
                "class": "File",
                "nameroot": "input.txt",
                "nameext": ".idx3",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "http://commonwl.org/cwltool#generation": 0
            },
            {
                "basename": "input.txt.idx4",
                "location": "file:///tmp/2g8czb4d/input.txt.idx4",
                "class": "File",
                "nameroot": "input.txt",
                "nameext": ".idx4",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "http://commonwl.org/cwltool#generation": 0
            },
            {
                "class": "File",
                "basename": "input.txt.idx5",
                "location": "file:///tmp/2g8czb4d/input.txt.idx5",
                "nameroot": "input.txt",
                "nameext": ".idx5",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "http://commonwl.org/cwltool#generation": 0
            },
            {
                "basename": "input.idx6.txt",
                "location": "file:///tmp/2g8czb4d/input.idx6.txt",
                "class": "File",
                "nameroot": "input.idx6",
                "nameext": ".txt",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "http://commonwl.org/cwltool#generation": 0
            },
            {
                "basename": "input.txt.idx7",
                "location": "file:///tmp/2g8czb4d/input.txt.idx7",
                "class": "File",
                "nameroot": "input.txt",
                "nameext": ".idx7",
                "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
                "size": 0,
                "http://commonwl.org/cwltool#generation": 0
            },
            {
                "class": "File",
                "location": "/home/michael/cwl-v1.2/tests/hello.txt",
                "size": 13,
                "basename": "hello.txt",
                "nameroot": "hello",
                "nameext": ".txt",
                "checksum": "sha1$47a013e660d408619d894b20806b1d5086aab03b",
                "http://commonwl.org/cwltool#generation": 0
            },
            {
                "basename": "input.txt_idx8",
                "location": "file:///tmp/2g8czb4d/input.txt_idx8",
                "class": "Directory"
            }
        ],
        "http://commonwl.org/cwltool#generation": 0
    },
    "outfile": {
        "location": "file:///tmp/6w04eivx/result.txt",
        "basename": "result.txt",
        "nameroot": "result",
        "nameext": ".txt",
        "class": "File",
        "checksum": "sha1$e2dc9daaef945ac15f01c238ed2f1660f60909a0",
        "size": 142,
        "http://commonwl.org/cwltool#generation": 0
    }
}
DEBUG [job search] Removing input staging directory /tmp/cz8vmupj
DEBUG [job search] Removing temporary directory /tmp/ylbmj2te
DEBUG Copying /home/michael/cwl-v1.2/tests/whale.txt to /home/michael/cwl-v1.2/input.txt
DEBUG Moving /tmp/2g8czb4d/input.txt.idx1 to /home/michael/cwl-v1.2/input.txt.idx1
DEBUG Moving /tmp/2g8czb4d/input.idx2 to /home/michael/cwl-v1.2/input.idx2
DEBUG Moving /tmp/2g8czb4d/input.txt.idx3 to /home/michael/cwl-v1.2/input.txt.idx3
DEBUG Moving /tmp/2g8czb4d/input.txt.idx4 to /home/michael/cwl-v1.2/input.txt.idx4
DEBUG Moving /tmp/2g8czb4d/input.txt.idx5 to /home/michael/cwl-v1.2/input.txt.idx5
DEBUG Moving /tmp/2g8czb4d/input.idx6.txt to /home/michael/cwl-v1.2/input.idx6.txt
DEBUG Moving /tmp/2g8czb4d/input.txt.idx7 to /home/michael/cwl-v1.2/input.txt.idx7
DEBUG Copying /home/michael/cwl-v1.2/tests/hello.txt to /home/michael/cwl-v1.2/hello.txt
DEBUG Moving /tmp/2g8czb4d/input.txt_idx8 to /home/michael/cwl-v1.2/input.txt_idx8
DEBUG Moving /tmp/2g8czb4d/input.txt_idx8/index to /home/michael/cwl-v1.2/input.txt_idx8/index
ERROR Unhandled error:
  [Errno 13] Permission denied: '/tmp/2g8czb4d/input.txt_idx8/index'
Traceback (most recent call last):
  File "/usr/lib/python3.9/shutil.py", line 814, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/tmp/2g8czb4d/input.txt_idx8/index' -> '/home/michael/cwl-v1.2/input.txt_idx8/index'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/michael/cwltool/cwltool/main.py", line 1256, in main
    (out, status) = real_executor(
  File "/home/michael/cwltool/cwltool/executors.py", line 60, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
  File "/home/michael/cwltool/cwltool/executors.py", line 150, in execute
    self.final_output[0] = relocateOutputs(
  File "/home/michael/cwltool/cwltool/process.py", line 401, in relocateOutputs
    stage_files(pm, stage_func=_relocate, symlink=False, fix_conflicts=True)
  File "/home/michael/cwltool/cwltool/process.py", line 297, in stage_files
    stage_func(entry.resolved, entry.target)
  File "/home/michael/cwltool/cwltool/process.py", line 372, in _relocate
    _relocate(dir_entry.path, fs_access.join(dst, dir_entry.name))
  File "/home/michael/cwltool/cwltool/process.py", line 374, in _relocate
    shutil.move(src, dst)
  File "/usr/lib/python3.9/shutil.py", line 835, in move
    os.unlink(src)
PermissionError: [Errno 13] Permission denied: '/tmp/2g8czb4d/input.txt_idx8/index'

Test 90 failed: /home/michael/cwltool/env3.9/bin/cwltool --podman --outdir=/tmp/tmpmpfg2dqp --quiet tests/stagefile.cwl tests/stagefile-job.yml
Test writable staged files.

~/cwl-v1.2$ cwltool --debug --podman tests/stagefile.cwl tests/stagefile-job.yml
INFO /home/michael/cwltool/env3.9/bin/cwltool 3.1.20210914121718
INFO Resolved 'tests/stagefile.cwl' to 'file:///home/michael/cwl-v1.2/tests/stagefile.cwl'
DEBUG [job stagefile.cwl] initializing from file:///home/michael/cwl-v1.2/tests/stagefile.cwl
DEBUG [job stagefile.cwl] {
    "infile": {
        "class": "File",
        "location": "file:///home/michael/cwl-v1.2/tests/whale.txt",
        "size": 1111,
        "basename": "whale.txt",
        "nameroot": "whale",
        "nameext": ".txt"
    }
}
DEBUG [job stagefile.cwl] path mappings is {
    "file:///home/michael/cwl-v1.2/tests/whale.txt": [
        "/home/michael/cwl-v1.2/tests/whale.txt",
        "/cXjrLS/bob.txt",
        "WritableFile",
        false
    ]
}
DEBUG [job stagefile.cwl] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "python"
    },
    {
        "position": [
            0,
            0
        ],
        "datum": "-c"
    },
    {
        "position": [
            0,
            1
        ],
        "datum": "f = open(\"bob.txt\", \"r+\")\nf.seek(8)\nf.write(\"Bob.    \")\nf.close()\n"
    }
]
DEBUG [job stagefile.cwl] initial work dir {
    "file:///home/michael/cwl-v1.2/tests/whale.txt": [
        "/home/michael/cwl-v1.2/tests/whale.txt",
        "/cXjrLS/bob.txt",
        "WritableFile",
        true
    ]
}
INFO [job stagefile.cwl] /tmp/el2sbb6z$ podman \
    run \
    -i \
    --mount=type=bind,source=/tmp/el2sbb6z,target=/cXjrLS \
    --mount=type=bind,source=/tmp/psbvh6q6,target=/tmp \
    --workdir=/cXjrLS \
    --read-only=true \
    --net=none \
    --user=1000:1000 \
    --rm \
    --cidfile=/tmp/pz888pso/20210914143946-293588.cid \
    --env=TMPDIR=/tmp \
    --env=HOME=/cXjrLS \
    python:2-slim \
    python \
    -c \
    'f = open("bob.txt", "r+")
f.seek(8)
f.write("Bob.    ")
f.close()
'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
IOError: [Errno 13] Permission denied: 'bob.txt'

Test 120 failed: /home/michael/cwltool/env3.9/bin/cwltool --podman --outdir=/tmp/tmpqanpf0we --quiet tests/writable-dir-docker.cwl tests/empty.json
Test empty writable dir with InitialWorkDirRequirement inside Docker
Returned non-zero
touch: emptyWritableDir/blurg: Permission denied
WARNING [job writable-dir-docker.cwl] completed permanentFail
WARNING Final process status is permanentFail

Test 235 failed: /home/michael/cwltool/env3.9/bin/cwltool --podman --outdir=/tmp/tmpyfh92_4z --quiet tests/symlink-legal.cwl tests/empty.json
symlink to file inside of working directory should be retrieved
Returned non-zero
ERROR Unhandled error, try again with --debug for more information:
  [Errno 13] Permission denied: '/tmp/34j4od_h/adir/original.txt'

Test 236 failed: /home/michael/cwltool/env3.9/bin/cwltool --podman --outdir=/tmp/tmp7gt1tuam --quiet tests/inp_update_wf.cwl tests/empty.json
inplace update has side effect on file content
Returned non-zero
WARNING Workflow checker warning:
tests/inp_update_wf.cwl:58:5: 'wait' is not an input parameter of ordereddict([('class',
                              'ExpressionTool'), ('cwlVersion', 'v1.2'), ('expression',
                              "$({'output': parseInt(inputs.file1.contents)})"), ('inputs',
                              [ordereddict([('id',
                              'file:///home/michael/cwl-v1.2/tests/inp_update_wf.cwl#step3/run/file1'),
                              ('loadContents', True), ('type', 'File')])]), ('outputs',
                              [ordereddict([('id',
                              'file:///home/michael/cwl-v1.2/tests/inp_update_wf.cwl#step3/run/output'),
                              ('type', 'int')])]), ('requirements', [ordereddict([('class',
                              'InlineJavascriptRequirement')])]), ('id',
                              '_:69e5f16e-17f1-45cb-a1b7-fefe399c1c13')]), expected file1
Traceback (most recent call last):
  File "/var/lib/cwl/stg91b03c2f-50ea-4747-8f20-1434647fbcbf/updateval.py", line 2, in <module>
    f = open(sys.argv[1], "r+")
IOError: [Errno 13] Permission denied: 'out.txt'
WARNING [job step2] completed permanentFail
WARNING [step step2] completed permanentFail
WARNING Final process status is permanentFail

lgtm-com · 2021-09-20T15:04:35Z

This pull request introduces 1 alert when merging 21995a4 into c014c36 - view on LGTM.com

new alerts:

1 for Unused local variable

lgtm-com · 2021-09-20T18:35:55Z

This pull request introduces 2 alerts when merging 6556e4d into c014c36 - view on LGTM.com

new alerts:

2 for Unused import

lgtm-com · 2021-09-21T09:55:52Z

This pull request introduces 2 alerts when merging da19b28 into c014c36 - view on LGTM.com

new alerts:

2 for Unused import

Fixes #803

lgtm-com · 2021-09-21T10:33:34Z

This pull request introduces 2 alerts when merging cec2fe5 into c014c36 - view on LGTM.com

new alerts:

2 for Unused import

mr-c force-pushed the podman-volume-777 branch 2 times, most recently from 7e36f11 to 21995a4 Compare September 20, 2021 14:50

mr-c force-pushed the podman-volume-777 branch from 21995a4 to 303f97d Compare September 20, 2021 15:27

mr-c marked this pull request as draft September 20, 2021 15:51

mr-c added 3 commits September 20, 2021 17:57

add registry to dockerPull references

d6d9082

fix assert message

574d3e1

support (rootless) podman container engine

f6bc7b5

mr-c force-pushed the podman-volume-777 branch from 303f97d to 8749083 Compare September 20, 2021 18:09

mr-c marked this pull request as ready for review September 20, 2021 18:11

mr-c enabled auto-merge September 20, 2021 18:12

mr-c force-pushed the podman-volume-777 branch 2 times, most recently from ac277dc to 6556e4d Compare September 20, 2021 18:23

mr-c disabled auto-merge September 21, 2021 08:37

mr-c force-pushed the podman-volume-777 branch from 6556e4d to da19b28 Compare September 21, 2021 09:32

mr-c force-pushed the podman-volume-777 branch from da19b28 to 2fa2c90 Compare September 21, 2021 10:09

podman/singularity + nodejs

cec2fe5

Fixes #803

mr-c force-pushed the podman-volume-777 branch from 2fa2c90 to cec2fe5 Compare September 21, 2021 10:20

mr-c merged commit bb6fe86 into main Sep 21, 2021

mr-c mentioned this pull request Sep 21, 2021

chmods 777 for podman #1516

Closed

mr-c deleted the podman-volume-777 branch September 26, 2021 08:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

explicit support for podman #1519

explicit support for podman #1519

mr-c commented Sep 14, 2021

codecov bot commented Sep 14, 2021 •

edited

Loading

mr-c commented Sep 14, 2021

lgtm-com bot commented Sep 20, 2021

lgtm-com bot commented Sep 20, 2021

lgtm-com bot commented Sep 21, 2021

lgtm-com bot commented Sep 21, 2021

explicit support for podman #1519

explicit support for podman #1519

Conversation

mr-c commented Sep 14, 2021

codecov bot commented Sep 14, 2021 • edited Loading

Codecov Report

mr-c commented Sep 14, 2021

lgtm-com bot commented Sep 20, 2021

lgtm-com bot commented Sep 20, 2021

lgtm-com bot commented Sep 21, 2021

lgtm-com bot commented Sep 21, 2021

codecov bot commented Sep 14, 2021 •

edited

Loading