Investigate failing concatenate_vcfs singularity tests

[maxulysse]

cf #1032

[asp8200]

This error is now on the dev-branch, right? I'll see if I can reproduce it when running pytest (or corresponding nf-cmd) locally.

[asp8200]

So I tried to run the following cmd locally:

TMPDIR=/home/aspe/nf_tmp PROFILE=singularity nextflow run main.nf -profile test --input ./tests/csv/3.0/mapped_joint_bam.csv --concatenate_vcfs --tools deepvariant,freebayes,haplotypecaller,manta,mpileup,strelka,tiddit --step variant_calling --outdir results

It crashed with the follow msg:

  error [java.lang.IllegalStateException]: java.lang.IllegalStateException: Failed to pull singularity image
  command: singularity pull  --name docker.io-google-deepvariant-1.5.0.img.pulling.1685002196453 docker://docker.io/google/deepvariant:1.5.0 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:7608715873ec5c02d370e963aa9b19a149023ce218887221d93fe671b3abbf58
    ...
    Copying blob sha256:13d6209546007ab81a359ab76d1eb2e3b8fd90a0b22e354bfd9766ddfc143bf7
    Copying blob sha256:123b4e4b7a6ebc6f5276294719f794504f2a086d037ee62b7de5ad03f52a3701
    Copying config sha256:5c4fc72736e42f383383e38a39ce139b38f853fb59e68b1ff5c24b927db0ec98
    Writing manifest to image destination
    Storing signatures
    2023/05/25 10:11:50  info unpack layer: sha256:7608715873ec5c02d370e963aa9b19a149023ce218887221d93fe671b3abbf58
    2023/05/25 10:11:50  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
    2023/05/25 10:11:50  warn xattr{/faststorage/home/aspe/NO_BACKUP/singularity_tmpdir/build-temp-085452828/rootfs/etc/gshadow} destination filesystem does not support xattrs, further warnings will be suppressed
    2023/05/25 10:11:56  info unpack layer: sha256:ff9c04d6f4fdbf7ee21a95fed7f66b0f0698b14a85616e4de50244179da2747f
    2023/05/25 10:11:56  info unpack layer: sha256:ccc0633ad137acb30952dc1289a5cfc14df75a91fe6604b305755252305fd83b
    2023/05/25 10:11:56  info unpack layer: sha256:31c4e73f93e5498ede44a871f4c43e9c1c9a10343f4742d9f8ca0cea6fbbcf13
    2023/05/25 10:11:56  info unpack layer: sha256:5f25a39487ec2bac075fd79c69796e619b0aff4acca2f3e9cac29b1bc3144786
    2023/05/25 10:12:26  info verified reader: 11452858 bytes of trailing data discarded from vrdr[/faststorage/home/aspe/NO_BACKUP/singularity_tmpdir/bundle-temp-698557131/blobs/sha256/5f25a39487ec2bac075fd79c69796e61
9b0aff4acca2f3e9cac29b1bc3144786]
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: opt/co
nda/pkgs/bcftools-1.10.2-h4f4756c_3/bin/color-chrs.pl: link: unpriv.link: unpriv.wrap target: operation not permitted

[maxulysse]

Can you check if deepvariant itself is working properly?

[asp8200]

Can you check if deepvariant itself is working properly?

I was just about to do that :-)

[asp8200]

FYI: The concatenate_vcfs-test works if I remove deepvariant, that is, this works:

TMPDIR=/home/aspe/nf_tmp PROFILE=singularity nextflow run main.nf -profile test --input ./tests/csv/3.0/mapped_joint_bam.csv --concatenate_vcfs --tools freebayes,haplotypecaller,manta,mpileup,strelka,tiddit --step variant_calling --outdir results

Something fishy with deepvariant

[maxulysse]

But tests were good for deepvariant: https://github.com/nf-core/sarek/actions/runs/5070041358/jobs/9111022814

[asp8200]

Hmmm 🤔 I also get the above-mentioned when trying to run the basic deepvariant test:

TMPDIR=/home/aspe/nf_tmp PROFILE=singularity nextflow run main.nf -profile test,tools_germline --tools deepvariant --outdir results

Now, I'll see if I can do just

singularity pull  --name docker.io-google-deepvariant-1.5.0.img.pulling.1685003040189 docker://docker.io/google/deepvariant:1.5.0

[asp8200]

Sorry about the confusion here. My issue with doing the singularity-pull was just related to the cluster on which I'm trying to run this. Anyways, I got the basic deepvariant test working, and I'll get back to testing concatenate_vcfs.