SNPEff version 5.2a

[erinyoung]

I was going through issues and found #760 and then realized that there's a new version of SNPEff!

EDIT : After some trial and error, this is now ready to review!!!

I wanted to resolve this issue and update the Dockerfile to the latest image. I originally just copied the files from 5.1 and changed the SNPEff ARG to 5.2A, but then I couldn't get the Dockerfile to build. I made a few changes by separating out SNPEff and SNPSIFT, and I think this image would be improved by having an initial "builder" stage for all the maven installs prior to app, but this image does not currently build for me.

On my local system, I get stuck at install SNPSift lines.

#0 1.484 WARNING: An illegal reflective access operation has occurred
#0 1.484 WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/usr/share/maven/lib/guice.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
#0 1.484 WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1
#0 1.484 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
#0 1.484 WARNING: All illegal access operations will be denied in a future release
#0 1.779 [INFO] Scanning for projects...
#0 1.999 [INFO] 
#0 1.999 [INFO] ------------------------< org.snpsift:SnpSift >-------------------------
#0 2.000 [INFO] Building SnpSift 5.1
#0 2.001 [INFO] --------------------------------[ jar ]---------------------------------
#0 2.181 Downloading from maven: https://repo1.maven.org/maven2/org/snpeff/SnpEff/5.1/SnpEff-5.1.pom
#0 2.481 Downloading from ncimvn-public: https://ncimvn.nci.nih.gov/nexus/content/groups/public/org/snpeff/SnpEff/5.1/SnpEff-5.1.pom
#0 2.988 Downloading from typesafe: http://repo.typesafe.com/typesafe/releases/org/snpeff/SnpEff/5.1/SnpEff-5.1.pom
#0 3.310 Downloading from hadoop-bam: http://hadoop-bam.sourceforge.net/maven/org/snpeff/SnpEff/5.1/SnpEff-5.1.pom
#0 3.507 Downloading from central: https://repo.maven.apache.org/maven2/org/snpeff/SnpEff/5.1/SnpEff-5.1.pom
#0 3.647 Downloading from maven: https://repo1.maven.org/maven2/org/antlr/antlr4/4.9.3/antlr4-4.9.3.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/antlr/antlr4/4.9.3/antlr4-4.9.3.pom (6.2 kB at 60 kB/s)
#0 3.753 Downloading from maven: https://repo1.maven.org/maven2/org/antlr/antlr4-master/4.9.3/antlr4-master-4.9.3.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/antlr/antlr4-master/4.9.3/antlr4-master-4.9.3.pom (4.4 kB at 76 kB/s)
#0 3.824 Downloading from maven: https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.9.3/antlr4-runtime-4.9.3.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/antlr/antlr4-runtime/4.9.3/antlr4-runtime-4.9.3.pom (3.6 kB at 74 kB/s)
#0 3.877 Downloading from maven: https://repo1.maven.org/maven2/org/antlr/antlr-runtime/3.5.2/antlr-runtime-3.5.2.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/antlr/antlr-runtime/3.5.2/antlr-runtime-3.5.2.pom (2.2 kB at 50 kB/s)
#0 3.925 Downloading from maven: https://repo1.maven.org/maven2/org/antlr/antlr-master/3.5.2/antlr-master-3.5.2.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/antlr/antlr-master/3.5.2/antlr-master-3.5.2.pom (12 kB at 267 kB/s)
#0 3.977 Downloading from maven: https://repo1.maven.org/maven2/org/antlr/ST4/4.3.1/ST4-4.3.1.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/antlr/ST4/4.3.1/ST4-4.3.1.pom (4.9 kB at 105 kB/s)
#0 4.028 Downloading from maven: https://repo1.maven.org/maven2/org/abego/treelayout/org.abego.treelayout.core/1.0.3/org.abego.treelayout.core-1.0.3.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/abego/treelayout/org.abego.treelayout.core/1.0.3/org.abego.treelayout.core-1.0.3.pom (5.2 kB at 110 kB/s)
#0 4.079 Downloading from maven: https://repo1.maven.org/maven2/org/glassfish/javax.json/1.0.4/javax.json-1.0.4.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/glassfish/javax.json/1.0.4/javax.json-1.0.4.pom (9.2 kB at 196 kB/s)
#0 4.130 Downloading from maven: https://repo1.maven.org/maven2/org/glassfish/json/1.0.4/json-1.0.4.pom
Downloaded from maven: https://repo1.maven.org/maven2/org/glassfish/json/1.0.4/json-1.0.4.pom (11 kB at 209 kB/s)
#0 4.189 Downloading from maven: https://repo1.maven.org/maven2/com/ibm/icu/icu4j/69.1/icu4j-69.1.pom
Downloaded from maven: https://repo1.maven.org/maven2/com/ibm/icu/icu4j/69.1/icu4j-69.1.pom (4.9 kB at 103 kB/s)
#0 4.298 [INFO] ------------------------------------------------------------------------
#0 4.298 [INFO] BUILD FAILURE
#0 4.298 [INFO] ------------------------------------------------------------------------
#0 4.300 [INFO] Total time:  2.535 s
#0 4.300 [INFO] Finished at: 2023-11-13T22:56:23Z
#0 4.300 [INFO] ------------------------------------------------------------------------
#0 4.302 [ERROR] Failed to execute goal on project SnpSift: Could not resolve dependencies for project org.snpsift:SnpSift:jar:5.1: Failed to collect dependencies at org.snpeff:SnpEff:jar:5.1: Failed to read artifact descriptor for org.snpeff:SnpEff:jar:5.1: Could not transfer artifact org.snpeff:SnpEff:pom:5.1 from/to typesafe (http://repo.typesafe.com/typesafe/releases/): Transfer failed for http://repo.typesafe.com/typesafe/releases/org/snpeff/SnpEff/5.1/SnpEff-5.1.pom: Connect to repo.typesafe.com:80 [repo.typesafe.com/52.87.133.68, repo.typesafe.com/52.2.125.187, repo.typesafe.com/52.54.60.92] failed: Connection refused (Connection refused) -> [Help 1]

SNPSift does not have a 5.2* tag.

I also added 'curl' and the command in question from #760 that wasn't working.

Pull Request (PR) checklist:

• Include a description of what is in this pull request in this message.
• The dockerfile successfully builds to a test target for the user creating the PR. (i.e. docker build --tag samtools:1.15test --target test docker-builds/samtools/1.15 )
• Directory structure as name of the tool in lower case with special characters removed with a subdirectory of the version number (i.e. spades/3.12.0/Dockerfile)
  • (optional) All test files are located in same directory as the Dockerfile (i.e. shigatyper/2.0.1/test.sh)
• Create a simple container-specific README.md in the same directory as the Dockerfile (i.e. spades/3.12.0/README.md)
  • If this README is longer than 30 lines, there is an explanation as to why more detail was needed
• Dockerfile includes the recommended LABELS
• Main README.md has been updated to include the tool and/or version of the dockerfile(s) in this PR
• Program_Licenses.md contains the tool(s) used in this PR and has been updated for any missing

[erinyoung]

This PR is now ready for review.

Things of note: SnpEff's dependency versions need to exist. This means that if the latest version for SNPSift is 5.1, that's what the SnpEff's dependency versions are. SnpEff can still be its latest version.

Also, in this version of the image, I attempted to add the scripts in SnpEff/scripts for the end-user. These have hardcoded path to the jarfiles which was causing issues, so I've adjusted the jarfile paths to match those in the image. I couldn't find tests for all of them, so right now I'm mostly hoping that they all work.

[Kincekara]

@erinyoung I guess you didn't want to use precompiled files since there is no version information.
I tried a staged build, and it seems it worked. The staged build decreases uncompressed size from 1.79 GB to 573MB.
I didn't test it extensively, but do you mind changing the dockerfile if it works in your tests?

ARG SNPEFF_VER="5.2a"
ARG SNPEFF_JAR_VER="5.2"
ARG SNPSIFT_VER="5.1"

## Builder ##
FROM ubuntu:focal as builder

ARG SNPEFF_VER
ARG SNPEFF_JAR_VER
ARG SNPSIFT_VER

# Install open using apt 
RUN apt-get update && apt-get install -y software-properties-common && \
  apt-get update && apt-get install -y \
  openjdk-11-jre \
  ant \
  maven \
  curl \
  wget && apt-get autoclean && rm -rf /var/lib/apt/lists/*

# Install dependencies and snpEff executables from the source code using Maven.
RUN wget https://github.com/pcingola/SnpEff/archive/refs/tags/v${SNPEFF_VER}.tar.gz && \
  tar -xvf v${SNPEFF_VER}.tar.gz && \
  rm v${SNPEFF_VER}.tar.gz && \
  cd /SnpEff-${SNPEFF_VER}/lib && \
  # install Antlr
  mvn install:install-file -Dfile=antlr-4.5.1-complete.jar -DgroupId=org.antlr -DartifactId=antlr -Dversion=4.5.1 -Dpackaging=jar && \
  # install BioJava core
  mvn install:install-file -Dfile=biojava3-core-3.0.7.jar -DgroupId=org.biojava -DartifactId=biojava3-core -Dversion=3.0.7 -Dpackaging=jar && \
  # install BioJava structure
  mvn install:install-file -Dfile=biojava3-structure-3.0.7.jar -DgroupId=org.biojava -DartifactId=biojava3-structure -Dversion=3.0.7 -Dpackaging=jar && \
  cd /SnpEff-${SNPEFF_VER} && \
  # skipping scripts_build/make.sh and instead putting relevant commands here
  mvn clean compile assembly:single jar:jar && \
  cp /SnpEff-${SNPEFF_VER}/target/SnpEff-${SNPEFF_JAR_VER}-jar-with-dependencies.jar /snpEff.jar && \
  # The version for the dependencies matches that of SNPSIFT
  mvn install:install-file -Dfile=/SnpEff-${SNPEFF_VER}/target/SnpEff-${SNPEFF_JAR_VER}.jar -DgroupId=org.snpeff -DartifactId=SnpEff -Dversion=${SNPSIFT_VER} -Dpackaging=jar -DgeneratePom=true --quiet && \
  mkdir /data

# Install dependencies and SnpSift executables from the source code using Maven.
RUN wget https://github.com/pcingola/SnpSift/archive/refs/tags/v${SNPSIFT_VER}.tar.gz && \
  tar -xvf v${SNPSIFT_VER}.tar.gz && \
  rm v${SNPSIFT_VER}.tar.gz && \
  cd /SnpSift-${SNPSIFT_VER} && \
  mvn clean compile assembly:single jar:jar && \
  cp target/SnpSift-${SNPSIFT_VER}-jar-with-dependencies.jar /SnpSift.jar && \
  mvn install:install-file -Dfile=target/SnpSift-${SNPSIFT_VER}.jar -DgroupId=org.snpsift -DartifactId=SnpSift -Dversion=${SNPSIFT_VER} -Dpackaging=jar -DgeneratePom=true 

# Modify java executables and set environment variable $PATH
RUN mv SnpEff-${SNPEFF_VER} snpEff && \
  mv snpEff.jar snpEff && \
  mv SnpSift.jar snpEff && \
  chmod +x /snpEff/snpEff.jar && \
  echo "#!/bin/bash" >> /snpEff/snpeff && \
  chmod +x /snpEff/SnpSift.jar && \
  echo "#!/bin/bash" >> /snpEff/snpsift && \
  echo "exec java -jar /snpEff/snpEff.jar """"$""@"""" " >> /snpEff/snpeff && \
  chmod +x /snpEff/snpeff && \
  echo "exec java -jar /snpEff/SnpSift.jar """"$""@"""" " >> /snpEff/snpsift && \
  chmod +x /snpEff/snpsift

# Modify scripts to jar location
RUN for file in $(grep -iw "snpEff.jar" /snpEff/scripts/*sh | cut -f 1 -d ":" ) ; do cat $file | sed 's/snpEff.jar/\/snpEff\/snpEff.jar/g' > $file.tmp ; mv $file.tmp $file  ; done && \
  for file in $(grep -iw "SnpSift.jar" /snpEff/scripts/*sh | cut -f 1 -d ":" ) ; do cat $file | sed 's/snpEff.jar/\/snpEff\/SnpSift.jar/g' > $file.tmp ; mv $file.tmp $file  ; done && \
  chmod +x /snpEff/scripts/*sh

## App ##
FROM ubuntu:focal as app

# Metadata
LABEL base.image="ubuntu:focal"
LABEL dockerfile.version="1"
LABEL software="SnpEff & SnpSift"
LABEL software.version=$SNPEFF_VER
LABEL snpeff.jarfile.version=$SNPEFF_JAR_VER
LABEL snpsift.software.version=$SNPSIFT_VER
LABEL description="Genetic variant annotation and effect prediction toolbox."
LABEL description.SnpSift="Used after SnpEff annotation to filter and manipulate annotated files."
LABEL website="https://pcingola.github.io/SnpEff"
LABEL license="https://github.com/pcingola/SnpEff/blob/master/LICENSE.md"
LABEL maintainer="Tom Iverson"
LABEL maintainer.email="[email protected]"

ARG DEBIAN_FRONTEND=noninteractive

# perl, python2, and r are required for the scripts
RUN apt-get update && apt-get install --no-install-recommends -y \
  openjdk-11-jre-headless \
  perl \
  r-base \
  python-is-python2 \
  curl \
  && apt-get autoclean && rm -rf /var/lib/apt/lists/*

COPY --from=builder /snpEff/snpeff /snpEff/snpsift /snpEff/snpEff.jar /snpEff/SnpSift.jar /snpEff/snpEff.config /snpEff/
COPY --from=builder /snpEff/scripts/ /snpEff/scripts/ 

ENV PATH="${PATH}:/snpEff:/snpEff/scripts" 

CMD snpEff -h

WORKDIR /data

# Run test of annotation
FROM app as test

COPY test_snpeff_snpsift.sh .

RUN bash test_snpeff_snpsift.sh 

# from issue https://github.com/StaPH-B/docker-builds/issues/760
RUN buildDbNcbi.sh CP014866.1

[erinyoung]

@Kincekara , I think adding in a builder stage is a great idea.

I copied and pasted your recommendation from this conversation, and everything still appears to work fine.

[Kincekara]

@erinyoung Thank you for your modifications. I realized test_snpeff_snpsift.sh needs wget to work properly. I added it to the test stage. I think it is ready to go now.
I will merge and deploy this PR.

[Kincekara]

Thank you for your PR! You can check deploy from here: https://github.com/StaPH-B/docker-builds/actions/runs/7019620274