Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Happy EDTA users with successful cases #15

Closed
oushujun opened this issue Aug 21, 2019 · 10 comments
Closed

Happy EDTA users with successful cases #15

oushujun opened this issue Aug 21, 2019 · 10 comments
Labels
installation Good for newcomers

Comments

@oushujun
Copy link
Owner

oushujun commented Aug 21, 2019 •
edited
Loading

Hi all,

Just update the testing result. It seems that new release TIR can close this issue.

  1. Please install a new env for the EDTA 20190802 release
  2. Follow the step by the Shujun provided.
  • EDTA_raw
  • EDTA_processF
  • EDTA -step final
  1. The time and resource of my plant genome (336M plant genome, 58% repeat estimated by the GenomeScope, 24 cores machine)
Step maxvmem time(h) raw_fa size
Helitron 7.914GB 2.352222 1.3Mb
MITE 1.529GB 1.815278 4.9kb
TIR 42.127GB 4.895556 20Mb
LTR 19.049GB 1.417222 2.5Mb
EDTA_Final 19.388GB 19.42389 19Mb

Thanks for the developing.

Bests,
Zhigui

Originally posted by @baozg in #4 (comment)
@oushujun oushujun changed the title Happy users with their genomes' TE being annotated by EDTA Happy EDTA users with successful cases Aug 21, 2019
@oushujun oushujun added the installation Good for newcomers label Aug 21, 2019
@QiushiLi
Copy link

QiushiLi commented Aug 21, 2019 •
edited by oushujun
Loading

Thanks for developing EDTA, and this is for your reference.

Fish genome ～480 Mb, EDTA.pl (commit 82b16f6) job finished in 7h.
Linux version 3.10.0-957.27.2.el7.x86_64 (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) -t 24

Step raw_fa size
TIR 32.3 Mb
MITE 20 Kb
LTR 988 Kb
Helitron 1.7 Mb
TElib 30.4 Mb

Cheers，
Qiushi

@philippbayer
Copy link

philippbayer commented Sep 16, 2019 •
edited
Loading

Hi

Finally, my rerun without splitting EDTA_raw.pl into subsets finished too :)

It's an amphidiploid plant genome, about 1GB in size, run with 15 threads, commit 757a96d (the one where MITEHunter is turned off)

It ran for 242.8 hours (10.1 days) if run sequentially (so no splitting up of EDTA_raw.pl steps)

Here's the whole log so you can see which steps took how long:

Wed Sep  4 21:07:28 AWST 2019   EDTA_raw: Check files and dependencies, prepare working directories.

Wed Sep  4 21:07:28 AWST 2019   Start to find LTR candidates.

Wed Sep  4 21:07:28 AWST 2019   Identify LTR retrotransposon candidates from scratch.

Tue Sep 10 00:34:18 AWST 2019   Finish finding LTR candidates.

Tue Sep 10 00:34:18 AWST 2019   Start to find TIR candidates.

Tue Sep 10 00:34:18 AWST 2019   Identify TIR candidates from scratch.

Species: others
rm: cannot remove `./TIR-Learner/*': No such file or directory
Finish finding TIR candidates.

Fri Sep 13 09:13:15 AWST 2019   Start to find MITE candidates.

Fri Sep 13 09:13:15 AWST 2019   Identify MITE candidates from scratch.

Fri Sep 13 09:13:15 AWST 2019   Warning: Because MITE-Hunter is too slow and only contribute limited new TIR candidates, it is taken down temporary until a better solution is found.

Fri Sep 13 09:13:15 AWST 2019   Finish finding MITE candidates.

Fri Sep 13 09:13:15 AWST 2019   Start to find Helitron candidates.

Fri Sep 13 09:13:15 AWST 2019   Identify Helitron candidates from scratch.

Fri Sep 13 23:32:36 AWST 2019   Finish finding Helitron candidates.

Fri Sep 13 23:32:36 AWST 2019   Execution of EDTA_raw.pl is finished!
Sun Sep 15 07:17:49 AWST 2019   EDTA basic and advcanced filters finished.

Sun Sep 15 07:17:49 AWST 2019   Perform EDTA final steps to generate a non-redundant comprehensive TE library:

                                Skip the RepeatModeler step (-sensitive 0).
Run EDTA.pl -step final -sensitive 1 if you want to use RepeatModeler.

Sun Sep 15 14:01:06 AWST 2019   EDTA final stage finished! Check out the final EDTA TE library: ragoo.fasta.EDTA.TElib.fa

Sizes:

size raw_fa size
TIR 55M
MITE 55M (copy of TIR due to no MITEHunter)
LTR 12M
Helitron 23M
TElib 46M

The final TElib has 22,921 sequences.

Running a final RepeatMasker run with the final TElib gives me about 75% of the genome being repeats, which makes me very happy :) I should check how many of these repeats are known within Dfam and Repbase, my guess is not that many as a previous run with an older Repbase version gave me a much lower count of repeats

MUCH LATER EDIT:

I've now rerun the final stage with -sensitive 1, which turns on RepeatModeler, which takes forever. WIth the same genome as above:

Mon Sep 30 17:04:54 AWST 2019   Perform EDTA final steps to generate a non-redundant comprehensive TE library:

                                Use RepeatModeler to identify any remaining TEs that are missed by structure-based methods.

Sat Oct  5 14:51:32 AWST 2019   EDTA final stage finished! Check out the final EDTA TE library: ragoo.fasta.EDTA.TElib.fa

So yeah, adding this step to my pipeline added 5 days of runtime! What this did is add 4,186 repeats of type 'unknown' to my TElib.fa, but these are all tiny (the total filesize increased from 46MB to 47MB). @oushujun is this a problem on my end? Should RepeatModeler have assigned classes to those repeats?

@liu3zhenlab
Copy link

(EDTA) v1.5
Successfully finished a maize genome:
TElib: 55Mb
Thanks Shujun.

@elsemikk
Copy link

Thank you so much for making this awesome tool! It makes TA annotation so much easier and I will for sure use it for my next genomes.

I tried it on a bird (~1.2Gb) with -sensitive 0 and a custom protein library and got this result:

step threads Real time User time Number
Raw Helitron 8 605m16s 521m34s 146
Raw TIR 8 1425m38s 8438m38s 5108
Raw LTR 8 49.51s 81m16s 157
Rest of pipeline 23 79m27 1334m50s 5278

(It probably would have been faster if I had given more threads to the TIR search instead of splitting equally between the three searches!)

Thank you,
Else

@oushujun oushujun pinned this issue Dec 1, 2019
@oushujun oushujun mentioned this issue Jan 2, 2020
Closed
@sunnycqcn
Copy link

sunnycqcn commented May 18, 2020 •
edited
Loading

I recently finished one plant genome about 1.8 Gb.
But I have a little bit confused the repeatmasker statistic results. It only reported LTR and DNA element content. I can nont check any secondary element content, like SINES.

file name: STH.fa
sequences: 3037
total length: 1601564480 bp (1584403095 bp excl N/X-runs)
GC level: 41.61 %
bases masked: 1309747987 bp ( 81.78 %)

           number of      length   percentage
           elements*    occupied  of sequence

SINEs: 0 0 bp 0.00 %
ALUs 0 0 bp 0.00 %
MIRs 0 0 bp 0.00 %

LINEs: 0 0 bp 0.00 %
LINE1 0 0 bp 0.00 %
LINE2 0 0 bp 0.00 %
L3/CR1 0 0 bp 0.00 %

LTR elements: 1584413 1074016988 bp 67.06 %
ERVL 0 0 bp 0.00 %
ERVL-MaLRs 0 0 bp 0.00 %
ERV_classI 0 0 bp 0.00 %
ERV_classII 0 0 bp 0.00 %

DNA elements: 832180 278509658 bp 17.39 %
hAT-Charlie 0 0 bp 0.00 %
TcMar-Tigger 0 0 bp 0.00 %

Unclassified: 78986 11993249 bp 0.75 %

Total interspersed repeats:1364519895 bp 85.20 %

Small RNA: 0 0 bp 0.00 %

Satellites: 0 0 bp 0.00 %
Simple repeats: 0 0 bp 0.00 %
Low complexity: 0 0 bp 0.00 %

  • most repeats fragmented by insertions or deletions
    have been counted as one element

The query species was assumed to be homo
RepeatMasker Combined Database: Dfam_3.0, RepBase-20170127

run with rmblastn version 2.9.0+
The query was compared to classified sequences in "STH.fa.EDTA.TElib.fa"

Wed May 6 11:50:19 EDT 2020 Dependency checking:
All passed!
Wed May 6 11:50:45 EDT 2020 Obtain raw TE libraries using various structure-based programs:
Thu May 7 03:10:34 EDT 2020 Obtain raw TE libraries finished.

Thu May 7 03:10:34 EDT 2020 Perform EDTA advcance filtering for raw TE candidates and generate the stage 1 library:

Thu May 7 06:13:03 EDT 2020 EDTA advcance filtering finished.

Thu May 7 06:13:03 EDT 2020 Perform EDTA final steps to generate a non-redundant comprehensive TE library:

			Use RepeatModeler to identify any remaining TEs that are missed by structure-based methods.

			RepeatModeler is finished, but no consensi.fa.classified files found.

			Skipping the CDS cleaning step (-cds [File]) since no CDS file is provided.

Sun May 10 02:08:29 EDT 2020 EDTA final stage finished! Check out the final EDTA TE library: STH.fa.EDTA.TElib.fa
Sun May 10 02:08:29 EDT 2020 Perform post-EDTA analysis for whole-genome annotation:

Sun May 10 09:50:57 EDT 2020 TE annotation using the EDTA library has finished! Check out:
Whole-genome TE annotation (total TE: 79.16%): STH.fa.EDTA.TEanno.gff
Low-threshold TE masking for MAKER gene annotation (masked: 48.06%): STH.fa.MAKER.masked

Sun May 10 09:51:05 EDT 2020 Evaluate the level of inconsistency for whole-genome TE annotation (slow step):

Mon May 18 06:27:53 EDT 2020 Evaluation of TE annotation finished! Check out these files:

			Overall: STH.fa.EDTA.TE.fa.stat.all.sum
			Nested: STH.fa.EDTA.TE.fa.stat.nested.sum
			Non-nested: STH.fa.EDTA.TE.fa.stat.redun.sum

Confusion matrix of STH.fa.EDTA.TE.fa.stat for the all category
DNA/D DNA/DTA DNA/DTC DNA/DTH DNA/DTM DNA/DTT DNA/Heeaning DNA/HelitrTM DNA/Helitron DNAa Gypsy L LT LT/Copia LTR/ LTR/Cocovering LTR/Copia LTR/Copig LTR/Coping LTR/G LTR/Gypnknown LTR/Gypof LTR/Gypsy LTR/ering LTR/u LTR/uly LTR/unTR/unknown LTR/ung LTR/unking LTR/unkn LTR/unknowR/unknown LTR/unknown LTR/unknownring LTTR/Gypsy LTy Lof Ly MITE/DTA MITE/DTC MITE/DTH MITE/DTM MITE/DTT covering Misclas_rate
DNA/D 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
DNA/DTA 0 115616 10068 2813 17086 57 0 0 18704 0 0 0 0 0 0 0 10275 0 0 0 0 0 6465 0 0 0 0 0 0 0 0 4218 0 0 0 0 0 2144 159 44 2403 2 0 0.3917
DNA/DTC 0 10936 61128 1089 7734 413 0 0 9215 0 0 0 0 0 0 0 3738 0 0 0 0 0 1445 0 0 0 0 0 0 0 0 2208 0 0 0 0 0 172 937 728 2618 16 0 0.4029
DNA/DTH 0 2528 1434 29335 3219 11 0 0 5818 0 0 0 0 0 0 0 1661 0 0 0 0 0 561 0 0 0 0 0 0 0 0 1438 0 0 0 0 0 275 1 699 471 0 0 0.3818
DNA/DTM 0 12351 6859 3702 348436 121 0 0 18988 0 0 0 0 0 0 0 25783 0 0 0 0 0 9673 0 0 0 0 0 0 0 0 6997 0 0 0 0 0 613 77 855 14894 59 0 0.2247
DNA/DTT 0 110 518 25 222 4388 0 0 202 0 0 0 0 0 0 0 138 0 0 0 0 0 3 0 0 0 0 0 0 0 0 709 0 0 0 0 0 0 0 2 856 4 0 0.3886
DNA/Heeaning 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
DNA/HelitrTM 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
DNA/Helitron 0 22063 11264 8713 96179 137 0 0 830187 0 0 0 0 0 0 0 154178 0 0 0 0 0 31254 0 0 0 0 0 0 0 0 20428 0 0 0 0 0 86 99 707 104423 93 0 0.3513
DNAa 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
Gypsy 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
L 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000
LT 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LT/Copia 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/Cocovering 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/Copia 0 9424 17496 2292 67110 66 0 0 29216 0 0 0 0 0 0 0 972675 0 0 0 0 0 40664 0 0 0 0 0 0 0 0 128559 0 0 0 0 0 1222 8 317 955 4 0 0.2341
LTR/Copig 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/Coping 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/G 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/Gypnknown 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/Gypof 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/Gypsy 0 10301 1774 622 26632 6 0 0 14217 0 0 0 0 0 0 0 49642 0 0 0 0 0 2675171 0 0 0 0 0 0 1 0 1111152 0 0 0 0 0 155 243 100 1210 1 0 0.3125
LTR/ering 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/u 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/uly 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/unTR/unknown 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/ung 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/unking 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/unkn 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/unknowR/unknown 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTR/unknown 0 6511 14195 3411 34422 135 0 0 28722 0 0 0 0 0 0 0 148343 0 0 0 0 0 799980 0 0 0 0 0 0 0 0 2486668 0 0 0 0 0 456 16 316 940 136 0 0.2944
LTR/unknownring 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTTR/Gypsy 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
LTy 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1.0000
Lof 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000
Ly 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000
MITE/DTA 0 2087 149 431 682 0 0 0 276 0 0 0 0 0 0 0 741 0 0 0 0 0 80 0 0 0 0 0 0 0 0 349 0 0 0 0 0 14944 0 14 503 0 0 0.2622
MITE/DTC 0 75 1473 0 79 0 0 0 105 0 0 0 0 0 0 0 23 0 0 0 0 0 103 0 0 0 0 0 0 0 0 11 0 0 0 0 0 0 6397 0 3 0 0 0.2264
MITE/DTH 0 41 226 930 666 1 0 0 508 0 0 0 0 0 0 0 129 0 0 0 0 0 30 0 0 0 0 0 0 0 0 79 0 0 0 0 0 6 0 5830 278 0 0 0.3317
MITE/DTM 0 998 1429 212 3682 232 0 0 534 0 0 0 0 0 0 0 598 0 0 0 0 0 400 0 0 0 0 0 0 0 0 854 0 0 0 0 0 362 2 511 33915 0 0 0.2244
MITE/DTT 0 1 0 0 43 73 0 0 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 1 0 0 0 327 0 0.2937
covering 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1.0000

@hkchi
Copy link
Contributor

hkchi commented Jun 11, 2020

Thank you for the awesome tool. Successfully finished three plant genomes (~3-3.5 Gb):

Step time size
Raw LTR 114-128h 542-695Mb
Raw TIR 56-99h 743-790kb
Raw Helitron 36-60h 186-196Mb
Rest of pipeline 52-53h 56-66Mb

Raw steps run with EDTA v1.7.8 from bioconda, the rest with EDTA v1.8.5. All run with 48 threads (Intel Platinum 8160F Skylake @ 2.1Ghz, RAM=192000M), --sensitive 0 in the FINAL step and CDS provided.

Cheers,
Kaichi

@oushujun oushujun mentioned this issue Jun 18, 2020
Closed
@yangyi09
Copy link

Hi,

Thanks for developing such a cool software,I have successfully annotated the TEs of an insect genome.

But I want to known if the library constructed by EDTA could be used as the consensus sequences for calculating the age of individual TEs?

Thank you,
Yang yi

@oushujun
Copy link
Owner Author

oushujun commented Jul 16, 2020 via email

@oushujun
Copy link
Owner Author

Thank you all for using and promoting EDTA since its early stage. As for now, the program has gained some popularity, and works using EDTA have been showing up online. This thread can be retired. Below are some studies citing EDTA:

Identification of key tissue-specific, biological processes by integrating enhancer information in maize gene regulatory networks

Homologous chromosomes in asexual rotifer Adineta vaga suggest automixis

The transposable elements of the Drosophila serrata reference panel

Chromosome-level Genome Assembly of a Regenerable Maize Inbred Line A188

The genome of the Xingu scale-backed antbird (Willisornis vidua nigrigula) reveals lineage-specific adaptations

Improved Brassica oleracea JZS assembly reveals significant changing of LTR-RT dynamics in different morphotypes

@oushujun oushujun unpinned this issue Aug 19, 2021
@Wenwen012345
Copy link

Dear all, Shujun

Thank you for the awesome tool. Successfully finished one plant （Rhododendron）genome (~850M):

size raw_fa size Percent
TIR 143M 15.7%
LTR 660M 43.2%
Helitron 15M 1.6%
TElib 16.9M 60.96%（Total）

command
EDTA.pl --genome genome.fa --anno 1 --threads 40 --step all --species others

The sub-item time is not very clear (it was interrupted several times in the middle), and the total time is about 20h.

Sincerely,
Wen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
installation Good for newcomers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@philippbayer @liu3zhenlab @oushujun @QiushiLi @hkchi @sunnycqcn @elsemikk @yangyi09 @Wenwen012345