Running REPET on a single machine

[EttoreZ]

I'm trying to use REPET (v2.2) TEannot on a single machine instead of on a cluster.

There is a way to parallelize subprocesses anyway?

I already have a configuration to run all the steps, and step 6 (Comparison with data banks) seems very long while running one subprocess at time.

Thank you

[Beuss]

Hi EttoreZ,

REPET demand a lot of resources for analysing a genome. That why it have been designed for running on a cluster. Unless you want to annotate a small few repeated genome, I would not advise you to run REPET on a Desktop computer.

About the parallelization, most of the steps are already parallelized and use SGE or Torque (Slurm and RabbitMq for v2.5) to distribute jobs on available CPUs.
My suggest is, do you need to run whole steps of TEannot ? Do you want to annotate only transposables elements ? If yes do not run steps 4 and 5.
If you already ran step 2 (BLASTER/CENSOR/RepeatMasker/BLASTER on randomized genome /CENSOR on randomized genome /RepeatMasker on randomized genome), you can reuse the computed thresholds and do not run analysis on the randomized genome.

And to answer to your question:

There is a way to parallelize subprocesses anyway?

The REPET development Team work hard to reduce computing time on REPET, so if a step is not paralellized, it means that there is actually no better way to realize the analysis.

I hope that my answer will help you.

Best regards.

[EttoreZ]

Thank you for your answer.

Can you confirm that step 6 is not parallelized? BLASTER (blastx) batches are running once by time and are the slowest step.

[Beuss]

The 1st part of step 6 is parallelized. There is one blastx on each chunk of your genome, then it convert result from chunk to chromosomes with one job.

For illustrating my purpose, this is a log of step 6 extracted from one of numerous REPET analysis of my PHD:

START TEannot.py
version 2.1
local date/time = 2013-7-26 / 2:0:43
project name = g17a
project directory = /home/beuss/repet_projects/g17/g17a

beginning of step 6
Beginning Blaster (version 2.25)
ven jui 26 02:00:47 CEST 2013
XDFORMAT-WashU 1.0 [04-May-2006] [linux26-x64-I32LPF64 2006-05-10T17:21:37]
Start:  2013-07-26T02:01:15
XDF Output Database:  repbase17.01_aaSeq_cleaned_TE.fa_cut
 Alphabet:  NCBIstdaa(1)
Input: "repbase17.01_aaSeq_cleaned_TE.fa_cut"
 No. of sequences (letters) written:  11,114  (8,390,702)
 Longest sequence written (in database):  4527  (4527)
Total cpu time:  0.04u 0.09s 0.13t   Elapsed:  00:00:00
End:  2013-07-26T02:01:15
ven jui 26 02:01:15 CEST 2013
End Blaster (version 2.25)

align 'repbase17.01_aaSeq_cleaned_TE.fa' and the chunks via Blaster with blastx
submitting job(s) with groupid 'g17a_TEannot_S6_blastx' (2013-07-26 02:01:15)
[B][COLOR="RoyalBlue"]waiting for 54 job(s) with groupid 'g17a_TEannot_S6_blastx' (2013-07-26 02:01:16)
all jobs with groupid 'g17a_TEannot_S6_blastx' are finished (2013-07-26 02:12:01)
execution time of all jobs (seconds): 7133.223021
execution time per job: n=54 mean=132.097 var=344.275 sd=18.555 min=61.823 med=137.742 max=143.063[/COLOR][/B]
[B][COLOR="SeaGreen"]submitting job(s) with groupid 'g17a_TEannot_S6_convChunk2Chr' (2013-07-26 02:12:02)
waiting for 1 job(s) with groupid 'g17a_TEannot_S6_convChunk2Chr' (2013-07-26 02:12:02)
execution time per job: n=1 mean=6.274 var=0.000 sd=0.000 min=6.274 med=6.274 max=6.274[/COLOR][/B]
step 6 finished successfully

local date/time = 2013-7-26 / 2:27:27
version 2.1
END TEannot.py

In blue you have the paralellized blastx and in green you have the chunk to chromosome conversion.

Do not hesitate if you any other question.