SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
BFAST - Help cutcopy11 SOLiD 17 10-17-2013 08:12 AM
BFAST bfast.submit.pl configuration epigen Bioinformatics 1 03-18-2011 07:51 AM
Support for parallelization of paired-end alignments with BWA Fabien Campagne Bioinformatics 0 12-17-2010 05:40 AM
BFAST thanks you for your help! (was: ... needs your help) nilshomer Bioinformatics 5 04-21-2010 08:29 PM
PubMed: Increased throughput by parallelization of library preparation for massive se Newsbot! Literature Watch 0 04-14-2010 03:01 AM

Reply
 
Thread Tools
Old 05-13-2011, 02:15 PM   #1
genome_anawk1
Junior Member
 
Location: USA

Join Date: May 2011
Posts: 7
Default bfast parallelization

Hello,

I am a new bfast user and have successfully run the full bfast job which included all the details. Now to speed up the processing (parallelize at various levels) at various steps I need to know the following from the authors and experts.

After using 10 unique set of indices suggested in the manual, I created 10 different index files.
refgenome.fa.cs.<j>.1.bif (j=1,..10).
I used the index files and the reference sequence and a unique reads.k.fastq file (k=1…N)
to <br>
search the indices (bfast index …)
perform local alignment (bfast localalign)
filter alignments (bfast postprocess)

1) My question is : Can I further parallelize the searches for indices (bfast index) step.
Can I run small jobs using independent index files for the above 3 steps and then merge the final *bam files after "bfast postprocess"?

2) how can I generate similar sized "reads.k.fastq" files . I noticed that some files generated are larger than others - perhaps twice as large. The compute time takes twice as long too.

Thanks in advance,
cheers

Last edited by genome_anawk1; 05-13-2011 at 02:21 PM.
genome_anawk1 is offline   Reply With Quote
Old 05-13-2011, 04:32 PM   #2
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Note: "bfast match" searches the index, while "bfast index" creates an index.

Here are the basic steps (remember to use multiple threads with the "-n" option).

1.
Partition the N reads into bins with R reads contained. Run these through "bfast match" to create ~X=N/R BMF files. Alternatively, if you have only one input file, use the "-s/-e" options to specify which reads to process for each invocation of "bfast match".

2.
For each BMF file from step 1, process them with "bfast localalign". You can also use "-s/-e" to further sub-divide each input file. Lets call the number of BAF files Y (since you may sub-divide, otherwise X equals Y).

3.
For each BAF file from step 2, process them with "bfast localalign" to get a SAM file.

4.
After all SAM files have been created, merge them with Picard or samtools (I recommend the former).

This is the best way to partition the various steps of bfast, since step 1 ("bfast match") may take much more memory (for large references) than step 2 ("bfast localalign") and so if you had a heterogenous cluster you could submit the step 1 jobs to high memory nodes and step 2 jobs to low memory nodes etc. This is the spirit of the "bfast.submit.pl" script found with the BFAST distribution. The latter script is not well supported (there be dragons).
nilshomer is offline   Reply With Quote
Old 05-13-2011, 05:22 PM   #3
genome_anawk1
Junior Member
 
Location: USA

Join Date: May 2011
Posts: 7
Default

Hi Nils,

Thanks very much for your prompt response. Apologies, I had some incorrect commands earlier. I am making the fixes here.

I did not use the "-s" or "-e" flags as I needed the algorithm to use color space information - the stress being on the accuracy. As the manual suggests (page 60), there was a trade off. Hence the long run time.

My parallelization question is regarding the bif files (index files in color space). For example I have 10 index files and N (N~100) reads file called reads.j.fastq (j=1..N). My jobs are split as shown below:

Using your set of masks, I create 10 bif files.
using mask_1
bfast index -f ref_genome.fa -m 111...11 -w 14 -i 1 -A 1

..
using mask_k
bfast index -f ref_genome.fa -m 110...11 -w 14 -i k -A 1

...
(k=1 ..10 such unique masks as suggested in the manual).

I am using the mouse genome, so is 10 an optimal basis for the hash masks. For now I am using "-w 14" but I guess that remains an open question for the mouse genome.

Then get the 10 unique bif files. For the next few steps do I need to keep ALL 10 bif files as inputs (alongwith the ref genome in color space and nucleotide space) for bfast match, bfast localalign, and bfast postprocess ..
Here is the parallelization that I now have ..

dir_wk_1/
bfast match -f ref_genome.fa -A 1 -r reads.1.fastq > bfast.matches_file.1.bmf
bfast localalign -f ref_genome.fa -m bfast.matches_file.1.bmf -A 1 > bfast.aligned.file.1.baf
bfast postprocess -f ref_genome.fa -i bfast.aligned.file.1.baf -A 1 > bfast.reported.file.1.sam


dir_wk_2/
bfast match -f ref_genome.fa -A 1 -r reads.2.fastq > bfast.matches_file.2.bmf
bfast localalign ..
bfast postprocess ...

...
dir_wk_N/
bfast match -f ref_genome.fa -A 1 -r reads.N.fastq > bfast.matches_file.N.bmf
bfast localalign ..
bfast postprocess ...

----------
Specifically do the 10 bif files have to be referenced in each and every N subdirectory. Can I make 10xN separate runs - where each run has only ONE bif file (index file) referenced to it and one reads.j.fastq file referenced to it.

Thanks very much, and apologies for the previous incorrect commands.
cheers,
new genome analyzer.

Last edited by genome_anawk1; 05-14-2011 at 08:12 AM.
genome_anawk1 is offline   Reply With Quote
Old 05-13-2011, 05:56 PM   #4
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Where's the "bfast match" command? You are definitely missing something. Try reading through the example in the manual's appendix.
nilshomer is offline   Reply With Quote
Old 05-14-2011, 08:13 AM   #5
genome_anawk1
Junior Member
 
Location: USA

Join Date: May 2011
Posts: 7
Default

Hi Nils,
Apologies, I fixed my previous query. I had copied the command incorrectly.
cheers,
new analyzer
genome_anawk1 is offline   Reply With Quote
Old 05-16-2011, 07:55 AM   #6
genome_anawk1
Junior Member
 
Location: USA

Join Date: May 2011
Posts: 7
Default

Hi Nils,

It is not obvious to me if one needs to use all the different index files / *.bif files (example 10 different files) at the same time for the three steps:
1- bfast match
2- bfast localalign
3- bfast postprocess

As posted previously, can I split jobs such that each of the ten bif files are processed separately. This will lead to 10xN separate jobs that run and consume less input time for reading in the dataset.

Hope you can please suggest. Thanks in advance

cheers,
genome_anawk1 is offline   Reply With Quote
Old 05-16-2011, 09:24 AM   #7
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by genome_anawk1 View Post
Hi Nils,

It is not obvious to me if one needs to use all the different index files / *.bif files (example 10 different files) at the same time for the three steps:
1- bfast match
2- bfast localalign
3- bfast postprocess

As posted previously, can I split jobs such that each of the ten bif files are processed separately. This will lead to 10xN separate jobs that run and consume less input time for reading in the dataset.

Hope you can please suggest. Thanks in advance

cheers,
You should use all index files to get full sensitivity, but you could run one "bfast match" per index file on the input reads, then use "bmfmerge" (see the "butil" folder) to merge the index results (BMF files).
nilshomer is offline   Reply With Quote
Old 09-28-2011, 10:27 PM   #8
pengchy
Senior Member
 
Location: China

Join Date: Feb 2009
Posts: 116
Default

To parallelize the running, whether I can split the reference genome into several, say 2, sections, and create index separately and then align separately then merge the bmf files?
Thanks.
pengchy is offline   Reply With Quote
Old 09-29-2011, 05:33 AM   #9
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by pengchy View Post
To parallelize the running, whether I can split the reference genome into several, say 2, sections, and create index separately and then align separately then merge the bmf files?
Thanks.
Split the reads, merge the SAM files.
nilshomer is offline   Reply With Quote
Old 05-17-2012, 05:36 PM   #10
rgarcia
Junior Member
 
Location: Mexico City

Join Date: May 2012
Posts: 2
Default are mate pairs split when partitioning reads?

Hello! I'm using this approach to parallelize the alignment of a run of SOLiD mate-pair reads. I converted the reads using solid2fastq, then partitioned the fastq files using the unix split command. I then successfully ran the match step on each partition.

I then realised that solid2fastq placed F3 reads at the beginning of the fastq while R3 at the end, so after spliting the fastq, F3 reads will be separately aligned from R3 reads. It is here suggested that I separately generate a sam file from each partition.

Does this mean that my reads won't be paired? Will samtools or picard pair my reads when I merge my sam files?

Thanks!

Quote:
Originally Posted by nilshomer View Post
Note: "bfast match" searches the index, while "bfast index" creates an index.

Here are the basic steps (remember to use multiple threads with the "-n" option).

1.
Partition the N reads into bins with R reads contained. Run these through "bfast match" to create ~X=N/R BMF files. Alternatively, if you have only one input file, use the "-s/-e" options to specify which reads to process for each invocation of "bfast match".

2.
For each BMF file from step 1, process them with "bfast localalign". You can also use "-s/-e" to further sub-divide each input file. Lets call the number of BAF files Y (since you may sub-divide, otherwise X equals Y).

3.
For each BAF file from step 2, process them with "bfast localalign" to get a SAM file.

4.
After all SAM files have been created, merge them with Picard or samtools (I recommend the former).

This is the best way to partition the various steps of bfast, since step 1 ("bfast match") may take much more memory (for large references) than step 2 ("bfast localalign") and so if you had a heterogenous cluster you could submit the step 1 jobs to high memory nodes and step 2 jobs to low memory nodes etc. This is the spirit of the "bfast.submit.pl" script found with the BFAST distribution. The latter script is not well supported (there be dragons).
rgarcia is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 08:14 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO