SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
Building bowtie index with mirBase hairpin.fa file Gators RNA Sequencing 6 05-07-2015 12:43 PM
Building bfast/btestindexes index for 15% divergence feederbing Bioinformatics 1 09-07-2011 04:02 PM
strange bowtie index building and mapping problem Gangcai Bioinformatics 0 08-04-2010 06:02 PM
tophat-bowtie building index repinementer Bioinformatics 1 07-17-2010 11:53 PM
BWA building index of full human (ensembl) fails inijman Bioinformatics 4 12-23-2009 06:00 AM

Reply
 
Thread Tools
Old 01-05-2010, 01:08 PM   #1
bre
Member
 
Location: United States

Join Date: Dec 2009
Posts: 11
Question Building bfast index with btestindexes

The command "bfast index" requires the three parameters -f (input file), -m (mask or spaced seed), and -w (hash width or index's index). For finding a value for -m, there is a separate program called "btestindexes". I'm not absolutely sure which options for btestindexes are required, but it looks like at least -a for an algorithm number, -r for a read length, -S for number of events to sample are required. I'll explain what I know so far.

There are two algorithms in btestindexes. These are selected by either a 0 or a 1 following -a.
Step 1: Finding a set of masks
This step is specified by -a 0. I think required options for this are -s for number of indexes to sample, -l for key size, -w for maximum key width, -n for maximum set size, and -t for accuracy percent threshold.
Step 2: Estimate mask accuracy
This step is specified by -a 1. Requirements for this step are -f for input file name and -I for maximum insertion length

Is there anyone who has built indexes with bfast or used btestindexes that can tell me what these parameters expect? More directly, what is expected for parameters -r, -S, -s, -l, -w, and -n?

Last edited by bre; 01-05-2010 at 01:24 PM.
bre is offline   Reply With Quote
Old 01-05-2010, 03:15 PM   #2
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by bre View Post
The command "bfast index" requires the three parameters -f (input file), -m (mask or spaced seed), and -w (hash width or index's index). For finding a value for -m, there is a separate program called "btestindexes". I'm not absolutely sure which options for btestindexes are required, but it looks like at least -a for an algorithm number, -r for a read length, -S for number of events to sample are required. I'll explain what I know so far.

There are two algorithms in btestindexes. These are selected by either a 0 or a 1 following -a.
Step 1: Finding a set of masks
This step is specified by -a 0. I think required options for this are -s for number of indexes to sample, -l for key size, -w for maximum key width, -n for maximum set size, and -t for accuracy percent threshold.
Step 2: Estimate mask accuracy
This step is specified by -a 1. Requirements for this step are -f for input file name and -I for maximum insertion length

Is there anyone who has built indexes with bfast or used btestindexes that can tell me what these parameters expect? More directly, what is expected for parameters -r, -S, -s, -l, -w, and -n?
Take a look at the manual, it will give recommended settings for Illumina and SOLiD reads. These should be used by default and avoid the set below. If you don't get the expected results, then...

to try to find a set of indexes on your own, check out the BFAST paper supplemental materials.
  • -r expects the read length (say 50bp reads for SOLiD etc)
  • -S is the number of reads to sample when assessing the sensitivity of an index (10,000 gives a good sampling)
  • -s is the number of indexes to sample when searching for the next index to include the set (10 is the number I use for Illumina/SOLiD sequencing)
  • -l is the key size i.e. the # of 1s in the mask
  • -w is the maximum width i.e. the # of 1s and 0s in the mask
  • -n is the maximum # of indexes in the set.
Again, the-l, -w, and -n options are clearly described in the BFAST paper and supplement and relate to the genome size, desired sensitivity and read length.
nilshomer is offline   Reply With Quote
Old 01-11-2010, 08:12 AM   #3
bre
Member
 
Location: United States

Join Date: Dec 2009
Posts: 11
Default

Quote:
Originally Posted by nilshomer View Post
Take a look at the manual, it will give recommended settings for Illumina and SOLiD reads. These should be used by default and avoid the set below. If you don't get the expected results, then...

to try to find a set of indexes on your own, check out the BFAST paper supplemental materials.
  • -r expects the read length (say 50bp reads for SOLiD etc)
  • -S is the number of reads to sample when assessing the sensitivity of an index (10,000 gives a good sampling)
  • -s is the number of indexes to sample when searching for the next index to include the set (10 is the number I use for Illumina/SOLiD sequencing)
  • -l is the key size i.e. the # of 1s in the mask
  • -w is the maximum width i.e. the # of 1s and 0s in the mask
  • -n is the maximum # of indexes in the set.
Again, the-l, -w, and -n options are clearly described in the BFAST paper and supplement and relate to the genome size, desired sensitivity and read length.
I'm sorry, but could you please direct me to the section in the manual which describes the recommended settings?
bre is offline   Reply With Quote
Old 01-11-2010, 08:27 AM   #4
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by bre View Post
I'm sorry, but could you please direct me to the section in the manual which describes the recommended settings?
See section 7.1.
nilshomer is offline   Reply With Quote
Old 01-11-2010, 10:41 AM   #5
bre
Member
 
Location: United States

Join Date: Dec 2009
Posts: 11
Default

I'm still having trouble finding arguments to these parameters. These are Sanger-style reads and here are some data characteristics:

TARGET
  • Number of sequences = 1,168
  • Minimum sequence length = 1,028
  • Maximum sequence length = 63,086,992
  • Average sequence length = 843,760.069349
  • Total number of bases: 985,511,761
  • Number of Ns in file = 18,289,543
  • Longest N stretch = 1,185,460
  • Number of sequences containing N = 771
QUERY
  • Number of sequences = 180,099
  • Minimum sequence length = 69
  • Maximum sequence length = 1,045
  • Average sequence length = 824.930960194
  • Total number of bases: 148,569,241
  • Number of Ns in file = 158,231
  • Longest N stretch = 363
  • Number of sequences containing N = 59,998

Given these, can you recommend arguments to pass to btestindexes? Perhaps bfast may not even be the tool I'm looking for.
bre is offline   Reply With Quote
Old 01-11-2010, 01:35 PM   #6
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by bre View Post
I'm still having trouble finding arguments to these parameters. These are Sanger-style reads and here are some data characteristics:

TARGET
  • Number of sequences = 1,168
  • Minimum sequence length = 1,028
  • Maximum sequence length = 63,086,992
  • Average sequence length = 843,760.069349
  • Total number of bases: 985,511,761
  • Number of Ns in file = 18,289,543
  • Longest N stretch = 1,185,460
  • Number of sequences containing N = 771
QUERY
  • Number of sequences = 180,099
  • Minimum sequence length = 69
  • Maximum sequence length = 1,045
  • Average sequence length = 824.930960194
  • Total number of bases: 148,569,241
  • Number of Ns in file = 158,231
  • Longest N stretch = 363
  • Number of sequences containing N = 59,998

Given these, can you recommend arguments to pass to btestindexes? Perhaps bfast may not even be the tool I'm looking for.
Typical short-read aligners expect reads between 25-150bp in length, and typically each run of the machine produces reads of the same length. Given your reads are between 70-1000bp in length, you may need to split the reads into various read length bins etc. to process with a short-read aligner.

Why not just use the recommended settings in the manual and bypass the 'btestindexes' step?
nilshomer is offline   Reply With Quote
Old 01-27-2010, 01:15 PM   #7
javijevi
Member
 
Location: Spain

Join Date: Jan 2010
Posts: 38
Default

I'm also a little bit confused with the selection of masks to create the indexes for BFAST...

In a previous post, you said: "Why not just use the recommended settings in the manual and bypass the 'btestindexes' step?"

I guess that's only right for datasets similar to those in the manual, that is, human reads, 50 bp-long for SOLiD, etc., isn't?

If so, I guess that they will not work fine for different datasets as, e.g., my data: SOLiD 25 bp-long reads, obtained from messenger RNAs from a plant, where reference genome is a draft genome composed of 3,761 scaffolds with very diverse length, 794 MB long (721 MB if not counting N's), where exons (source of transcripts) are estimated to sum 40 MB.

So, am I forced to use the btestindexes utility to generate my set of masks and related indexes? Or can I use some kind of rationale to select a proper set of masks?
javijevi is offline   Reply With Quote
Old 01-27-2010, 01:25 PM   #8
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by javijevi View Post
I'm also a little bit confused with the selection of masks to create the indexes for BFAST...

In a previous post, you said: "Why not just use the recommended settings in the manual and bypass the 'btestindexes' step?"

I guess that's only right for datasets similar to those in the manual, that is, human reads, 50 bp-long for SOLiD, etc., isn't?

If so, I guess that they will not work fine for different datasets as, e.g., my data: SOLiD 25 bp-long reads, obtained from messenger RNAs from a plant, where reference genome is a draft genome composed of 3,761 scaffolds with very diverse length, 794 MB long (721 MB if not counting N's), where exons (source of transcripts) are estimated to sum 40 MB.

So, am I forced to use the btestindexes utility to generate my set of masks and related indexes? Or can I use some kind of rationale to select a proper set of masks?
The 'btestindexes' utility is a pain to use and does not always return great masks sets. I have found the following masks work well with 25bp reads regardless of genome size:

Code:
1111111111111111
11111110011110011111
111111111001111111
1111000111111111111
111101000110010011111111
1111111111110001111
1111100100111110011111
1111111110011001100111
1100111001110011111111
11110011011110010011111
The only caveat is that BFAST was not designed for short-reads (<30bp) since enumeration and exhaustive strategies (like BWA) are more efficient and sensitive.
nilshomer is offline   Reply With Quote
Old 01-27-2010, 01:57 PM   #9
javijevi
Member
 
Location: Spain

Join Date: Jan 2010
Posts: 38
Default

Quote:
Originally Posted by nilshomer View Post
The only caveat is that BFAST was not designed for short-reads (<30bp) since enumeration and exhaustive strategies (like BWA) are more efficient and sensitive.
Oops... I missed this point... Are you sure? (well, you're the developer, so what a silly question...) But, do you think it is a so important point? Is there such a big difference between 30 and 25 bp-long reads regarding the selection of the tool? Do you think I must use bwa? Does it manage gaps in such a good way as bfast does it (MAQ do not manage gaps in non-paired reads, and AFAIK bwa is the evolution of MAQ)?

I've seen in your bfast paper that, for 25 bp SOLiD reads, bfast, bwa and maq percentage of mapped reads are 66, 75 and 64 %, respectively. But, what about the quality of the mapping (maq do not manage gapping, and gets similar percentage to bfast)?
javijevi is offline   Reply With Quote
Old 01-27-2010, 02:01 PM   #10
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by javijevi View Post
Oops... I missed this point... Are you sure? (well, you're the developer, so what a silly question...) But, do you think it is a so important point? Is there such a big difference between 30 and 25 bp-long reads regarding the selection of the tool? Do you think I must use bwa? Does it manage gaps in such a good way as bfast does it (MAQ do not manage gaps in non-paired reads, and AFAIK bwa is the evolution of MAQ)?


I've seen in your bfast paper that, for 25 bp SOLiD reads, bfast, bwa and maq percentage of mapped reads are 66, 75 and 64 %, respectively. But, what about the quality of the mapping (maq do not manage gapping, and gets similar percentage to bfast)?
I think you will be fine if you use BFAST.
nilshomer is offline   Reply With Quote
Old 01-27-2010, 03:01 PM   #11
javijevi
Member
 
Location: Spain

Join Date: Jan 2010
Posts: 38
Default

Quote:
Originally Posted by nilshomer View Post
I think you will be fine if you use BFAST.
OK, I will try BFAST, but I do not know why do you select this set of masks for 25 bp reads:

- why do they have different lengths (one of them being 24 bp long, that is, only one base shorter than reads?
- why some of them have so many zeroes? For instance, if I understood well, the mask 111101000110010011111111 is supposed to allow alignment when there are so many differences as, e.g., one SNP, followed by a base match, followed by three consecutive base mismatches (or an indel of three bases), followed by two base matches, followed by two consecutive base mismatches (or an indel of two bases), followed by one base match, followed by two consecutive base mismatches (or an indel of two bases), isn't? Or maybe I'm wrong...
- are not the two following masks exactly the same?: 1111000111111111111 and 1111111111110001111
- shouldn't there be a mask allowing for a single mismatch?

I know there are too many questions, so that there is the possibility that I'm absolutely wrong in my suppositions about masks and indexes...
javijevi is offline   Reply With Quote
Old 01-27-2010, 03:10 PM   #12
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by javijevi View Post
OK, I will try BFAST, but I do not know why do you select this set of masks for 25 bp reads:

- why do they have different lengths (one of them being 24 bp long, that is, only one base shorter than reads?
- why some of them have so many zeroes? For instance, if I understood well, the mask 111101000110010011111111 is supposed to allow alignment when there are so many differences as, e.g., one SNP, followed by a base match, followed by three consecutive base mismatches (or an indel of three bases), followed by two base matches, followed by two consecutive base mismatches (or an indel of two bases), followed by one base match, followed by two consecutive base mismatches (or an indel of two bases), isn't? Or maybe I'm wrong...
- are not the two following masks exactly the same?: 1111000111111111111 and 1111111111110001111
- shouldn't there be a mask allowing for a single mismatch?

I know there are too many questions, so that there is the possibility that I'm absolutely wrong in my suppositions about masks and indexes...
Remember, these masks are applied across the read, and they are designed to compliment each other. There is a wide array of literature on finding spaced seeds. Suppose you knew where the errors and SNPs occurred within the read, the masks are designed such that at least one mask when applied at a least one offset from the start of the read does not have '1' that matches the SNP (two color changes) or error (one color change). I would recommend reading the BFAST manuscript, especially the supplemental materials if you are interested in a more in-depth description http://dx.doi.org/10.1371/journal.pone.0007767.
nilshomer is offline   Reply With Quote
Old 01-28-2010, 10:19 AM   #13
javijevi
Member
 
Location: Spain

Join Date: Jan 2010
Posts: 38
Default

Quote:
Originally Posted by nilshomer View Post
Remember, these masks are applied across the read, and they are designed to compliment each other. There is a wide array of literature on finding spaced seeds. Suppose you knew where the errors and SNPs occurred within the read, the masks are designed such that at least one mask when applied at a least one offset from the start of the read does not have '1' that matches the SNP (two color changes) or error (one color change). I would recommend reading the BFAST manuscript, especially the supplemental materials if you are interested in a more in-depth description http://dx.doi.org/10.1371/journal.pone.0007767.
I strongly apologize for having missed that great supplementary material in the BFAST manuscript. I thought it was just data results about tools comparisons and simulations, and not that deep and informative description of the problem and the solution offered by BFAST. It has been very illustrative for a biologist like me.

In that supplementary material can be read that "in the BFAST distribution, we provide greedy sets containing S=40 masks, as well as the accuracy tables for using just the first s=1, s=2, s=3, , so that one can conveniently choose a subset with any desired accuracy level, and which is furthermore already optimized relative to that accuracy specification", and also "BFAST distribution includes greedy search optimized mask sets for reads from L=20 to L=100, and up to S=40 masks long, with associated accuracy tables for all choices of fewer masks s < S, so that it is convenient to choose a mask set for the read length and accuracy levels of interest for any specific alignment problem".

However, I cannot find that index sets in the BFAST distribution I downloaded on Jan/2010, version 0.6.2a

Can anybody show me where do I have to look for them?, or Is there any other way of getting them?
javijevi is offline   Reply With Quote
Old 01-28-2010, 04:42 PM   #14
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by javijevi View Post
I strongly apologize for having missed that great supplementary material in the BFAST manuscript. I thought it was just data results about tools comparisons and simulations, and not that deep and informative description of the problem and the solution offered by BFAST. It has been very illustrative for a biologist like me.

In that supplementary material can be read that "in the BFAST distribution, we provide greedy sets containing S=40 masks, as well as the accuracy tables for using just the first s=1, s=2, s=3, , so that one can conveniently choose a subset with any desired accuracy level, and which is furthermore already optimized relative to that accuracy specification", and also "BFAST distribution includes greedy search optimized mask sets for reads from L=20 to L=100, and up to S=40 masks long, with associated accuracy tables for all choices of fewer masks s < S, so that it is convenient to choose a mask set for the read length and accuracy levels of interest for any specific alignment problem".

However, I cannot find that index sets in the BFAST distribution I downloaded on Jan/2010, version 0.6.2a

Can anybody show me where do I have to look for them?, or Is there any other way of getting them?
They are not up, but can be generated by btestindexes. Use the recommended settings!
nilshomer is offline   Reply With Quote
Old 02-02-2010, 11:12 PM   #15
saberdsl
Junior Member
 
Location: nanjing,china

Join Date: Jan 2010
Posts: 1
Default

hello nilshomer,can you tell me the masks work well with 35bp solid reads
saberdsl is offline   Reply With Quote
Old 02-03-2010, 07:52 AM   #16
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by saberdsl View Post
hello nilshomer,can you tell me the masks work well with 35bp solid reads
Use the recommended indexes for 25bp reads found in the manual. I have found that those work well with 35bp reads.
nilshomer is offline   Reply With Quote
Old 02-03-2010, 08:07 AM   #17
drio
Senior Member
 
Location: 4117'49"N / 24'42"E

Join Date: Oct 2008
Posts: 323
Default

Quote:
Originally Posted by saberdsl View Post
hello nilshomer,can you tell me the masks work well with 35bp solid reads
I have gotten good results with the indexes recommend in the BF book (section 7.1.2 -- yes for 50bp reads). Feel free to remove the indexes that use masks > than you read length.

Let us know how it goes.
__________________
-drd
drio is offline   Reply With Quote
Old 05-18-2010, 03:18 AM   #18
KevinLam
Senior Member
 
Location: SEA

Join Date: Nov 2009
Posts: 203
Default

Does anyone have recommended masks for mouse mm9 solid 50 bp reads?
KevinLam is offline   Reply With Quote
Old 05-18-2010, 09:16 AM   #19
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Quote:
Originally Posted by KevinLam View Post
Does anyone have recommended masks for mouse mm9 solid 50 bp reads?
Use the recommended indexes for hg18 for 50bp reads.
nilshomer is offline   Reply With Quote
Old 05-30-2010, 02:50 PM   #20
elinor
Junior Member
 
Location: Santa Cruz, CA

Join Date: Feb 2010
Posts: 2
Default

I have question regarding the "masks" in the index. What are the masks and why do we need them? Thanks for the response!
elinor is offline   Reply With Quote
Reply

Tags
bfast, btestindexes, indexes

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 01:03 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO