SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with BWA mapping of Illumina PE short insert size fragments (FFPE material) LadyGray Bioinformatics 2 10-22-2012 01:20 AM
BWA align SOLiD PE data gives poor mapping & 0.5% properly paired alig Bioinformatics 3 07-08-2011 08:44 AM
BWA behaviour with Mate Pair data + Multi read mapping apratap Bioinformatics 5 06-23-2011 04:38 PM
Illumina mapping with bwa bpetersen Bioinformatics 5 03-14-2011 11:04 PM
BWA mapping fastq files with Illumina quality maricu Bioinformatics 3 11-19-2010 11:18 AM

Reply
 
Thread Tools
Old 09-02-2010, 01:41 PM   #1
nirav99
Member
 
Location: Houston, TX

Join Date: Aug 2010
Posts: 10
Smile Error found while mapping Illumina data using BWA

While mapping Illumina data against BWA, I noticed several errors :

ERROR: Record 58, Read name HWI-ST142_0217:1:63:8795:27966#0, Mate negative strand flag does not match read negative strand flag of mate

The corresponding reads are

HWI-ST142_0217:1:63:8795:27966#0 89 chr1 27 0 52M1D48M = 27 0 ACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCTAACCCGAACCCAACCCTAACCCTAACCCTAACCCTAACCCTACCCTAACCCTAACCCTA BBBBBBB___T^^ddc^bccddYdaa``]^O_U]\bGbbbZ_PZ\ZPT\Zbdaadeeee`ddddd`caccb\bbbbb\bbdcTdddcadddd`dadcdcdXT:A:R NM:i:3 SM:i:0 AM:i:0 X0:i:3 X1:i:1 XM:i:2 XO:i:1 XG:i:1 MD:Z:46T5^T29A18 XA:Z:chr15,+100338770,17M1D83M,3;chr4,-62,82M1D18M,3;chr1,-21,52M1D48M,4;

HWI-ST142_0217:1:63:8795:27966#0 181 chr1 27 0 * = 27 0 GGGTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTTAGGGTGGTTAGGG bT\^`^bddad\bdadbbdd``Yc`b\d`dbdd^d`ccc\`bYb`TUXYPeT\eecfdfdeeccfcefcfe\eeccfefeffeffffffffffcfcfeff

The flag of these reads show up as

1 0 1 1 0 0 1 - first read (forward strand of mate, reverse strand of query)
1 0 1 1 0 1 0 1 - second read (reverse strand of mate, reverse strand of query)

The sixth bit (from the left) for both these reads (mate strand) conflicts with the fifth bit (strand of the query).

Is this the reason for the error I am seeing ? Is there a way to prevent such errors from occurring ?

Last edited by nirav99; 09-02-2010 at 01:47 PM. Reason: Adding new line for clarity of reading
nirav99 is offline   Reply With Quote
Old 12-07-2010, 11:13 AM   #2
nirav99
Member
 
Location: Houston, TX

Join Date: Aug 2010
Posts: 10
Default

Picard's command line utility FixMateInformation fixes this error.
nirav99 is offline   Reply With Quote
Old 12-11-2010, 07:15 PM   #3
cliff
Member
 
Location: USA

Join Date: Oct 2009
Posts: 41
Default

hi, nirav99

I got the same error "Mate negative strand flag does not match read negative strand flag of mate" and I tried FixMateInformation as:

java -Xmx2g -jar FixMateInformation.jar INPUT=test.bam OUTPUT=test_fixed.bam VALIDATION_STRINGENCY=SILENT

but I didn't get the output file and got this error:
Exception in thread "main" net.sf.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; streamed file (filename not available)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:199)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:189)
at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:120)
at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:37)
at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:185)
at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
at net.sf.picard.sam.FixMateInformation.doWork(FixMateInformation.java:145)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:156)
at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:117)
at net.sf.picard.sam.FixMateInformation.main(FixMateInformation.java:74)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:197)


....
cliff is offline   Reply With Quote
Old 12-13-2010, 07:29 AM   #4
nirav99
Member
 
Location: Houston, TX

Join Date: Aug 2010
Posts: 10
Default

Hi Cliff,

Please check the disk space where you are running this. The exception indicates

"Caused by: java.io.IOException: No space left on device"
nirav99 is offline   Reply With Quote
Old 12-13-2010, 07:31 AM   #5
cliff
Member
 
Location: USA

Join Date: Oct 2009
Posts: 41
Default

thanks for getting back to me. I actually have 15TB left..
cliff is offline   Reply With Quote
Old 12-13-2010, 10:09 AM   #6
nilshomer
Nils Homer
 
nilshomer's Avatar
 
Location: Boston, MA, USA

Join Date: Nov 2008
Posts: 1,285
Default

Could it be that the temporary path is not pointing towards your 15TB?
nilshomer is offline   Reply With Quote
Old 12-13-2010, 11:43 AM   #7
drio
Senior Member
 
Location: 4117'49"N / 24'42"E

Join Date: Oct 2008
Posts: 323
Default

The scratching happens in /tmp/<user_name>, so the problem was most likely /tmp. Use TMP_DIR to specify a different temporary dir.
__________________
-drd
drio is offline   Reply With Quote
Old 12-15-2010, 06:25 AM   #8
cliff
Member
 
Location: USA

Join Date: Oct 2009
Posts: 41
Default

I specified a temporary directory which has enough space by

java -Xmx4g -Djava.io.tmpdir=/home/temp -jar

it still failed...
cliff is offline   Reply With Quote
Old 12-15-2010, 06:29 AM   #9
nirav99
Member
 
Location: Houston, TX

Join Date: Aug 2010
Posts: 10
Default

Hi Cliff,

TMP_DIR is a command line parameter for Picard tools.

So, the way to do it would be

java -Xmx4G -jar FixMateInformation.jar I=input.bam O=output.bam TMP_DIR=/home/temp VALIDATION_STRINGENCY=LENIENT
nirav99 is offline   Reply With Quote
Old 12-15-2010, 12:37 PM   #10
cliff
Member
 
Location: USA

Join Date: Oct 2009
Posts: 41
Default

Hi, Nirav99

Thanks! I tried your command and still failed..
cliff is offline   Reply With Quote
Old 04-25-2011, 08:54 AM   #11
Jeremy37
Member
 
Location: Montreal, Canada

Join Date: Feb 2011
Posts: 17
Default Similar error found when using SortSam

Hi Cliff. Did you find a solution to your problem? I'm getting the same error, except for me it occurs when calling SortSam. Similar to your case we have lots of space available (62 TB).

java -Xmx20g -Xms8g -jar SortSam.jar MAX_RECORDS_IN_RAM=2000000 VALIDATION_STRINGENCY=LENIENT CREATE_INDEX=true SORT_ORDER=coordinate INPUT=sequence.sam OUTPUT=sequence.sorted.bam


Runtime.totalMemory()=17176592384
Exception in thread "main" net.sf.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; streamed file (filename not available)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:199)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:189)
at net.sf.samtools.util.BinaryCodec.writeString(BinaryCodec.java:285)
at net.sf.samtools.BinaryTagCodec.writeTag(BinaryTagCodec.java:168)
at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:144)
at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:37)
at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:201)
at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
at net.sf.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:157)
at net.sf.picard.sam.SortSam.doWork(SortSam.java:67)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:157)
at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:117)
at net.sf.picard.sam.SortSam.main(SortSam.java:79)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:297)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:197)
... 12 more



Quote:
Originally Posted by cliff View Post
Hi, Nirav99

Thanks! I tried your command and still failed..
Jeremy37 is offline   Reply With Quote
Old 10-04-2012, 01:14 PM   #12
rahilsethi
Member
 
Location: Pittsburgh, PA

Join Date: May 2010
Posts: 22
Unhappy Picard java exception

Hi I got similar error using Picard MergeSamFiles.jar
Wondering if anyone found an answer for it

Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /scratch/temp/sortingcollection.4193001834577553277.tmp (Too many open files)
at net.sf.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:445)
at net.sf.samtools.util.SortingCollection$MergingIterator.<init>(SortingCollection.java:384)
at net.sf.samtools.util.SortingCollection.iterator(SortingCollection.java:254)
at net.sf.samtools.util.SortingCollection.iterator(SortingCollection.java:43)
at net.sf.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:190)
at net.sf.samtools.AsyncSAMFileWriter.synchronouslyClose(AsyncSAMFileWriter.java:42)
at net.sf.samtools.util.AbstractAsyncWriter.close(AbstractAsyncWriter.java:78)
at net.sf.picard.sam.MergeSamFiles.doWork(MergeSamFiles.java:154)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:177)
at net.sf.picard.sam.MergeSamFiles.main(MergeSamFiles.java:79)
Caused by: java.io.FileNotFoundException: /scratch/temp/sortingcollection.4193001834577553277.tmp (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at net.sf.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:439)
... 9 more
Even in my case the space available in /scratch is around 40T
rahilsethi is offline   Reply With Quote
Old 10-22-2012, 09:53 AM   #13
MolecularToast
Junior Member
 
Location: Iowa

Join Date: Aug 2010
Posts: 9
Default

If anyone is still listening to this thread - what versions of bwa is everyone using who is getting this error?
MolecularToast is offline   Reply With Quote
Old 10-22-2012, 10:20 AM   #14
rahilsethi
Member
 
Location: Pittsburgh, PA

Join Date: May 2010
Posts: 22
Default

I did not use bwa. I used SHRiMP 2.2.2 for mapping. samtools 1.8 for converting sam to bam and Picard version: 1.74 for combining mapped bam files.
rahilsethi is offline   Reply With Quote
Old 10-23-2012, 10:44 AM   #15
jflowers
Member
 
Location: New York, NY

Join Date: Oct 2011
Posts: 41
Default

Regarding the java.io.FileNotFoundException: (Too many open files), there are a number of things to do. Picard sorting seems to open so many files that it can overload default limits on unix systems. To deal with this, you can either (1) ask you sys admin to increase the allowable number of open files using ulimit -n or (2) for some picard programs you can increase the value of the MAX_RECORDS_IN_RAM command-line parameter which instructs Picard to store more records in fewer files and reduce the number of open files. (watch out for increased memory usage though). If your using markdups and see this error you can 3) use the command-line para
meter MAX_FILE_HANDLES_FOR_READ_ENDS_MAP. By reducing this number, you reduce the number of concurrently open files.

On our hpc, ulimit -n == 4096, so I use MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=4000
jflowers is offline   Reply With Quote
Old 01-14-2013, 06:30 PM   #16
qh2012
Junior Member
 
Location: china

Join Date: Jan 2013
Posts: 1
Default

Hi Cliff. Did you find a solution to your problem? I'm getting the same error.
qh2012 is offline   Reply With Quote
Reply

Tags
bam, bwa, illumina

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 06:37 PM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO