Newbie bwa question (HiSeq data)Hi, Sorry for the newbie question! I am trying to c

bsmith030465 · 02-23-2015, 12:41 PM

Hi,

Sorry for the newbie question!

I am trying to convert output from HiSeq whole genome sequence to bam format that will be used by GATK. For any one sample, I have multiple files, e.g.

C12XXZ_s1_1_SL123.fastq.gz
C12XXZ_s1_2_SL123.fastq.gz
.
.
C12XXZ_s8_1_SL123.fastq.gz
C12XXZ_s8_2_SL123.fastq.gz

.
.
A456ZXX_s1_1_SL123.fastq.gz
A456ZXX_s1_2_SL123.fastq.gz
.
.
A456ZXX_s8_1_SL123.fastq.gz
A456ZXX_s8_2_SL123.fastq.gz

Do I need to separately combine each lane forward and reverse reads into a bam file (i.e I will get C12XXZ_s1.bam, C12XXZ_s2.bam,….A456ZXX_s1.bam, A456ZXX_s2.bam) and then just to concatenate all these files to get one bam for the sample?

thanks!

felvis56 · 02-23-2015, 02:14 PM

Hi,

I'm pretty new to this too.

I think you need to unzip the files and do a paired end alignment to a reference to generate a sam file (combining both files). You can then convert to bam file using samtools view and then sort the file with samtools sort.

The samtools manual is available online. I usually use bowtie2 for reference alignments but bwa will handle paired end too.

Good luck.

GenoMax · 02-23-2015, 05:13 PM

Most NGS tools will read/use compressed data files. There is no need to uncompress your data.

If you received your data for each sample in the form of multiple smaller files those can be concatenated into a single file (per sample per read, i.e. R1 and R2, forward and reverse reads) before doing the alignments followed by BAM conversions.

felvis56 · 02-24-2015, 12:57 AM

Thanks GenoMax. I am used to MiSeq which doesn't split files. Could you cat the bam files rather than the fq files? I'm not sure my computer would handle an alignment with such large files.

sarvidsson · 02-24-2015, 01:12 AM

Quote:

Originally Posted by felvis56

Thanks GenoMax. I am used to MiSeq which doesn't split files. Could you cat the bam files rather than the fq files? I'm not sure my computer would handle an alignment with such large files.

You can't really concatenate BAMs as you'd want these coordinate-sorted. You can merge them, however.

You can do the following -
1. merge FASTQs, align, sort BAM
2. align, sort BAMs, merge BAMs

The memory-intensive step will be the BAM sort or BAM merge in either case. With a single machine I'd do 1. With a cluster I'd do 2 as alignment would be parallelized.

bsmith030465 · 02-26-2015, 07:47 AM

Thanks for the replies. I have tried the following steps (see below), but when I try to validate the bam files with picard, I get errors. Am I doing things correctly?

Let me know if you want more details on the first few steps...

1. Align using ‘bwa aln’ using hg19 (output is a .sai file)
2. Combine paired end reads using ‘bwa sampe’ (output is a .sam file)
3. sam to bam using ‘samtools view -bS’ (output is a .bam file)
4. sort bam using ‘samtools sort’ (output is *_sorted.bam file)
5. merge bam using ‘samtools merge’ (output is SL54.bam file, where SL54 is sample id)
6. check format:

% samtools view -H SL54.bam

@HD VN:1.3 SO:coordinate
@SQ SN:chr1:1-249250621 LN:249250621
@SQ SN:chr2:1-243199373 LN:243199373
@SQ SN:chr3:1-198022430 LN:198022430
@SQ SN:chr4:1-191154276 LN:191154276
@SQ SN:chr5:1-180915260 LN:180915260
@SQ SN:chr6:1-171115067 LN:171115067
@SQ SN:chr7:1-159138663 LN:159138663
@SQ SN:chr8:1-146364022 LN:146364022
@SQ SN:chr9:1-141213431 LN:141213431
@SQ SN:chr10:1-135534747 LN:135534747
@SQ SN:chr11:1-135006516 LN:135006516
@SQ SN:chr12:1-133851895 LN:133851895
@SQ SN:chr13:1-115169878 LN:115169878
@SQ SN:chr14:1-107349540 LN:107349540
@SQ SN:chr15:1-102531392 LN:102531392
@SQ SN:chr16:1-90354753 LN:90354753
@SQ SN:chr17:1-81195210 LN:81195210
@SQ SN:chr18:1-78077248 LN:78077248
@SQ SN:chr19:1-59128983 LN:59128983
@SQ SN:chr20:1-63025520 LN:63025520
@SQ SN:chr21:1-48129895 LN:48129895
@SQ SN:chr22:1-51304566 LN:51304566
@SQ SN:chrX:1-155270560 LN:155270560
@SQ SN:chrY:1-59373566 LN:59373566
@SQ SN:chrM:1-16571 LN:16571
@RG ID:C3CG0ACXX_s1_SL54 SM:SL54 PL:illumina LB:C3CG0ACXX PU:GSLv2-7
@RG ID:C3CG0ACXX_s2_SL54 SM:SL54 PL:illumina LB:C3CG0ACXX PU:GSLv2-7

7. Check read group information:

$ % samtools view -H *_merged.bam | grep '^@RG'
@RG ID:C3CG0ACXX_s1_SL54 SM:SL54 PL:illumina LB:C3CG0ACXX PU:GSLv2-7
@RG ID:C3CG0ACXX_s2_SL54 SM:SL54 PL:illumina LB:C3CG0ACXX PU:GSLv2-7

8. Validate bam file using picard

$ java -jar picard.jar ValidateSamFile INPUT=SL54.bam

ERROR: Record 9, Read name HWI-ST354R:507:C3CG0ACXX:2:2112:15621:60548, Mate negative strand flag does not match read negative strand flag of mate
ERROR: Record 42, Read name HWI-ST354R:507:C3CG0ACXX:2:1113:16541:70397, Mate negative strand flag does not match read negative strand flag of mate
ERROR: Record 95, Read name HWI-ST354R:507:C3CG0ACXX:1:2309:4978:73872, Mate negative strand flag does not match read negative strand flag of mate
ERROR: Record 103, Read name HWI-ST354R:507:C3CG0ACXX:1:2114:15765:51586, Mate negative strand flag does not match read negative strand flag of mate
ERROR: Record 121, Read name HWI-ST354R:507:C3CG0ACXX:2:2205:9115:84058, Mate negative strand flag does not match read negative strand flag of mate
ERROR: Record 348, Read name HWI-ST354R:507:C3CG0ACXX:1:2312:3854:62202, Mate negative strand flag does not match read negative strand flag of mate
ERROR: Record 372, Read name HWI-ST354R:507:C3CG0ACXX:1:1212:7328:6583, Mate negative strand flag does not match read negative strand flag of mate

bsmith030465 · 02-26-2015, 09:08 AM

Further:

1. I tried using the picard FixMateInformation function, but that didn't seem to help.
2. Tried 'samtools fixmate '*.bam' before merging the files and that seems to help.

Instead of many error messages (in previous post), I just get 5 or 6:

ERROR: Record 1280018, Read name HWI-ST354R:507:C3CG0ACXX:1:1212:20725:8037, MAPQ should be 0 for unmapped read.
ERROR: Record 1280018, Read name HWI-ST354R:507:C3CG0ACXX:1:1212:20725:8037, bin field of BAM record does not equal value computed based on alignment start and end, and length of sequence to which read is aligned
ERROR: Record 4285715, Read name HWI-ST354R:507:C3CG0ACXX:1:2309:16032:37084, MAPQ should be 0 for unmapped read.
ERROR: Record 4431970, Read name HWI-ST354R:507:C3CG0ACXX:1:2312:18848:30563, MAPQ should be 0 for unmapped read.
ERROR: Record 4431970, Read name HWI-ST354R:507:C3CG0ACXX:1:2312:18848:30563, bin field of BAM record does not equal value computed based on alignment start and end, and length of sequence to which read is aligned

On reading other threads, I gather that these will vanish/be ignored if the 'LENIENT' option is used in the ValidateSamFile function. However, will ignoring these cause problems with downstream analysis?

many thanks for the replies!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Newbie question regarding mapping of RNA-seq data	analog900	RNA Sequencing	12	01-20-2015 08:56 AM
Newbie's Question on align published Chip-Seq data	muhe1985	General	0	01-13-2014 05:46 PM
newbie BWA alignment question	Opusone	Bioinformatics	2	04-04-2013 10:59 AM
How to use BWA - a newbie's question	henrye	Bioinformatics	1	03-12-2012 01:47 PM
Newbie Question Here	Schoenbrau	General	1	12-05-2010 08:20 PM

02-23-2015, 12:41 PM	#1
bsmith030465 Member Location: ca Join Date: Nov 2014 Posts: 11	Newbie bwa question (HiSeq data)Hi, Sorry for the newbie question! I am trying to c Hi, Sorry for the newbie question! I am trying to convert output from HiSeq whole genome sequence to bam format that will be used by GATK. For any one sample, I have multiple files, e.g. C12XXZ_s1_1_SL123.fastq.gz C12XXZ_s1_2_SL123.fastq.gz . . C12XXZ_s8_1_SL123.fastq.gz C12XXZ_s8_2_SL123.fastq.gz . . A456ZXX_s1_1_SL123.fastq.gz A456ZXX_s1_2_SL123.fastq.gz . . A456ZXX_s8_1_SL123.fastq.gz A456ZXX_s8_2_SL123.fastq.gz Do I need to separately combine each lane forward and reverse reads into a bam file (i.e I will get C12XXZ_s1.bam, C12XXZ_s2.bam,….A456ZXX_s1.bam, A456ZXX_s2.bam) and then just to concatenate all these files to get one bam for the sample? thanks!

02-23-2015, 02:14 PM	#2
felvis56 Member Location: Glasgow Join Date: Dec 2012 Posts: 11	Hi, I'm pretty new to this too. I think you need to unzip the files and do a paired end alignment to a reference to generate a sam file (combining both files). You can then convert to bam file using samtools view and then sort the file with samtools sort. The samtools manual is available online. I usually use bowtie2 for reference alignments but bwa will handle paired end too. Good luck.

02-23-2015, 05:13 PM	#3
GenoMax Senior Member Location: East Coast USA Join Date: Feb 2008 Posts: 7,143	Most NGS tools will read/use compressed data files. There is no need to uncompress your data. If you received your data for each sample in the form of multiple smaller files those can be concatenated into a single file (per sample per read, i.e. R1 and R2, forward and reverse reads) before doing the alignments followed by BAM conversions.

02-24-2015, 12:57 AM	#4
felvis56 Member Location: Glasgow Join Date: Dec 2012 Posts: 11	Thanks GenoMax. I am used to MiSeq which doesn't split files. Could you cat the bam files rather than the fq files? I'm not sure my computer would handle an alignment with such large files.

02-26-2015, 07:47 AM	#6
bsmith030465 Member Location: ca Join Date: Nov 2014 Posts: 11	Thanks for the replies. I have tried the following steps (see below), but when I try to validate the bam files with picard, I get errors. Am I doing things correctly? Let me know if you want more details on the first few steps... 1. Align using ‘bwa aln’ using hg19 (output is a .sai file) 2. Combine paired end reads using ‘bwa sampe’ (output is a .sam file) 3. sam to bam using ‘samtools view -bS’ (output is a .bam file) 4. sort bam using ‘samtools sort’ (output is _sorted.bam file) 5. merge bam using ‘samtools merge’ (output is SL54.bam file, where SL54 is sample id) 6. check format: % samtools view -H SL54.bam @HD VN:1.3 SO:coordinate @SQ SN:chr1:1-249250621 LN:249250621 @SQ SN:chr2:1-243199373 LN:243199373 @SQ SN:chr3:1-198022430 LN:198022430 @SQ SN:chr4:1-191154276 LN:191154276 @SQ SN:chr5:1-180915260 LN:180915260 @SQ SN:chr6:1-171115067 LN:171115067 @SQ SN:chr7:1-159138663 LN:159138663 @SQ SN:chr8:1-146364022 LN:146364022 @SQ SN:chr9:1-141213431 LN:141213431 @SQ SN:chr10:1-135534747 LN:135534747 @SQ SN:chr11:1-135006516 LN:135006516 @SQ SN:chr12:1-133851895 LN:133851895 @SQ SN:chr13:1-115169878 LN:115169878 @SQ SN:chr14:1-107349540 LN:107349540 @SQ SN:chr15:1-102531392 LN:102531392 @SQ SN:chr16:1-90354753 LN:90354753 @SQ SN:chr17:1-81195210 LN:81195210 @SQ SN:chr18:1-78077248 LN:78077248 @SQ SN:chr19:1-59128983 LN:59128983 @SQ SN:chr20:1-63025520 LN:63025520 @SQ SN:chr21:1-48129895 LN:48129895 @SQ SN:chr22:1-51304566 LN:51304566 @SQ SN:chrX:1-155270560 LN:155270560 @SQ SN:chrY:1-59373566 LN:59373566 @SQ SN:chrM:1-16571 LN:16571 @RG ID:C3CG0ACXX_s1_SL54 SM:SL54 PL:illumina LB:C3CG0ACXX PU:GSLv2-7 @RG ID:C3CG0ACXX_s2_SL54 SM:SL54 PL:illumina LB:C3CG0ACXX PU:GSLv2-7 7. Check read group information: $ % samtools view -H _merged.bam \| grep '^@RG' @RG ID:C3CG0ACXX_s1_SL54 SM:SL54 PL:illumina LB:C3CG0ACXX PU:GSLv2-7 @RG ID:C3CG0ACXX_s2_SL54 SM:SL54 PL:illumina LB:C3CG0ACXX PU:GSLv2-7 8. Validate bam file using picard $ java -jar picard.jar ValidateSamFile INPUT=SL54.bam ERROR: Record 9, Read name HWI-ST354R:507:C3CG0ACXX:2:2112:15621:60548, Mate negative strand flag does not match read negative strand flag of mate ERROR: Record 42, Read name HWI-ST354R:507:C3CG0ACXX:2:1113:16541:70397, Mate negative strand flag does not match read negative strand flag of mate ERROR: Record 95, Read name HWI-ST354R:507:C3CG0ACXX:1:2309:4978:73872, Mate negative strand flag does not match read negative strand flag of mate ERROR: Record 103, Read name HWI-ST354R:507:C3CG0ACXX:1:2114:15765:51586, Mate negative strand flag does not match read negative strand flag of mate ERROR: Record 121, Read name HWI-ST354R:507:C3CG0ACXX:2:2205:9115:84058, Mate negative strand flag does not match read negative strand flag of mate ERROR: Record 348, Read name HWI-ST354R:507:C3CG0ACXX:1:2312:3854:62202, Mate negative strand flag does not match read negative strand flag of mate ERROR: Record 372, Read name HWI-ST354R:507:C3CG0ACXX:1:1212:7328:6583, Mate negative strand flag does not match read negative strand flag of mate

02-26-2015, 09:08 AM	#7
bsmith030465 Member Location: ca Join Date: Nov 2014 Posts: 11	Further: 1. I tried using the picard FixMateInformation function, but that didn't seem to help. 2. Tried 'samtools fixmate '*.bam' before merging the files and that seems to help. Instead of many error messages (in previous post), I just get 5 or 6: ERROR: Record 1280018, Read name HWI-ST354R:507:C3CG0ACXX:1:1212:20725:8037, MAPQ should be 0 for unmapped read. ERROR: Record 1280018, Read name HWI-ST354R:507:C3CG0ACXX:1:1212:20725:8037, bin field of BAM record does not equal value computed based on alignment start and end, and length of sequence to which read is aligned ERROR: Record 4285715, Read name HWI-ST354R:507:C3CG0ACXX:1:2309:16032:37084, MAPQ should be 0 for unmapped read. ERROR: Record 4431970, Read name HWI-ST354R:507:C3CG0ACXX:1:2312:18848:30563, MAPQ should be 0 for unmapped read. ERROR: Record 4431970, Read name HWI-ST354R:507:C3CG0ACXX:1:2312:18848:30563, bin field of BAM record does not equal value computed based on alignment start and end, and length of sequence to which read is aligned On reading other threads, I gather that these will vanish/be ignored if the 'LENIENT' option is used in the ValidateSamFile function. However, will ignoring these cause problems with downstream analysis? many thanks for the replies!