SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
how to convert general fastq to fastq int format? feng Bioinformatics 19 08-13-2013 09:25 AM
BGZF ERROR: BEDtools on SPARC with Solaris 10 MILLHILL Bioinformatics 0 03-08-2011 07:05 AM
Bedtools : BGZF ERROR sgaa201 Introductions 0 03-04-2011 05:48 AM
BAM, Zlib and BGZF library xinwu Bioinformatics 4 09-10-2010 12:06 AM
qseq files versus sequence.txt files drio Illumina/Solexa 3 11-09-2009 09:02 AM

Reply
 
Thread Tools
Old 11-08-2011, 06:36 AM   #1
maubp
Peter (Biopython etc)
 
Location: Dundee, Scotland, UK

Join Date: Jul 2009
Posts: 1,303
Default Using BGZF (Blocked GNU Zip Format) for general sequence files

BAM files are compressed using a variant of GZIP (GNU ZIP), called BGZF (Blocked GNU Zip Format). Anyone who has read the SAM/BAM Specification will have seen the terms BGZF and virtual offsets, but what you may not realise is how general purpose this is for random access sections of any large compressed file.

http://blastedbio.blogspot.com/2011/...tter-gzip.html

I wrote the above blog post looking at BGZF applied to FASTA, SwissProt and UniProt-XML sequences. In short: BGZF files are bigger than GZIP files, but they are much faster for random access.

So, should we all be considering using BGZF in preference to GZIP?
maubp is offline   Reply With Quote
Old 11-23-2011, 05:38 AM   #2
maubp
Peter (Biopython etc)
 
Location: Dundee, Scotland, UK

Join Date: Jul 2009
Posts: 1,303
Default

I haven't done a proof of principle implementation, but I believe efficient random access to BZIP2 files is also possible using their block structure. However, BZIP2 decompression is much more CPU intensive which would be a concern for fast random access:
http://blastedbio.blogspot.com/2011/...-to-bzip2.html
maubp is offline   Reply With Quote
Old 11-23-2011, 06:40 AM   #3
arolfe
Member
 
Location: 02119

Join Date: Jul 2011
Posts: 29
Default

What are the use cases where you see BGZF as useful for fasta or fastq files? It seems to only help in the random access case, eg "database" of genome sequences, but wouldn't make much difference over GZIP for files that are generally processed sequentially, eg sequencing reads.

Also, OOC, what's your preferred fasta/fastq index format?
arolfe is offline   Reply With Quote
Old 11-23-2011, 06:53 AM   #4
maubp
Peter (Biopython etc)
 
Location: Dundee, Scotland, UK

Join Date: Jul 2009
Posts: 1,303
Default

Quote:
Originally Posted by arolfe View Post
What are the use cases where you see BGZF as useful for fasta or fastq files? It seems to only help in the random access case, eg "database" of genome sequences, but wouldn't make much difference over GZIP for files that are generally processed sequentially, eg sequencing reads.
Yes, exactly - I see BGZF as being useful for databases of sequences (e.g. FASTA, SwissProt, GenBank, etc) where you need random access.

Where you just need sequential access, you can treat BGZF like GZIP and pipe the decompressed data to a tool, or otherwise decompress on the fly.

So BGZF works for both, and doesn't take that much more space than traditional GZIP (depending on the file format).

Quote:
Originally Posted by arolfe View Post
Also, OOC, what's your preferred fasta/fastq index format?
In terms of indexing large FASTA/FASTQ, I mainly use an SQLite database mapping identifiers to file offets (and raw data length), via Biopython.

For FASTA/FASTQ raw reads, random access by ID is not such a common need, but again BGZF could be used here for random access to a compressed file. Rather I advocate moving to unaligned BAM for raw reads, see http://blastedbio.blogspot.com/2011/...ve-sambam.html and this thread http://seqanswers.com/forums/showthread.php?t=14941
maubp is offline   Reply With Quote
Old 08-19-2012, 11:21 AM   #5
salturki
Member
 
Location: Cambridge, UK

Join Date: May 2008
Posts: 11
Thumbs up

Thanks maubp for the insightful article.

I am wondering if you have considered implementing tabix in Biopython as well?

I am looking for a pure python tabix-like module to be used in a cross-platform solution. I tried to install Pysam and the tabix's python package (shipped with its source code) but couldn't build them on Windows.

I ended up compiling tabix/bgzip on windows using cygwin. The final software is shipped with few dll files from cygwin in order for bgzip and tabix to work.

If you are not planning to write such module, I would appreciate any pseudocode suggestions.

Cheers
salturki is offline   Reply With Quote
Old 08-19-2012, 02:46 PM   #6
maubp
Peter (Biopython etc)
 
Location: Dundee, Scotland, UK

Join Date: Jul 2009
Posts: 1,303
Default

I'm interested in tabix, and it should be possible to implement in Python building on the BGZF support included in Biopython 1.60 - but I've not had time to look into it.
maubp is offline   Reply With Quote
Old 02-08-2013, 01:56 PM   #7
SamH
Member
 
Location: Moscow, ID

Join Date: Sep 2010
Posts: 14
Default using bgzf for BAM files?

Hi All,
I wonder, has anyone tried using the Biopython bgzf support for parsing BAM files?
Specifically it would be nice to access a BAM line by line, however it doesn't seem to quite work correctly. The data comes out garbled for me:

i.e.
Code:
from Bio import bgzf
iter = bgzf.BgzfReader("454.local.bowtie2.bam", 'rb')

for i in range(20):
    print iter.readline()


BAM�@HD VN:1.0  SO:unsorted
@SQ     SN:chrI LN:228539
@SQ     SN:chrII        LN:813067
@SQ     SN:chrIII       LN:316396
@SQ     SN:chrIV        LN:1527223
@SQ     SN:chrV LN:572028
@SQ     SN:chrVI        LN:269964
@SQ     SN:chrVII       LN:1084769
@SQ     SN:chrVIII      LN:562680
@SQ     SN:chrIX        LN:440022
@SQ     SN:chrX LN:744918
@SQ     SN:chrXI        LN:666321
@SQ     SN:chrXII       LN:1073453
@SQ     SN:chrXIII      LN:922770
@SQ     SN:chrXIV       LN:778492
@SQ     SN:chrXV        LN:1091177
@SQ     SN:chrXVI       LN:945865
@SQ     SN:contig00341  LN:6318
@PG     ID:bowtie2      PN:bowtie2      VN:2.0.0-beta7
chrI�|chrII
           h
            chrIII��chrIV�MchrV|chrVI�chrVIIachrVIII�chrIXֶchrX�]
                                                                chrXI�*
thanks!

Sam
SamH is offline   Reply With Quote
Old 02-08-2013, 02:33 PM   #8
maubp
Peter (Biopython etc)
 
Location: Dundee, Scotland, UK

Join Date: Jul 2009
Posts: 1,303
Default

Quote:
Originally Posted by SamH View Post
Hi All,
I wonder, has anyone tried using the Biopython bgzf support for parsing BAM files?
Specifically it would be nice to access a BAM line by line, however it doesn't seem to quite work correctly. The data comes out garbled for me:
Hi Sam,

That behaviour is expected and correct. BGZF is just a variant of gzip, once you decompress that you have a 'naked' BAM file which is a binary representation of the SAM format - although as you noticed it does contain an embedded plain text SAM header. All the Biopython Bio.bgzf code did for you was decompress it. Biopython doesn't currently have a BAM parser.

Have you looked at pysam which is a Python wrapper for the samtools
C API? http://code.google.com/p/pysam/

Peter
maubp is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 04:40 PM.


Powered by vBulletin® Version 3.8.6
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.