SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
Maximum fragment size matth431 Illumina/Solexa 10 05-20-2014 04:59 AM
Variable amplicon size for MiSeq, bias? Maximum size? ShellfishGene Illumina/Solexa 2 02-12-2014 06:47 PM
Zorro maximum size flobpf Bioinformatics 1 11-13-2011 08:39 PM
maximum size sonicated DNA? avilella Sample Prep / Library Generation 0 07-20-2011 02:31 AM
Maximum database size for BWA robs Bioinformatics 0 06-24-2010 07:30 PM

Reply
 
Thread Tools
Old 03-09-2014, 04:12 PM   #1
dvanic
Member
 
Location: Sydney, Australia

Join Date: Jan 2012
Posts: 61
Default HTSeq 0.6.1 Maximum alignment buffer size exceeded error

Hi! I'm usg HTSeq v 0.6.1 to count reads mapping to features (the htseq-count script). I'm getting a

Code:
18700000 SAM alignment record pairs processed.
Error occured when processing SAM input (record #40455805 in file /scratch/me/accepted_hits.bam):
  Maximum alignment buffer size exceeded while pairing SAM alignments.
  [Exception type: ValueError, raised in __init__.py:671]
My accepted_hits.bam file is generated by mapping with STAR and then coordinate-sorted and converted to BAM using samtools.

I am using the htseq command
Code:
htseq-count --format=bam --order=pos --mode=union --idattr=gene_id --stranded=reverse --type=exon} $WORKDIR/accepted_hits.bam gencode.v19.gtf
What is wrong and how do I fix it? (Apart from running htseq on the sam file)?

Thanks in advance,
Darya
dvanic is offline   Reply With Quote
Old 03-10-2014, 04:24 PM   #2
Simon Anders
Senior Member
 
Location: Heidelberg, Germany

Join Date: Feb 2010
Posts: 994
Default

The "--order=pos" option is new and may need a bit of fine-tuning. Perhaps the buffer is a bit small.

For now, maybe sort the file be read name (with "samtools sort -n") and then use "--order=name".
Simon Anders is offline   Reply With Quote
Old 03-13-2014, 06:43 AM   #3
markf
Junior Member
 
Location: Belgium

Join Date: Sep 2009
Posts: 3
Default

I can confirm this happening:

Code:
...
37400000 SAM alignment record pairs processed.
37500000 SAM alignment record pairs processed.
Error occured when processing SAM input (line 78096327):
  Maximum alignment buffer size exceeded while pairing SAM alignments.
  [Exception type: ValueError, raised in __init__.py:671
markf is offline   Reply With Quote
Old 03-16-2014, 11:29 PM   #4
colindaven
Senior Member
 
Location: Germany

Join Date: Oct 2008
Posts: 415
Default

Same here with
a) BAM ordered by position
b) STAR alignments
colindaven is offline   Reply With Quote
Old 03-19-2014, 08:47 AM   #5
Bukowski
Senior Member
 
Location: UK

Join Date: Jan 2010
Posts: 390
Default

Have hit the same problem with STAR aligner, and positional ordering.

Quote:
12200000 SAM alignment record pairs processed.
Error occured when processing SAM input (record #27408270 in file alignments/S6.bam):
Maximum alignment buffer size exceeded while pairing SAM alignments.
[Exception type: ValueError, raised in __init__.py:671]

Last edited by Bukowski; 03-19-2014 at 08:51 AM.
Bukowski is offline   Reply With Quote
Old 04-16-2014, 07:04 AM   #6
mehaffeymg
Junior Member
 
Location: Bethesda, MD

Join Date: Jul 2009
Posts: 2
Default

I get the same "Maximum alignment buffer size exceeded" whether running my bam file as position or name sorted. It gets farther with the name sorted file than the position sorted file (output size of 1 Gb vs. 600+ Mb respectively) but still fails. My input bam is 11Gb of around 142 million reads.
mehaffeymg is offline   Reply With Quote
Old 04-16-2014, 08:01 AM   #7
Simon Anders
Senior Member
 
Location: Heidelberg, Germany

Join Date: Feb 2010
Posts: 994
Default

When you used the name-sorted file, have you remembered to omit the "-r pos"?
Simon Anders is offline   Reply With Quote
Old 04-17-2014, 07:42 AM   #8
mehaffeymg
Junior Member
 
Location: Bethesda, MD

Join Date: Jul 2009
Posts: 2
Default Working now

Quote:
Originally Posted by Simon Anders View Post
When you used the name-sorted file, have you remembered to omit the "-r pos"?
Excellent catch. I reran without that flag and it worked just fine.

Thank you!
mehaffeymg is offline   Reply With Quote
Old 05-13-2014, 06:54 AM   #9
sidderb
Member
 
Location: Cambridge UK

Join Date: Aug 2011
Posts: 14
Default

As Simon reads these I just wanted to flag up that I have run into the same issue with a position sorted alignment from STAR as well.

I was so excited to see this feature in 0.6 as the position to name based ordering is an annoyingly long step in my workflow! Fingers crossed for a fix soon...
__________________
@sidderb
sidderb is offline   Reply With Quote
Old 05-22-2014, 11:04 AM   #10
dakl
Member
 
Location: sweden

Join Date: May 2009
Posts: 15
Default

I get the same with reads aligned with star, and position sorted using samtools.

Code:
tools/samtools view /home/daniel.klevebring/cust001/tests/fullRNA/L8760/rnaseq/L8760T/L8760T.star.50.stranded.PE.bam 
| htseq-count  --stranded=reverse --mode=intersection-nonempty  --order=pos - /mnt/hds/proj/cust001/autoseq_genome/genes/genes_fixed.gtf > /home/daniel.kleveb
ring/cust001/tests/fullRNA/L8760/rnaseq/L8760T/L8760T.star.50.stranded.PE.htseq-count.txt

...
18900000 SAM alignment record pairs processed.
Error occured when processing SAM input (line 40808946):
  Maximum alignment buffer size exceeded while pairing SAM alignments.
  [Exception type: ValueError, raised in __init__.py:671] 

$ htseq-count |grep version
Public License v3. Part of the 'HTSeq' framework, version 0.6.1p1.
I don't always get it, and it runs with larger files without a problem. What's the best fix for this? Is there one?
dakl is offline   Reply With Quote
Old 08-13-2014, 12:10 PM   #11
catbus
Member
 
Location: San Francisco

Join Date: Feb 2011
Posts: 21
Default

It appears that this problem is not addressed in 0.6.1p2 either.

It would be extremely nice if there were a way to use htseq-count on positionally-sorted data (which is the default Tophat output). Maybe there could be an option that htseq-count could use to just increase the buffer size? I have 48 GB of RAM on this machine, so it is not possible that htseq-count could actually consume it all on one file.
catbus is offline   Reply With Quote
Old 11-17-2014, 07:24 AM   #12
golharam
Member
 
Location: Philadelphia, PA

Join Date: Dec 2009
Posts: 55
Default

I get this problem as well using tophat2 aligned BAM files, position sorted. I will revert to name sorting.
golharam is offline   Reply With Quote
Old 02-27-2015, 04:56 AM   #13
dc3000
Junior Member
 
Location: midwest, USA

Join Date: Nov 2010
Posts: 7
Default

Is there a way to increase the buffer size?

I got this error with a pos-sorted.bam file (3.2GB) created with tophat2

Code:
[xxx@dev-intel14-phi 20131017mRNA_gfCC]$ htseq-count --format=bam --stranded=no --order=pos \
> ~/RTSF/hiseq/20131017mRNA_gfCC/ms12cc2/accepted_hits.bam \
> ~/RTSF/hiseq/20131017mRNA_gfCC/genes.gtf \
> > ~/RTSF/hiseq/20131017mRNA_gfCC/htseq/ms12cc2.txt
100000 GFF lines processed.
...
16100000 SAM alignment record pairs processed.
16200000 SAM alignment record pairs processed.
Error occured when processing SAM input (record #35593187 in file /mnt/home/xxx/RTSF/hiseq/20131017mRNA_gfCC/ms12cc2/accepted_hits.bam):
  Maximum alignment buffer size exceeded while pairing SAM alignments.
  [Exception type: ValueError, raised in __init__.py:671]
[xxx@dev-intel14-phi 20131017mRNA_gfCC]$ Write failed: Broken pipe
dc3000 is offline   Reply With Quote
Old 02-27-2015, 10:48 AM   #14
catbus
Member
 
Location: San Francisco

Join Date: Feb 2011
Posts: 21
Default

Two options:

You can either re-sort your files lexographically/alphabetically and run htseq-counts with the lexigraphic order (in other words, NOT --order=pos)

or

You can switch to the much faster and generally more user friendly (easier options, easier output file format) program "featureCounts". Part of the "subread" package. http://subread.sourceforge.net/

I would switch to subread. htseq-counts is incredibly slow and very poor at handling large files without crashing.


[QUOTE=dc3000;161213]Is there a way to increase the buffer size?

I got this error with a pos-sorted.bam file (3.2GB) created with tophat2

[CODE]
[xxx@dev-intel14-phi 20131017mRNA_gfCC]$ htseq-count --format=bam --stranded=no --order=pos \
catbus is offline   Reply With Quote
Old 02-27-2015, 10:49 AM   #15
catbus
Member
 
Location: San Francisco

Join Date: Feb 2011
Posts: 21
Default

P.S. subread only takes about 10 minutes to install. It doesn't have any complicated dependencies. Once again: http://subread.sourceforge.net/
catbus is offline   Reply With Quote
Old 05-12-2015, 08:51 AM   #16
SrCardgage
my other car is a limozeen
 
Location: New Haven, CT, USA

Join Date: Feb 2012
Posts: 23
Default disheartening

It's rather disheartening that this has been a known issue for more than a year and there is still no fix.

I just switched to DESeq from Cufflinks and am learning my THIRD method of generating the counts matrix to feed into DESeq.

I'm beginning to wonder if any published bioinfo tools are actually usable in the real world.

SrCardgage is offline   Reply With Quote
Old 06-02-2015, 04:04 PM   #17
pengchy
Senior Member
 
Location: China

Join Date: Feb 2009
Posts: 116
Default

Maybe we just need a more work to sort the bam file according to read name "samtools sort -n"
pengchy is offline   Reply With Quote
Old 10-16-2015, 12:27 PM   #18
fanli
Senior Member
 
Location: California

Join Date: Jul 2014
Posts: 198
Default

It's a really easy fix. Editing my file (yours may be in a different location):
Code:
sudo vim +625 /usr/lib64/python2.7/site-packages/HTSeq-0.6.1p1-py2.7-linux-x86_64.egg/HTSeq/__init__.py
in vim, replace the max_buffer_size default setting (my new setting is bolded):
Code:
def pair_SAM_alignments_with_buffer( alignments, max_buffer_size=30000000 ):
fanli is offline   Reply With Quote
Old 08-19-2016, 10:14 PM   #19
arkanion
Member
 
Location: Singapore

Join Date: Jul 2016
Posts: 10
Default htseq-count bug

I have tried the above suggestion changing the limit to 30G but still get the same error.

Any fix yet?

Last edited by arkanion; 08-19-2016 at 10:16 PM. Reason: improved info
arkanion is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 05:11 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO