How to correct BAM file EOF marker absent error

[vinay052003]

Hi,
I am trying to run bamUtils's splitBam program to split my bam file into read groups but getting the error: BGZF EOF marker is missing.
Samtools also gives the similar error "EOF marker absent. I understand that my file is not corrupt but just an older BAM/SAM version. Is there anyway to fix these errors so that bamUtils can be run on it.

[maubp]

Try using 'samtools view' BAM to BAM, or if you can program just append the 28 byte EOF marker. It might be possible to do that with Unix cat, if you can work out how to escape the binary values...

[swbarnes2]

Does the "no EOF" error actually stop the SAMtools command from functioning properly? If not, I'd say just leave it be.

[maubp]

Does the "no EOF" error actually stop the SAMtools command from functioning properly? If not, I'd say just leave it be.

The warning from samtools is harmless (but scary - and if you get in the habit of ignoring it you may miss a truly truncated file).

I don't know if bamUtils is less tolerant and simply aborts.

[vinay052003]

Thanks maub for the replies.
bamUtils simply aborts the program.. which is somewhat annoying. I have large bam files from whole genome. Now converting them to newer version with samtools takes hours.
Both Samtools and Picard can easily process "EOF absent" files.

[maubp]

You should probably contact the bamUtils authors to treat this as a warning not an error.

If you are going to use the 'samtools view' trick to 'fix' the BAM files, can you pipe this directly into bamUtils? That should be faster (less disk IO), and if you select BAM with no compression even faster still.

The trick of appending the missing 28 bytes EOF is probably the most efficient solution, as long as you don't mind risking editing the BAM file in situ. You could try this little Python script I just wrote - but please backup your data first and let me know if it works nicely - it has had only minimal testing: https://github.com/peterjc/picobio/b...gzf_add_eof.py

[vinay052003]

I am trying to get in touch with the bamUtils authors. Thanks for the trick about piping samtools output to bamUtils and for the script as well.

[vinay052003]

Hi maubp,
I used your script and its seems to run without any error but takes several hours before it finishes. Could it be due to the fact that my bam files are in 100gb size?

[maubp]

Several hours? Wow. I haven't tried it on any files that large but that does surprise me. I can think of a few variations that might help (e.g. opening in append mode to add the block). I may have time to look at this tomorrow...

[vinay052003]

Its all right.. no rush. I've got reply from bamUtil's developers and I think the problem will be fixed. Thanks a lot for the help.

[maubp]

OK, yes, I could reproduce the slowness with bigger files (and found it didn't seem to work as expected on Mac OS 10.6 which I'd not tried before).

A slight difference to use append mode explicitly worked wonders - it should now be sub-second as I originally intended. Could you retry (again, backup the data first - this does edit in place and is new code):

https://github.com/peterjc/picobio/blob/master/sambam/bgzf_add_eof.py

Note this is now v0.0.1 (the original you tried I have dubbed v0.0.0).

[vinay052003]

I'll give it a shot and let you know. Actually I forgot to mention that I was using Mac Osx as well.

Thanks.