Picard error

[JohnK]

Hi,

I'm having trouble understanding some of the params with Picard- well not really understanding but I'm getting this error here:

...
INFO 2010-10-26 22:45:32 MarkDuplicates Read 40000000 records. Tracking 2669417 as yet unmatched pairs. 69542 records in RAM. Last sequence index: 2
INFO 2010-10-26 22:45:47 MarkDuplicates Read 41000000 records. Tracking 2725267 as yet unmatched pairs. 47482 records in RAM. Last sequence index: 2
INFO 2010-10-26 22:46:15 MarkDuplicates Read 42000000 records. Tracking 2779116 as yet unmatched pairs. 25059 records in RAM. Last sequence index: 2
[Tue Oct 26 22:46:36 CDT 2010] net.sf.picard.sam.MarkDuplicates done.
Runtime.totalMemory()=778698752
Exception in thread "main" net.sf.picard.PicardException: Exception writing ReadEnds to file.
at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:74)
at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:32)
at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:185)
at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:305)
at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:109)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:150)
at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:93)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:71)
... 7 more
...

Has anyone ever gotten this writing read ends to file error?

[francois.sabot]

Caused by: java.io.IOException: No space left on device

No more space in the hard drive ?

[JohnK]

I thought that was what it would be, but is this a simple matter of modifying MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP to greater than the number of mapped reads i have in the .bam file?

[westerman]

Originally Posted by JohnK

I thought that was what it would be, but is this a simple matter of modifying MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP to greater than the number of mapped reads i have in the .bam file?

No. Your problem is *writing* to the disk. Not reading from it.

[mrawlins]

I would check whether or not your drive is full, write-protected and/or that you have permissions to write to the drive/directory in question. It mostly looks like the drive is full and you'd have to delete files off your drive.

[JohnK]

Thanks, ladies/guys. My admin-dude found the issue for me and it was along the lines of what you were both saying in a way. I posted the reason on another thread.

[RockChalkJayhawk]

Any ideas for why it can't create a ReadEnds File?

Runtime.totalMemory()=166526976
Exception in thread "main" net.sf.picard.PicardException: Error creating temporary ReadEnds file
	at net.sf.picard.sam.DiskReadEndsMap.getOutputStreamForSequence(DiskReadEndsMap.java:175)
	at net.sf.picard.sam.DiskReadEndsMap.put(DiskReadEndsMap.java:147)
	at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:278)
	at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:109)
	at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:160)
	at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:93)
Caused by: java.io.FileNotFoundException: /media/1B3B8E0F52CC3359/mRNAsnp/DREM.4603601404094682558.tmp/32279.read_ends (Too many open files)
	at java.io.FileOutputStream.open(Native Method)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
	at net.sf.picard.sam.DiskReadEndsMap.getOutputStreamForSequence(DiskReadEndsMap.java:170)
	... 5 more

it says there are too many open files, but I should have plenty

 cat /proc/sys/fs/file-max
197316

[francois.sabot]

From http://lj4newbies.blogspot.com/2007/...pen-files.html (first google results for Java Too Many open Files)

This is because too many file descriptors're opened by tomcat. File descriptor can be limited in both system level and shell level.

To check maximum number of fd in system type 'cat /proc/sys/fs/file-max'. In my case it is 65536(someone said it should set to 200000). Tomcat error when try to open socket number 272 so I think 65536 is ok for me for now. Anyway if u want to set it add 'fs.file-max = 200000' to /etc/sysctl.conf

So, the descriptors seem to be too numerous

[drio]

Your friendly sysadmin will help you here. 200k may be not enough FDs. It depens on your environment. Are you sharing the box among many users?

[RockChalkJayhawk]

It's just me.

[ribozyme]

Thank you all!

But it seems i dont have the right to change the /etc/sysctl.conf, any other solutions?

thanks.

[tatinhawk]

Hi everyone,

I have the same problem when trying to run SamSort.jar for a 33GB BAM, I asked my sys-admin to expand the file-max limit to 8,000,000 and.. I still got the same error. So there maybe something else...

Thanks.

[westerman]

tatinhawk: Given that this thread is around 9 months old and has talked about at least two different type of errors, what "same error" are you referring to?

[tatinhawk]

Originally Posted by tatinhawk

Hi everyone,

I have the same problem when trying to run SamSort.jar for a 33GB BAM, I asked my sys-admin to expand the file-max limit to 8,000,000 and.. I still got the same error. So there maybe something else...

Thanks.

sorry ... I forgot to mention that the error I have is the " Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /temp/sortingcollection.1760628862445939889.tmp (Too many open files)"

[ngcrawford]

On my university cluster the max number of open files is set to 1024, and I was getting the same error reported above. Setting the following picard flag to something less than that max seems to resolve the problem.

MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=[some number lower than the output of `ulimit -n`]