AddOrReplaceReadGroups error (that is probably a more general system or Java error)

[efoss]

In running Picard Tools' AddOrReplaceReadGroups, I got an error that included the following:

Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.IOException: No space left on device

This sounds like I have run out of disk space, but I haven't. My command was the following:

java -d64 -Xmx25g -jar /home/efoss/source_codes/picard-tools-1.82/AddOrReplaceReadGroups.jar \
	INPUT=AML_PAERAH_73407_RNA_partialperlclean_noMT.sam \
	OUTPUT=AML_PAERAH_73407_RNA_partialperlclean_noMT_AORRGaddedsorted.sam \
	SORT_ORDER=coordinate \
	RGID=RGID_PAERAH_73407 \
	RGLB=PAERAH_73407_library \
	RGPL=Illumina \
	RGPU=run_barcode_arbitrary \
	RGSM=AML_PAERAH_73407_RNAseq \
	RGCN=FHCRC \
	RGDS=AML_PAERAH_73407_RNAseq \
	RGDT=2012-12-22

I found some threads discussing these problems, but couldn't find a solution there. Can anyone suggest a solution? The full error message is pasted below.

Thank you.

Eric

[efoss@gizmod64:~/sequencing/MAE_in_AML$ java -d64 -Xmx25g -jar /home/efoss/source_codes/picard-tools-1.82/AddOrReplaceReadGroups.jar \
>         INPUT=AML_PAERAH_73407_RNA_partialperlclean_noMT.sam \
>         OUTPUT=AML_PAERAH_73407_RNA_partialperlclean_noMT_AORRGaddedsorted.sam \
>         SORT_ORDER=coordinate \
>         RGID=RGID_PAERAH_73407 \
>         RGLB=PAERAH_73407_library \
>         RGPL=Illumina \
>         RGPU=run_barcode_arbitrary \
>         RGSM=AML_PAERAH_73407_RNAseq \
>         RGCN=FHCRC \
>         RGDS=AML_PAERAH_73407_RNAseq \
>         RGDT=2012-12-22
[Mon Dec 24 12:29:53 PST 2012] net.sf.picard.sam.AddOrReplaceReadGroups INPUT=AML_PAERAH_73407_RNA_partialperlclean_noMT.sam OUTPUT=AML_PAERAH_73407_RNA_partialperlclean_noMT_AORRGaddedsorted.sam SORT_ORDER=coordinate RGID=RGID_PAERAH_73407 RGLB=PAERAH_73407_library RGPL=Illumina RGPU=run_barcode_arbitrary RGSM=AML_PAERAH_73407_RNAseq RGCN=FHCRC RGDS=AML_PAERAH_73407_RNAseq RGDT=2012-12-21T16:00:00-0800    VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false
[Mon Dec 24 12:29:53 PST 2012] Executing as efoss@gizmod64 on Linux 3.2.0-24-generic amd64; OpenJDK 64-Bit Server VM 1.6.0_24-b24; Picard version: 1.82(1311)
INFO    2012-12-24 12:29:53     AddOrReplaceReadGroups  Created read group ID=RGID_PAERAH_73407 PL=Illumina LB=PAERAH_73407_library SM=AML_PAERAH_73407_RNAseq

[Mon Dec 24 12:30:01 PST 2012] net.sf.picard.sam.AddOrReplaceReadGroups done. Elapsed time: 0.13 minutes.
Runtime.totalMemory()=1910898688
FAQ:  [url]http://sourceforge.net/apps/mediawiki/picard/index.php?title=Main_Page[/url]
Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.IOException: No space left on device
        at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:228)
        at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:150)
        at net.sf.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:170)
        at net.sf.picard.sam.AddOrReplaceReadGroups.doWork(AddOrReplaceReadGroups.java:100)
        at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:177)
        at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:119)
        at net.sf.picard.sam.AddOrReplaceReadGroups.main(AddOrReplaceReadGroups.java:66)
Caused by: java.io.IOException: No space left on device
        at java.io.FileOutputStream.write(Native Method)
        at org.xerial.snappy.SnappyOutputStream.writeInt(SnappyOutputStream.java:105)
        at org.xerial.snappy.SnappyOutputStream.dump(SnappyOutputStream.java:126)
        at org.xerial.snappy.SnappyOutputStream.flush(SnappyOutputStream.java:100)
        at org.xerial.snappy.SnappyOutputStream.close(SnappyOutputStream.java:137)
        at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:219)
        ... 6 more
efoss@gizmod64:~/sequencing/MAE_in_AML$

[aaronh]

I've had issues when sorting if my tmp space fills up. Did you try specifying the tmp dir?

[efoss]

I've had issues when sorting if my tmp space fills up. Did you try specifying the tmp dir?

Hi aaronh,

I'm afraid I don't know what the tmp dir is. Googling around, it looks like it's a temporary directory used for 'scratch' space. How can I specify it?

Since posting this, I ended up finding these two links.

http://stackoverflow.com/questions/3338732/hudson-fails-with-error-no-space-left-on-device-though-there-is-enough-space-o

http://www.ivankuznetsov.com/2010/02/no-space-left-on-device-running-out-of-inodes.html

I couldn't get the solution in the second link to work, but what I read here suggested that there was not a lack of disk space, as one might think from the error message, but instead a shortage of 'i-nodes' (don't know anything about those either). That suggested to me that using a more powerful system might fix things. I was running my commands on 12 cores with 48 gigs of ram. I switched to a more powerful linux server and then everything worked fine. So that's good, but I still would like to understand what is going on here. Also, use of the more powerful server for tasks that can take a long time (> 20 minutes) is discouraged, so my solution to use it is suboptimal.

Thanks.

Eric

[aaronh]

Try using java -Djava.io.tmpdir=/path/to/tmpdir -jar Add...

http://stackoverflow.com/questions/1924136/environment-variable-to-control-java-io-tmpdir

Some linux servers have a dedicated tmp partition that can fill up with all the little files that the sorting creates before merging. You can see if your server has this by using the 'df' command and also see where you default tmp space is by using the 'env' command.

[efoss]

Hi aaronh,

Thanks very much for the help. I really appreciate it.

Best wishes,

Eric