Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Picard error

    Hi,

    I'm having trouble understanding some of the params with Picard- well not really understanding but I'm getting this error here:

    ...
    INFO 2010-10-26 22:45:32 MarkDuplicates Read 40000000 records. Tracking 2669417 as yet unmatched pairs. 69542 records in RAM. Last sequence index: 2
    INFO 2010-10-26 22:45:47 MarkDuplicates Read 41000000 records. Tracking 2725267 as yet unmatched pairs. 47482 records in RAM. Last sequence index: 2
    INFO 2010-10-26 22:46:15 MarkDuplicates Read 42000000 records. Tracking 2779116 as yet unmatched pairs. 25059 records in RAM. Last sequence index: 2
    [Tue Oct 26 22:46:36 CDT 2010] net.sf.picard.sam.MarkDuplicates done.
    Runtime.totalMemory()=778698752
    Exception in thread "main" net.sf.picard.PicardException: Exception writing ReadEnds to file.
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:74)
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:32)
    at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:185)
    at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
    at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:305)
    at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:109)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:150)
    at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:93)
    Caused by: java.io.IOException: No space left on device
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:260)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:71)
    ... 7 more
    ...

    Has anyone ever gotten this writing read ends to file error?

  • #2
    Caused by: java.io.IOException: No space left on device
    No more space in the hard drive ?
    Francois Sabot, PhD

    Be realistic. Demand the Impossible.
    www.wikiposon.org

    Comment


    • #3
      I thought that was what it would be, but is this a simple matter of modifying MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP to greater than the number of mapped reads i have in the .bam file?

      Comment


      • #4
        Originally posted by JohnK View Post
        I thought that was what it would be, but is this a simple matter of modifying MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP to greater than the number of mapped reads i have in the .bam file?
        No. Your problem is *writing* to the disk. Not reading from it.

        Comment


        • #5
          I would check whether or not your drive is full, write-protected and/or that you have permissions to write to the drive/directory in question. It mostly looks like the drive is full and you'd have to delete files off your drive.

          Comment


          • #6
            Thanks, ladies/guys. My admin-dude found the issue for me and it was along the lines of what you were both saying in a way. I posted the reason on another thread.

            Comment


            • #7
              Picard Exception

              Any ideas for why it can't create a ReadEnds File?
              Code:
              Runtime.totalMemory()=166526976
              Exception in thread "main" net.sf.picard.PicardException: Error creating temporary ReadEnds file
              	at net.sf.picard.sam.DiskReadEndsMap.getOutputStreamForSequence(DiskReadEndsMap.java:175)
              	at net.sf.picard.sam.DiskReadEndsMap.put(DiskReadEndsMap.java:147)
              	at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:278)
              	at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:109)
              	at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:160)
              	at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:93)
              Caused by: java.io.FileNotFoundException: /media/1B3B8E0F52CC3359/mRNAsnp/DREM.4603601404094682558.tmp/32279.read_ends (Too many open files)
              	at java.io.FileOutputStream.open(Native Method)
              	at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
              	at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
              	at net.sf.picard.sam.DiskReadEndsMap.getOutputStreamForSequence(DiskReadEndsMap.java:170)
              	... 5 more
              it says there are too many open files, but I should have plenty
              Code:
               cat /proc/sys/fs/file-max
              197316

              Comment


              • #8
                From http://lj4newbies.blogspot.com/2007/...pen-files.html (first google results for Java Too Many open Files)

                This is because too many file descriptors're opened by tomcat. File descriptor can be limited in both system level and shell level.

                To check maximum number of fd in system type 'cat /proc/sys/fs/file-max'. In my case it is 65536(someone said it should set to 200000). Tomcat error when try to open socket number 272 so I think 65536 is ok for me for now. Anyway if u want to set it add 'fs.file-max = 200000' to /etc/sysctl.conf
                So, the descriptors seem to be too numerous
                Francois Sabot, PhD

                Be realistic. Demand the Impossible.
                www.wikiposon.org

                Comment


                • #9
                  Your friendly sysadmin will help you here. 200k may be not enough FDs. It depens on your environment. Are you sharing the box among many users?
                  -drd

                  Comment


                  • #10
                    It's just me.

                    Comment


                    • #11
                      Thank you all!

                      But it seems i dont have the right to change the /etc/sysctl.conf, any other solutions?

                      thanks.

                      Comment


                      • #12
                        Hi everyone,

                        I have the same problem when trying to run SamSort.jar for a 33GB BAM, I asked my sys-admin to expand the file-max limit to 8,000,000 and.. I still got the same error. So there maybe something else...

                        Thanks.

                        Comment


                        • #13
                          tatinhawk: Given that this thread is around 9 months old and has talked about at least two different type of errors, what "same error" are you referring to?

                          Comment


                          • #14
                            Originally posted by tatinhawk View Post
                            Hi everyone,

                            I have the same problem when trying to run SamSort.jar for a 33GB BAM, I asked my sys-admin to expand the file-max limit to 8,000,000 and.. I still got the same error. So there maybe something else...

                            Thanks.
                            sorry ... I forgot to mention that the error I have is the " Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /temp/sortingcollection.1760628862445939889.tmp (Too many open files)"

                            Comment


                            • #15
                              On my university cluster the max number of open files is set to 1024, and I was getting the same error reported above. Setting the following picard flag to something less than that max seems to resolve the problem.

                              MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=[some number lower than the output of `ulimit -n`]

                              Comment

                              Latest Articles

                              Collapse

                              • seqadmin
                                Strategies for Sequencing Challenging Samples
                                by seqadmin


                                Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                                03-22-2024, 06:39 AM
                              • seqadmin
                                Techniques and Challenges in Conservation Genomics
                                by seqadmin



                                The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                                Avian Conservation
                                Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                                03-08-2024, 10:41 AM

                              ad_right_rmr

                              Collapse

                              News

                              Collapse

                              Topics Statistics Last Post
                              Started by seqadmin, Yesterday, 06:37 PM
                              0 responses
                              10 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, Yesterday, 06:07 PM
                              0 responses
                              10 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 03-22-2024, 10:03 AM
                              0 responses
                              51 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 03-21-2024, 07:32 AM
                              0 responses
                              67 views
                              0 likes
                              Last Post seqadmin  
                              Working...
                              X