Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Tophat No Such File Error

    Hi All,
    I'm tophatting some data, and it's giving me a strange error:
    Code:
    $ nohup nice tophat   ../bowtie_dbs/dpse_chromosomes GSM499403_Dpseudo_R23s_8_bowtie_75.txt.fq,GSM499403_Dpseudo_R33s_5_bowtie_75.txt.fq,GSM49943_Dpseudo_R33s_6_bowtie_75.txt.fq &> tophat_io2 &
    $ # Time Passes
    $ cat tophat_io2 
    nohup: ignoring input
    
    [Tue May  4 15:40:52 2010] Beginning TopHat run (v1.0.13)
    -----------------------------------------------
    [Tue May  4 15:40:52 2010] Preparing output location ./tophat_out/
    [Tue May  4 15:40:53 2010] Checking for Bowtie index files
    [Tue May  4 15:40:53 2010] Checking for reference FASTA file
    [Tue May  4 15:40:53 2010] Checking for Bowtie
    	Bowtie version:		 0.12.1.0
    [Tue May  4 15:40:53 2010] Checking reads
    	seed length:	 75bp
    	format:		 fastq
    	quality scale:	 phred33 (default)
    [Tue May  4 16:06:03 2010] Mapping reads against dpse_chromosomes with Bowtie
    [Tue May  4 18:54:28 2010] Joining segment hits
    Traceback (most recent call last):
      File "/mnt/data/sw/bin/tophat", line 1635, in <module>
        sys.exit(main())
      File "/mnt/data/sw/bin/tophat", line 1595, in main
        user_supplied_juncs)
      File "/mnt/data/sw/bin/tophat", line 1395, in spliced_alignment
        segment_len)
      File "/mnt/data/sw/bin/tophat", line 1085, in split_reads
        reads_file = open(reads_filename)
    IOError: [Errno 2] No such file or directory: './tophat_out/tmp//left_kept_reads_missing.fq'
    left_kept_reads_missing.fq is, indeed, missing. Any idea what's going on here?

  • #2
    I have the same problem. I use the same versions of tophat and bowtie. It doesn´t give any error with the test data.

    Comment


    • #3
      I was just about to post about this.. same deal here. I then went on to run the same command and it completed the run, but there are only 114 lines in accepted_hits.sam and 21 lines in junctions.bed.

      Here is the two runs:
      Code:
      [heather@frankie (Mon May 24 14:24:08) /home/heather/group/heather/appseq]% tophat -o may_24 --segment-mismatches 3 equcab2 filt_trim_tag_all.fq 
       
      [Mon May 24 14:24:14 2010] Beginning TopHat run (v1.0.13)
      -----------------------------------------------
      [Mon May 24 14:24:14 2010] Preparing output location may_24/
      [Mon May 24 14:24:14 2010] Checking for Bowtie index files
      [Mon May 24 14:24:14 2010] Checking for reference FASTA file
      [Mon May 24 14:24:14 2010] Checking for Bowtie
              Bowtie version:          0.12.3.0
      [Mon May 24 14:24:14 2010] Checking reads
              seed length:     78bp
              format:          fastq
              quality scale:   phred33 (default)
      [Mon May 24 14:49:13 2010] Mapping reads against equcab2 with Bowtie
      Error: cannot open reads file may_24/left_kept_reads.fq for reading
      [Mon May 24 14:49:14 2010] Joining segment hits
      Traceback (most recent call last):
        File "/usr/bin/tophat", line 1635, in <module>
          sys.exit(main())
        File "/usr/bin/tophat", line 1595, in main
          user_supplied_juncs)
        File "/usr/bin/tophat", line 1395, in spliced_alignment
          segment_len)
        File "/usr/bin/tophat", line 1085, in split_reads
          reads_file = open(reads_filename)
      IOError: [Errno 2] No such file or directory: 'may_24/tmp//left_kept_reads_missing.fq'
      [heather@frankie (Mon May 24 14:49:14) /home/heather/group/heather/appseq]% tophat -o may_24 --segment-mismatches 3 equcab2 filt_trim_tag_all.fq
       
      [Mon May 24 14:50:14 2010] Beginning TopHat run (v1.0.13)
      -----------------------------------------------
      [Mon May 24 14:50:14 2010] Preparing output location may_24/
      [Mon May 24 14:50:14 2010] Checking for Bowtie index files
      [Mon May 24 14:50:14 2010] Checking for reference FASTA file
      [Mon May 24 14:50:14 2010] Checking for Bowtie
              Bowtie version:          0.12.3.0
      [Mon May 24 14:50:14 2010] Checking reads
              seed length:     78bp
              format:          fastq
              quality scale:   phred33 (default)
      [Mon May 24 15:13:25 2010] Mapping reads against equcab2 with Bowtie
      Error: cannot open reads file may_24/left_kept_reads.fq for reading
      [Mon May 24 15:13:25 2010] Joining segment hits
              Splitting reads into 3 segments
      [Mon May 24 15:13:25 2010] Mapping reads against equcab2 with Bowtie
      [Mon May 24 15:14:28 2010] Mapping reads against equcab2 with Bowtie
      [Mon May 24 15:14:46 2010] Mapping reads against equcab2 with Bowtie
      [Mon May 24 15:15:00 2010] Searching for junctions via segment mapping
      [Mon May 24 15:22:33 2010] Retrieving sequences for splices
      [Mon May 24 15:27:38 2010] Indexing splices
      [Mon May 24 15:27:47 2010] Mapping reads against segment_juncs with Bowtie
      [Mon May 24 15:27:48 2010] Mapping reads against segment_juncs with Bowtie
      [Mon May 24 15:27:48 2010] Mapping reads against segment_juncs with Bowtie
      [Mon May 24 15:27:48 2010] Joining segment hits
      [Mon May 24 15:27:48 2010] Reporting output tracks
      -----------------------------------------------
      Run complete [00:37:42 elapsed]
      [heather@frankie (Mon May 24 15:27:56) /home/heather/group/heather/appseq]%
      And here are all of the file*.log from logs/. I had about 15 million total reads.
      Code:
      # reads processed: 587
      # reads with at least one reported alignment: 313 (53.32%)
      # reads that failed to align: 265 (45.14%)
      # reads with alignments suppressed due to -m: 9 (1.53%)
      Reported 1306 alignments to 1 output stream(s)
      # reads processed: 587
      # reads with at least one reported alignment: 494 (84.16%)
      # reads that failed to align: 88 (14.99%)
      # reads with alignments suppressed due to -m: 5 (0.85%)
      Reported 1423 alignments to 1 output stream(s)
      Error while flushing and closing output
      Command: bowtie -q --un may_24/tmp//left_kept_reads_missing.fq --max /dev/null -v 3 -p 1 -k 40 -m 40 equcab2 may_24/left_kept_reads.fq 
      # reads processed: 587
      # reads with at least one reported alignment: 398 (67.80%)
      # reads that failed to align: 187 (31.86%)
      # reads with alignments suppressed due to -m: 2 (0.34%)
      Reported 1012 alignments to 1 output stream(s)
      # reads processed: 587
      # reads with at least one reported alignment: 20 (3.41%)
      # reads that failed to align: 567 (96.59%)
      Reported 125 alignments to 1 output stream(s)
      Error while flushing and closing output
      Command: bowtie -q --un may_24/tmp//left_kept_reads_missing.fq --max /dev/null -v 3 -p 1 -k 40 -m 40 equcab2 may_24/left_kept_reads.fq 
      # reads processed: 587
      # reads with at least one reported alignment: 6 (1.02%)
      # reads that failed to align: 581 (98.98%)
      Reported 14 alignments to 1 output stream(s)
      # reads processed: 587
      # reads with at least one reported alignment: 13 (2.21%)
      # reads that failed to align: 573 (97.61%)
      # reads with alignments suppressed due to -m: 1 (0.17%)
      Reported 38 alignments to 1 output stream(s)

      Comment


      • #4
        Believe it or not, this is an out of memory error. It is disguised because bowtie was called by tophat and failed silently - refer to the logs you will see a log explaining this.

        If you are running on pbs or a cluster you may need to allocate more RAM - try -vmem (mem does not work on most linux beyond 1 node)

        Comment


        • #5
          Thank you! Well, luckily more RAM was ordered last week, so hopefully it'll work when I try next time.

          Comment


          • #6
            If you have at least 5Gb, it should be OK and your problem lies elsewhere - check in the output/log directory for the exact problem - it could be anything that prevents bowtie doing its thing - maybe even permissions.

            Comment


            • #7
              This is being run on a single workstation, not a cluster. It now has 16 GB RAM (divided between two CPUs). It seems like bowtie is creating the "wrong" file.. I have a full strace log, but this seems to be the relevant portion:

              Code:
               
              stat("app_tophat/left_kept_reads.fq", 0x7fff1bcda710) = -1 ENOENT (No such file or directory)
              open("app_tophat/left_kept_reads.fq", O_WRONLY|O_CREAT|O_APPEND, 0666) = 4
              fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
              mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fad7d211000
              fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
              lseek(4, 0, SEEK_SET)                   = 0
              fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
              open("app_tophat/logs/prep_reads.log", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 5
              fstat(5, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
              write(3, "/usr/bin/prep_reads --min-anchor "..., 469) = 469
              write(3, "\n"..., 1)                    = 1
              pipe([6, 7])                            = 0
              fcntl(7, F_GETFD)                       = 0
              fcntl(7, F_SETFD, FD_CLOEXEC)           = 0
              clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fad7d1e2780) = 5987
              close(7)                                = 0
              mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fad7aacd000
              read(6, ""..., 1048576)                 = 0
              mremap(0x7fad7aacd000, 1052672, 4096, MREMAP_MAYMOVE) = 0x7fad7aacd000
              close(6)                                = 0
              munmap(0x7fad7aacd000, 4096)            = 0
              wait4(5987, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 5987
              --- SIGCHLD (Child exited) @ 0 (0) ---
              close(4)                                = 0
              munmap(0x7fad7d211000, 4096)            = 0
              close(5)                                = 0
              stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3519, ...}) = 0
              write(2, "[Wed Jun  2 11:25:27 2010] Mappin"..., 68[Wed Jun  2 11:25:27 2010] Mapping reads against equcab2 with Bowtie) = 68
              write(2, "\n"..., 1
              )                    = 1
              stat("app_tophat/tmp/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
              stat("/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=4096, ...}) = 0
              lstat("/tmp/filewFaQvp", 0x7fff1bcda430) = -1 ENOENT (No such file or directory)
              open("app_tophat/logs/filewFaQvp.log", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 4
              fstat(4, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
              pipe([5, 6])                            = 0
              pipe([7, 8])                            = 0
              fcntl(8, F_GETFD)                       = 0
              fcntl(8, F_SETFD, FD_CLOEXEC)           = 0
              clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fad7d1e2780) = 5993
              close(8)                                = 0
              close(6)                                = 0
              mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fad7aacd000
              read(7, ""..., 1048576)                 = 0
              mremap(0x7fad7aacd000, 1052672, 4096, MREMAP_MAYMOVE) = 0x7fad7aacd000
              close(7)                                = 0
              munmap(0x7fad7aacd000, 4096)            = 0
              fcntl(5, F_GETFL)                       = 0 (flags O_RDONLY)
              fstat(5, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
              mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fad7d211000
              lseek(5, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)
              fstat(5, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
              munmap(0x7fad7d211000, 4096)            = 0
              write(3, "bowtie -q --un app_tophat/tmp//le"..., 133) = 133
              write(3, " "..., 1)                     = 1
              write(3, "|"..., 1)                     = 1
              write(3, " "..., 1)                     = 1
              write(3, "/usr/bin/fix_map_ordering --fastq"..., 65) = 65
              write(3, " "..., 1)                     = 1
              write(3, ">"..., 1)                     = 1
              write(3, " "..., 1)                     = 1
              write(3, "app_tophat/tmp/filewFaQvp"..., 25) = 25
              write(3, "\n"..., 1)                    = 1
              open("app_tophat/tmp//left_kept_reads.bwtout", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 6
              fstat(6, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
              pipe([7, 8])                            = 0
              fcntl(8, F_GETFD)                       = 0
              fcntl(8, F_SETFD, FD_CLOEXEC)           = 0
              clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fad7d1e2780) = 5994
              close(8)                                = 0
              mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fad7aacd000
              read(7, ""..., 1048576)                 = 0
              mremap(0x7fad7aacd000, 1052672, 4096, MREMAP_MAYMOVE) = 0x7fad7aacd000
              close(7)                                = 0
              munmap(0x7fad7aacd000, 4096)            = 0
              close(6)                                = 0
              wait4(5994, Error: cannot open reads file app_tophat/left_kept_reads.fq for reading
              [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], 0, NULL) = 5994
              --- SIGCHLD (Child exited) @ 0 (0) ---
              Run output and logs:
              Code:
              % tophat -o app_tophat --solexa1.3-quals equcab2 filt_trim_tag_all.fastq
               
              [Wed Jun  2 10:38:35 2010] Beginning TopHat run (v1.0.13)
              -----------------------------------------------
              [Wed Jun  2 10:38:35 2010] Preparing output location app_tophat/
              [Wed Jun  2 10:38:35 2010] Checking for Bowtie index files
              [Wed Jun  2 10:38:35 2010] Checking for reference FASTA file
              [Wed Jun  2 10:38:35 2010] Checking for Bowtie
                      Bowtie version:          0.12.3.0
              [Wed Jun  2 10:38:35 2010] Checking reads
                      seed length:     78bp
                      format:          fastq
                      quality scale:   phred64 (reads generated with GA pipeline version >= 1.3)
              [Wed Jun  2 11:01:19 2010] Mapping reads against equcab2 with Bowtie
              Error: cannot open reads file app_tophat/left_kept_reads.fq for reading
              [Wed Jun  2 11:01:19 2010] Joining segment hits
              Traceback (most recent call last):
                File "/usr/bin/tophat", line 1635, in <module>
                  sys.exit(main())
                File "/usr/bin/tophat", line 1595, in main
                  user_supplied_juncs)
                File "/usr/bin/tophat", line 1395, in spliced_alignment
                  segment_len)
                File "/usr/bin/tophat", line 1085, in split_reads
                  reads_file = open(reads_filename)
              IOError: [Errno 2] No such file or directory: 'app_tophat/tmp//left_kept_reads_missing.fq'
               
              % cat files0rgWJ.log 
              Error while flushing and closing output
              Command: bowtie -q --un app_tophat/tmp//left_kept_reads_missing.fq --max /dev/null -v 2 -p 1 -k 40 -m 40 equcab2 app_tophat/left_kept_reads.fq 
               
              % cat long_spanning_reads.log 
              long_spanning_reads v1.0.13
              --------------------------------------------
              Opening app_tophat/left_kept_reads.fq for reading
              Opening /dev/null for reading
              Opening app_tophat/tmp//left_kept_reads.bwtout for reading
              Loading spliced hits...done
              Loading junctions...done
               
              % cat prep_reads.log 
              prep_reads v1.0.13
              ---------------------------
              25833 out of 16604098 reads have been filtered out
               
              % cat run.log 
              /usr/bin/tophat -o app_tophat --solexa1.3-quals equcab2 filt_trim_tag_all.fastq
              /usr/bin/prep_reads --min-anchor 8 --splice-mismatches 0 --min-report-intron 50 --max-report-intron 500000 --min-isoform-fraction 0.15 --output-dir app_tophat/ --max-multihits 40 --segment-length 25 --segment-mismatches 2 --min-closure-exon 100 --min-closure-intron 50 --max-closure-intron 5000 --min-coverage-intron 50 --max-coverage-intron 20000 --min-segment-intron 50 --max-segment-intron 500000 --no-microexon-search --phred64-quals --fastq filt_trim_tag_all.fastq
              bowtie -q --un app_tophat/tmp//left_kept_reads_missing.fq --max /dev/null -v 2 -p 1 -k 40 -m 40 equcab2 app_tophat/left_kept_reads.fq | /usr/bin/fix_map_ordering --fastq app_tophat/left_kept_reads.fq - > app_tophat/tmp/files0rgWJ
              /usr/bin/long_spanning_reads --min-anchor 8 --splice-mismatches 0 --min-report-intron 50 --max-report-intron 500000 --min-isoform-fraction 0.15 --output-dir app_tophat/ --max-multihits 40 --segment-length 25 --segment-mismatches 2 --min-closure-exon 100 --min-closure-intron 50 --max-closure-intron 5000 --min-coverage-intron 50 --max-coverage-intron 20000 --min-segment-intron 50 --max-segment-intron 500000 --no-microexon-search --phred64-quals app_tophat/left_kept_reads.fq /dev/null app_tophat/tmp//left_kept_reads.bwtout  > app_tophat/tmp/fileLLKacr
              Last edited by raela; 06-02-2010, 10:24 AM. Reason: add additional run logs

              Comment


              • #8
                There is a folder called "logs" in your output. The answer will be a bowtie failure in one of those. It will advise why it failed. My gues is one of;
                1. Insufficient disk space (You need a LOT relative to the size of file - say 10X)
                2. Insufficient permissions on the output directory

                If you aren't specifying the output directory to tophat, you probably should and ensure it has chmod 755 (should be overkill) just to be safe.

                Comment


                • #9
                  The second code block of my previous post has the contents of all of the files in logs/ from that run. It produced filewFaQvp.log, long_spanning_reads.log, prep_reads.log, run.log. The bowtie error appears to be:

                  % cat files0rgWJ.log
                  Error while flushing and closing output
                  Command: bowtie -q --un app_tophat/tmp//left_kept_reads_missing.fq --max /dev/null -v 2 -p 1 -k 40 -m 40 equcab2 app_tophat/left_kept_reads.fq

                  I do not think it's an issue of permissions - output directory is created with drwxrwxr-x for heather:users. The directory I am running tophat from has the same permissions. As for disk space, I have 681GB available on that partition. My reference genome is 2.4GB in fasta format and my reads file is 3.3GB.

                  Any other ideas? Thank you for the help so far!

                  Comment


                  • #10
                    Were you ever able to fine a solution to your problem? I'm having a similar error with version 1.0.14. My error could be due to the fact that my reference was mistakenly in color space while my reads were not. Downloading the other reference files and going to give it another go.

                    I have 24 Gigs of RAM, 300G free hard drive space.

                    Comment


                    • #11
                      I have the same problem with tophat. bowtie alone runs fine, so it is hard to believe that bowtie crashes silently. In any case, looks like the bowtie index was built for colorspace. When I rebuilt the bowtie index without -C option tophat ran fine.
                      Last edited by tsucheta; 09-08-2010, 08:25 PM.

                      Comment


                      • #12
                        I also have the same problem with TopHat.
                        Could any guy tell us how to solve it?

                        Comment


                        • #13
                          I was having this problem for some experiments too, but then I noticed that I was using an older version of Bowtie (v0.12.3) with a new version of Tophat (v1.1.1). I suspect these is an incompatibility in how some of the output files are named/handled.

                          Try updating both Tophat and Bowtie to the most recent versions. Double-check your PATH to ensure that you are calling the newest versions.

                          Comment


                          • #14
                            I have also solved this problem.
                            The version of TopHat I applied is v1.1.2 and the version of Bowtie is v0.12.7.0.
                            Acturally, I don't think the problem is triggered by the conflicted versions.
                            I re-formatted the genome reference and everything is ok.
                            So I guess there's something wrong with the formatted reference.

                            Comment


                            • #15
                              Hello,

                              I am having the same problem, could you please explain it in detail what re-formatting the genome reference stands for? what did you do actually?

                              Thanks in advance.

                              Comment

                              Latest Articles

                              Collapse

                              • seqadmin
                                Strategies for Sequencing Challenging Samples
                                by seqadmin


                                Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                                03-22-2024, 06:39 AM
                              • seqadmin
                                Techniques and Challenges in Conservation Genomics
                                by seqadmin



                                The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                                Avian Conservation
                                Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                                03-08-2024, 10:41 AM

                              ad_right_rmr

                              Collapse

                              News

                              Collapse

                              Topics Statistics Last Post
                              Started by seqadmin, Yesterday, 06:37 PM
                              0 responses
                              10 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, Yesterday, 06:07 PM
                              0 responses
                              9 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 03-22-2024, 10:03 AM
                              0 responses
                              49 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 03-21-2024, 07:32 AM
                              0 responses
                              67 views
                              0 likes
                              Last Post seqadmin  
                              Working...
                              X