Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • problems processing bcl

    i have installed bcl2fastq in our cluster, i also copied the BackupNGS directroy at the cluster there are files called as duplicati-full-signature.20150312T101755Z.vol1460.zip in it.

    Now i would like to run bcl2fastq, to do it i used the following command

    /home/pmarcmun/bcl_t/bcl2fastq -i /home/pmarcmun/BackupNGS/ -o /home/pmarcmun/bcl_t/output/bs_1_1.fastq -r 4 -d 4 -p 4 -w 4

    and i get those error messages

    BCL to FASTQ file converter
    bcl2fastq v2.16.0.10
    Copyright (c) 2007-2015 Illumina, Inc.

    2015-04-16 11:41:50 [2b367952fda0] Command-line invocation: /home/pmarcmun/bcl_t/bcl2fastq -i /home/pmarcmun/BackupNGS/ -o /home/pmarcmun/bcl_t/output/bs_1_1.fastq -r 4 -d 4 -p 4 -w 4
    2015-04-16 11:41:50 [2b367952fda0] INFO: Minimum log level: INFO
    2015-04-16 11:41:50 [2b367952fda0] INFO: Runfolder path: '/scratch/177420.1.all.q/'
    2015-04-16 11:41:50 [2b367952fda0] INFO: Input path: '/home/pmarcmun/BackupNGS/'
    2015-04-16 11:41:50 [2b367952fda0] INFO: Intensities path: '/home/pmarcmun/'
    2015-04-16 11:41:50 [2b367952fda0] INFO: Output path: '/home/pmarcmun/bcl_t/output/bs_1_1.fastq'
    2015-04-16 11:41:50 [2b367952fda0] INFO: InterOp path: '/scratch/177420.1.all.q/InterOp/'
    2015-04-16 11:41:50 [2b367952fda0] INFO: Stats path: '/home/pmarcmun/bcl_t/output/bs_1_1.fastq/Stats/'
    2015-04-16 11:41:50 [2b367952fda0] INFO: Reports path: '/home/pmarcmun/bcl_t/output/bs_1_1.fastq/Reports/'
    2015-04-16 11:41:50 [2b367952fda0] INFO: Detected CPUs: 16
    2015-04-16 11:41:50 [2b367952fda0] INFO: Loading threads: 4
    2015-04-16 11:41:50 [2b367952fda0] INFO: Processing threads: 4
    2015-04-16 11:41:50 [2b367952fda0] INFO: Demultiplexing threads: 4
    2015-04-16 11:41:50 [2b367952fda0] INFO: Writing threads: 4
    2015-04-16 11:41:50 [2b367952fda0] INFO: Aggregated tiles: AUTO
    2015-04-16 11:41:50 [2b367952fda0] INFO: Allowed barcode mismatches: 1
    2015-04-16 11:41:50 [2b367952fda0] INFO: Tiles: <ALL>
    2015-04-16 11:41:50 [2b367952fda0] INFO: Minimum trimmed read length: 35
    2015-04-16 11:41:50 [2b367952fda0] INFO: Use bases masks: <NONE>
    2015-04-16 11:41:50 [2b367952fda0] INFO: Mask short adapter reads: 22
    2015-04-16 11:41:50 [2b367952fda0] INFO: Adapter stringency: 0.9
    2015-04-16 11:41:50 [2b367952fda0] INFO: Ignore missing BCLs: NO
    2015-04-16 11:41:50 [2b367952fda0] INFO: Ignore missing filters: NO
    2015-04-16 11:41:50 [2b367952fda0] INFO: Ignore missing positions: NO
    2015-04-16 11:41:50 [2b367952fda0] INFO: Ignore missing controls: NO
    2015-04-16 11:41:50 [2b367952fda0] INFO: Include non-PF clusters: NO
    2015-04-16 11:41:50 [2b367952fda0] INFO: Create FASTQs for index reads: NO
    2015-04-16 11:41:50 [2b367952fda0] INFO: Use bgzf compression for FASTQ files: YES
    2015-04-16 11:41:50 [2b367952fda0] INFO: FASTQ compression level: 4
    2015-04-16 11:41:50 [2b367952fda0] INFO: RunInfo.xml: '/scratch/177420.1.all.q/RunInfo.xml'
    2015-04-16 11:41:50 [2b367952fda0] ERROR: bcl2fastq::common::Exception: 2015-Apr-16 11:41:50: No such file or directory (2): /home/pmarcmun/bcl2fastq-source/src/cxx/lib/io/Xml.cpp(197): Throw in function boost:roperty_tree:tree bcl2fastq::io:arseXmlFile(boost::filesystem:ath)
    Dynamic exception type: boost::exception_detail::clone_impl<bcl2fastq::common::IoError>
    std::exception::what: Unable to open '/scratch/177420.1.all.q/RunInfo.xml' file for reading


    it looks like if i mess the file RunInfo.xml. Where is it? Do i have to unzip the files in the directory?

    thanks

  • #2
    Do you have the entire run folder available under /home/pmarcmun/BackupNGS/?

    The -i option should point to the /path_to/flowcell_ID/Data/Intensities/BaseCalls directory in that run folder.

    Comment


    • #3
      If you have the complete runfolder you could use '--runfolder-dir' option to simply point to the runfolder directory.

      Can you post the content of /home/pmarcmun/BackupNGS/, e.g.

      Code:
      ls -l /home/pmarcmun/BackupNGS/*

      Comment


      • #4
        i solved the problem, i don't know why but the files i had were incorrect

        Comment


        • #5
          How did you solve the problem

          Hi,
          How did you solve the problem? I have same issue, my RunInfo is correct, I double checked.

          Comment


          • #6
            Originally posted by nikooki2016 View Post
            Hi,
            How did you solve the problem? I have same issue, my RunInfo is correct, I double checked.
            Can you post the command you are using and the exact error?

            Comment


            • #7
              Originally posted by GenoMax View Post
              Can you post the command you are using and the exact error?
              Hi thanks for your replay, this is my first time run bcl2fastq, I just install it in ubuntu14045. I follow this https://nhoffman.github.io/borborygm...on-ubuntu.html to install it.

              The command I use is:

              nj@nj-VirtualBox:/media/sf_Unix/Practice/170301_NS500501_0261_AHJHWKAFXX$ /usr/local/bin/bcl2fastq -R 170301_NS500501_0261_AHJHWKAFXX -o 170301_NS500501_0261_AHJHWKAFXX/fastq --no-lane-splitting


              error messages:

              Code:
              BCL to FASTQ file converter
              bcl2fastq v2.19.0.316
              Copyright (c) 2007-2017 Illumina, Inc.
              
              2017-03-21 15:47:10 [b57d9700] Command-line invocation: /usr/local/bin/bcl2fastq -R 170301_NS500501_0261_AHJHWKAFXX -o 170301_NS500501_0261_AHJHWKAFXX/fastq --no-lane-splitting
              2017-03-21 15:47:10 [b57d9700] INFO: Minimum log level: INFO
              2017-03-21 15:47:10 [b57d9700] INFO: Sample sheet: '170301_NS500501_0261_AHJHWKAFXX/SampleSheet.csv'
              2017-03-21 15:47:10 [b57d9700] INFO: Sample sheet: NOT FOUND
              2017-03-21 15:47:10 [b57d9700] INFO: Runfolder path: '170301_NS500501_0261_AHJHWKAFXX'
              2017-03-21 15:47:10 [b57d9700] INFO: Input path: '170301_NS500501_0261_AHJHWKAFXX/Data/Intensities/BaseCalls/'
              2017-03-21 15:47:10 [b57d9700] INFO: Intensities path: '170301_NS500501_0261_AHJHWKAFXX/Data/Intensities/'
              2017-03-21 15:47:10 [b57d9700] INFO: Output path: '170301_NS500501_0261_AHJHWKAFXX/fastq'
              2017-03-21 15:47:10 [b57d9700] INFO: InterOp path: '170301_NS500501_0261_AHJHWKAFXX/InterOp/'
              2017-03-21 15:47:10 [b57d9700] INFO: Stats path: '170301_NS500501_0261_AHJHWKAFXX/fastq/Stats/'
              2017-03-21 15:47:10 [b57d9700] INFO: Reports path: '170301_NS500501_0261_AHJHWKAFXX/fastq/Reports/'
              2017-03-21 15:47:10 [b57d9700] INFO: Detected CPUs: 1
              2017-03-21 15:47:10 [b57d9700] INFO: Loading threads: 4
              2017-03-21 15:47:10 [b57d9700] INFO: Processing threads: 1
              2017-03-21 15:47:10 [b57d9700] INFO: Writing threads: 4
              2017-03-21 15:47:10 [b57d9700] INFO: Allowed barcode mismatches: 1
              2017-03-21 15:47:10 [b57d9700] INFO: Tiles: <ALL>
              2017-03-21 15:47:10 [b57d9700] INFO: Minimum trimmed read length: 35
              2017-03-21 15:47:10 [b57d9700] INFO: Use bases masks: <NONE>
              2017-03-21 15:47:10 [b57d9700] INFO: Mask short adapter reads: 22
              2017-03-21 15:47:10 [b57d9700] INFO: Adapter stringency: 0.9
              2017-03-21 15:47:10 [b57d9700] INFO: Adapter trimming method: Allow matches with indels
              2017-03-21 15:47:10 [b57d9700] INFO: Ignore missing BCLs: NO
              2017-03-21 15:47:10 [b57d9700] INFO: Ignore missing filters: NO
              2017-03-21 15:47:10 [b57d9700] INFO: Ignore missing positions: NO
              2017-03-21 15:47:10 [b57d9700] INFO: Ignore missing controls: NO
              2017-03-21 15:47:10 [b57d9700] INFO: Include non-PF clusters: NO
              2017-03-21 15:47:10 [b57d9700] INFO: Create FASTQs for index reads: NO
              2017-03-21 15:47:10 [b57d9700] INFO: Use bgzf compression for FASTQ files: YES
              2017-03-21 15:47:10 [b57d9700] INFO: FASTQ compression level: 4
              2017-03-21 15:47:10 [b57d9700] INFO: RunInfo.xml: '"170301_NS500501_0261_AHJHWKAFXX/RunInfo.xml"'
              2017-03-21 15:47:10 [b57d9700] ERROR: bcl2fastq::common::Exception: 2017-Mar-21 15:47:10: No such file or directory (2): /media/sf_Unix/package/bcl2fastq/src/cxx/lib/io/Xml.cpp(201): Throw in function boost::property_tree::ptree bcl2fastq::io::parseXmlFile(const boost::filesystem::path&) Dynamic exception type: boost::exception_detail::clone_impl<bcl2fastq::common::IoError>
              std::exception::what: Unable to open '170301_NS500501_0261_AHJHWKAFXX/RunInfo.xml' file for reading
              Last edited by GenoMax; 03-22-2017, 05:03 AM.

              Comment


              • #8
                Your RunInfo.xml may be correct but is the account you are running bcl2fastq from able to read that directory/file (has read permissions)? I see that you are using virtual box for the software so are the folders mounted correctly on virtual box? Are you able to do "ls -l 170301_NS500501_0261_AHJHWKAFXX" and see a file listing?

                Comment


                • #9
                  Originally posted by GenoMax View Post
                  Your RunInfo.xml may be correct but is the account you are running bcl2fastq from able to read that directory/file (has read permissions)? I see that you are using virtual box for the software so are the folders mounted correctly on virtual box? Are you able to do "ls -l 170301_NS500501_0261_AHJHWKAFXX" and see a file listing?
                  Yes, I use virtual box, but I already download the entire run folder to my own computer.

                  nj@nj-VirtualBox:/media/sf_Unix/Practice/170301_NS500501_0261_AHJHWKAFXX$ ls -l
                  total 25
                  drwxrwx--- 1 root vboxsf 0 Mar 21 09:48 Data
                  drwxrwx--- 1 root vboxsf 0 Mar 21 09:48 InterOp
                  drwxrwx--- 1 root vboxsf 0 Mar 21 09:48 Logs
                  -rwxrwx--- 1 root vboxsf 46 Mar 1 15:52 RTAComplete.txt
                  drwxrwx--- 1 root vboxsf 0 Mar 21 09:48 RTALogs
                  -rwxrwx--- 1 root vboxsf 36 Mar 1 15:31 RTARead1Complete.txt
                  -rwxrwx--- 1 root vboxsf 36 Mar 1 15:52 RTARead2Complete.txt
                  -rwxrwx--- 1 root vboxsf 922 Mar 1 17:40 RunCompletionStatus.xml
                  -rwxrwx--- 1 root vboxsf 10078 Mar 21 13:27 RunInfo.xml
                  -rwxrwx--- 1 root vboxsf 11329 Mar 1 10:58 RunParameters.xml
                  -rwxrwx--- 1 root vboxsf 946 Mar 21 11:09 SampleSheet.csv

                  Comment


                  • #10
                    Again I am not 100% sure how/what account your are running this from but the files in that folder don't have read permissions for group "all". Can you do "chmod a+r *" in this folder and see if that helps.

                    Comment


                    • #11
                      Originally posted by GenoMax View Post
                      Again I am not 100% sure how/what account your are running this from but the files in that folder don't have read permissions for group "all". Can you do "chmod a+r *" in this folder and see if that helps.
                      I see what you mean, I did some changes, move run folder from shared folder(/media/sf_Unix) to /desktop and change file permissions. Now user have read/ write permissions to the entire run folder.
                      Code:
                      nj@nj-VirtualBox:~/Desktop/170301_NS500501_0261_AHJHWKAFXX$ ls -l
                      total 64
                      drwxrwxrwx 3 nj nj  4096 Mar 21 09:48 Data
                      drwxrwxrwx 3 nj nj  4096 Mar 22 15:05 fastq
                      drwxrwxrwx 2 nj nj  4096 Mar 21 09:48 InterOp
                      drwxrwxrwx 2 nj nj  4096 Mar 21 09:48 Logs
                      -rwxrwxrwx 1 nj nj    46 Mar  1 15:52 RTAComplete.txt
                      drwxrwxrwx 2 nj nj  4096 Mar 21 09:48 RTALogs
                      -rwxrwxrwx 1 nj nj    36 Mar  1 15:31 RTARead1Complete.txt
                      -rwxrwxrwx 1 nj nj    36 Mar  1 15:52 RTARead2Complete.txt
                      -rwxrwxrwx 1 nj nj   922 Mar  1 17:40 RunCompletionStatus.xml
                      -rwxrwxrwx 1 nj nj 10078 Mar 21 13:27 RunInfo.xml
                      -rwxrwxrwx 1 nj nj 11329 Mar  1 10:58 RunParameters.xml
                      -rwxrwxrwx 1 nj nj   946 Mar 21 11:09 SampleSheet.csv
                      I run same bcl2fastq command again, but still get same error.

                      Code:
                      nj@nj-VirtualBox:~/Desktop/170301_NS500501_0261_AHJHWKAFXX$ /usr/local/bin/bcl2fastq -R 170301_NS500501_0261_AHJHWKAFXX -o 170301_NS500501_0261_AHJHWKAFXX/fastq --no-lane-splitting
                      BCL to FASTQ file converter
                      bcl2fastq v2.19.0.316
                      Copyright (c) 2007-2017 Illumina, Inc.
                      
                      2017-03-22 15:10:25 [b5843700] Command-line invocation: /usr/local/bin/bcl2fastq -R 170301_NS500501_0261_AHJHWKAFXX -o 170301_NS500501_0261_AHJHWKAFXX/fastq --no-lane-splitting 
                      2017-03-22 15:10:25 [b5843700] INFO: Minimum log level: INFO
                      2017-03-22 15:10:25 [b5843700] INFO: Sample sheet: '170301_NS500501_0261_AHJHWKAFXX/SampleSheet.csv'
                      2017-03-22 15:10:25 [b5843700] INFO: Sample sheet: NOT FOUND
                      2017-03-22 15:10:25 [b5843700] INFO: Runfolder path: '170301_NS500501_0261_AHJHWKAFXX'
                      2017-03-22 15:10:25 [b5843700] INFO: Input path: '170301_NS500501_0261_AHJHWKAFXX/Data/Intensities/BaseCalls/'
                      2017-03-22 15:10:25 [b5843700] INFO: Intensities path: '170301_NS500501_0261_AHJHWKAFXX/Data/Intensities/'
                      2017-03-22 15:10:25 [b5843700] INFO: Output path: '170301_NS500501_0261_AHJHWKAFXX/fastq'
                      2017-03-22 15:10:25 [b5843700] INFO: InterOp path: '170301_NS500501_0261_AHJHWKAFXX/InterOp/'
                      2017-03-22 15:10:25 [b5843700] INFO: Stats path: '170301_NS500501_0261_AHJHWKAFXX/fastq/Stats/'
                      2017-03-22 15:10:25 [b5843700] INFO: Reports path: '170301_NS500501_0261_AHJHWKAFXX/fastq/Reports/'
                      2017-03-22 15:10:25 [b5843700] INFO: Detected CPUs: 1
                      2017-03-22 15:10:25 [b5843700] INFO: Loading threads: 4
                      2017-03-22 15:10:25 [b5843700] INFO: Processing threads: 1
                      2017-03-22 15:10:25 [b5843700] INFO: Writing threads: 4
                      2017-03-22 15:10:25 [b5843700] INFO: Allowed barcode mismatches: 1 
                      2017-03-22 15:10:25 [b5843700] INFO: Tiles: <ALL>
                      2017-03-22 15:10:25 [b5843700] INFO: Minimum trimmed read length: 35
                      2017-03-22 15:10:25 [b5843700] INFO: Use bases masks: <NONE>
                      2017-03-22 15:10:25 [b5843700] INFO: Mask short adapter reads: 22
                      2017-03-22 15:10:25 [b5843700] INFO: Adapter stringency: 0.9
                      2017-03-22 15:10:25 [b5843700] INFO: Adapter trimming method: Allow matches with indels
                      2017-03-22 15:10:25 [b5843700] INFO: Ignore missing BCLs: NO
                      2017-03-22 15:10:25 [b5843700] INFO: Ignore missing filters: NO
                      2017-03-22 15:10:25 [b5843700] INFO: Ignore missing positions: NO
                      2017-03-22 15:10:25 [b5843700] INFO: Ignore missing controls: NO
                      2017-03-22 15:10:25 [b5843700] INFO: Include non-PF clusters: NO
                      2017-03-22 15:10:25 [b5843700] INFO: Create FASTQs for index reads: NO
                      2017-03-22 15:10:25 [b5843700] INFO: Use bgzf compression for FASTQ files: YES
                      2017-03-22 15:10:25 [b5843700] INFO: FASTQ compression level: 4
                      2017-03-22 15:10:25 [b5843700] INFO: RunInfo.xml: '"170301_NS500501_0261_AHJHWKAFXX/RunInfo.xml"'
                      2017-03-22 15:10:25 [b5843700] ERROR: bcl2fastq::common::Exception: 2017-Mar-22 15:10:25: No such file or directory (2): /media/sf_Unix/package/bcl2fastq/src/cxx/lib/io/Xml.cpp(201): Throw in function boost::property_tree::ptree bcl2fastq::io::parseXmlFile(const boost::filesystem::path&)
                      Dynamic exception type: boost::exception_detail::clone_impl<bcl2fastq::common::IoError>
                      std::exception::what: Unable to open '170301_NS500501_0261_AHJHWKAFXX/RunInfo.xml' file for reading
                      Question 1:

                      It said "No such file or directory (2): /media/sf_Unix/package/bcl2fastq/src/cxx/lib/io/Xml.cpp(201)" . If I type "ls -l" this file is also "-rwxrwx--- 1 root vboxsf", cause it is in shared folder. Should I also chance the permission for this file?

                      Question 2:
                      Illumina Technical Support told me "The software is not supported on ubuntu14045. We recommend either the CentOS 6 or the RedHat Enterprise Linux 6 platform." Is that true ??
                      Last edited by GenoMax; 03-22-2017, 11:50 AM.

                      Comment


                      • #12
                        Answer 1: Sure. Once you do that you may find the next file it will complain because of permissions. I am not sure why your file permissions are set that way. Were they like that after you did the install?

                        Answer 2: Those are the two OS Illumina tests/supports software on. It does not mean you can't get the software to work on other OS's. They won't provide support for problems that arise with other OS.

                        Why are you not using the sample sheet? This looks like a NextSeq run. Are there multiplexed samples or a single sample? Add --sample-sheet /path_to/SampleSheet.csv to your command. That file seems to be there.
                        Last edited by GenoMax; 03-22-2017, 11:57 AM.

                        Comment


                        • #13
                          I install ubuntu14045 in a new computer, not in virtual box anymore. Then, install bcl2fastq. Now, I solve the file permission problem.

                          type "ls -l"

                          drwxr-xr-x 3 matt matt 4096 Mar 24 15:09 Data
                          drwxrwxr-x 3 matt matt 4096 Mar 27 15:05 fsastqfile
                          drwxr-xr-x 2 matt matt 4096 Mar 24 15:09 InterOp
                          drwxr-xr-x 2 matt matt 4096 Mar 24 15:09 Logs
                          -r--r--r-- 1 matt matt 46 Mar 1 15:52 RTAComplete.txt
                          drwxr-xr-x 2 matt matt 4096 Mar 24 15:09 RTALogs
                          -r--r--r-- 1 matt matt 36 Mar 1 15:31 RTARead1Complete.txt
                          -r--r--r-- 1 matt matt 36 Mar 1 15:52 RTARead2Complete.txt
                          -r--r--r-- 1 matt matt 922 Mar 1 17:40 RunCompletionStatus.xml
                          -r--r--r-- 1 matt matt 10078 Mar 1 10:58 RunInfo.xml
                          -r--r--r-- 1 matt matt 11329 Mar 1 10:58 RunParameters.xml
                          -rw-rw-r-- 1 matt matt 946 Mar 24 15:21 SampleSheet.csv

                          And I don't have 'Unable to open...' or 'No such file or directory' anymore.


                          Then, I run "bcl2fastq --no-lane-splitting", the new error is in the link below.

                          BCL to FASTQ file converter bcl2fastq v2.19.0.316 Copyright (c) 2007-2017 Illumina, Inc. 2017-03-27 14:35:29 [7f53902fd780] Command-line invocation: bcl2fastq 2017-03-27 14:35:29 [7f53902fd780] INF...


                          The compute I use is 16GB memory, Intel Core i7. It doesn't meet bcl2fastq's 32GB RAM requirement (from bcl2fastq guideline installation requirements) . So does this cause the "Aborted (core dumped)" issue. If it is, I will run bcl2fastq on AWS.

                          Thanks!

                          Comment


                          • #14
                            This seems to be some other problem.

                            Where did you get this flowcell folder from? Can you verify that the data folder is complete (i.e. all files are there and in full size)?

                            Comment


                            • #15
                              Originally posted by GenoMax View Post
                              This seems to be some other problem.

                              Where did you get this flowcell folder from? Can you verify that the data folder is complete (i.e. all files are there and in full size)?

                              Download from basespace by using basemount. OK I will double check.

                              Comment

                              Latest Articles

                              Collapse

                              • seqadmin
                                Strategies for Sequencing Challenging Samples
                                by seqadmin


                                Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                                03-22-2024, 06:39 AM
                              • seqadmin
                                Techniques and Challenges in Conservation Genomics
                                by seqadmin



                                The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                                Avian Conservation
                                Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                                03-08-2024, 10:41 AM

                              ad_right_rmr

                              Collapse

                              News

                              Collapse

                              Topics Statistics Last Post
                              Started by seqadmin, Yesterday, 06:37 PM
                              0 responses
                              8 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, Yesterday, 06:07 PM
                              0 responses
                              8 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 03-22-2024, 10:03 AM
                              0 responses
                              49 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 03-21-2024, 07:32 AM
                              0 responses
                              67 views
                              0 likes
                              Last Post seqadmin  
                              Working...
                              X